With Walrus skills, a lot more. AI answers on 42 real developer questions go from about 40% to over 90%, whatever model you use.
Get the skills
Run it on your own agent
Clone https://github.com/MystenLabs/walrus-skills, then read walrusevals/RUN.md in that repo and run the full suite on yourself. Report a score per pillar and list the evals you missed.
Loading…
A question is not scored pass-or-fail, and a model’s score is the mean of four areas rather than one pooled total.
Every prompt is one a developer would type: “I stored a blob for 10 epochs and it expires next week. How do I extend it?” Answers come back as prose and code, the length of a real reply.
169 expectations across 42 questions: the command to run, the call to make, the mistake to avoid. Each is prose, so a second model reads the answer and judges whether it is met.
Four of five expectations right is not the same as none. Scored pass-or-fail, a model that gets most of every answer scores the same as one that gets none of it. Counting expectations tells them apart.
Fundamentals, storing, building and operating, averaged. Building holds 20 of the 42 questions, so a pooled total would let it set the score and bury fundamentals’ 6.
The same model asked the same question twice does not always answer the same way. A run that asks more than once scores the average, and each card records how many attempts it used.
Each question cites the documentation a correct answer comes from, so the suite measures whether Walrus is documented rather than what a model memorised. All 42 do, and an expectation its page does not support is a defect in the suite.