Your vibe eval passed. You ran it again and it failed. Now what?
I do evaluation and safety work on agentic harnesses for a living, and I still write bad skills. Being the subject matter expert does not save you. That is rather the point. Evaluating an agent once is already hard and almost nobody has the patience for it. Doing it across an organisation is harder: the practice has to outlive the engineer who set it up and survive the next model deprecation. It is toil, and it is unsexy work.
Most teams shipping agent skills measure nothing at all. The ones who start usually get a wrong number first, because LLMs are probabilistic machines and one green run tells you almost nothing. pass@k and pass^k came out of code generation benchmarks, where you sample k candidates and keep the best one. Your agent runs once, in front of a user, so the number you actually want is a different one.
Then there is what counts as a failure. A rate limit at 3am, a judge returning unparseable JSON, a timeout, and an agent that edited the test instead of fixing the code are four different events, and that is before the agents that cheat on purpose. All of it still only measures whether the agent did the job, never what it touched on the way: files it had no business reading, a secret that ended up in a transcript. Capability and containment need different assertions, and one run can answer both.
The hard parts, though, are organisational. Who is allowed to write the eval, when the person who knows what a good answer looks like is a domain expert rather than the engineer who owns the pipeline? And how do you reproduce an environment realistic enough to mean anything: the same system prompt, the same skill neighbourhood, the same sandbox, maybe even the same access your engineers have?
Key Takeaways:
A metric and an outcome taxonomy you can defend in a review. Why pass@k inflates agent reliability, what to report instead, and why pass, task failure, judge error, infrastructure error, timeout and reward hacking have to be six outcomes rather than one red bar.
How to assert on containment, not just capability. Did the agent stay in its sandbox, keep secrets out of the transcript, and solve the task rather than game the check. Same run, different assertions. The reward hacking examples are the ones people tend to remember.
A spec format a domain expert can author without an engineer, so the knowledge stays with the person who has it.
Why the model, the backend and the judge belong at the invocation and never in the committed spec. This one is cheap to get right in week one and expensive to retrofit in month nine.
How to run control arms across a whole set of skills instead of one at a time, which turns "is this context earning its tokens?" into a measurement. From my own runs: pass rate 33% to 100%, tokens 290K down to 180K, wall time 1m01s down to 42s.
