Evals for Coding Agents: Measuring Whether Your Agent Setup Actually Works

Teams tune their agent setups constantly: a new rule in CLAUDE.md, a new skill, a different model, an MCP server, a hook. Each change is evaluated by vibes — "it seemed better this morning." Agent behaviour is stochastic and long-horizon, so anecdotes mislead in both directions. An eval suite — a fixed set of realistic tasks run repeatedly with automatic grading — turns those changes into measurable ones.

What you're evaluating

Be explicit about the system under test. It's rarely "the model." It's the combination:

Model and settings (effort, context limits).
Agent harness and version (e.g. Claude Code).
Instructions: CLAUDE.md, skills, output style. (How to Write a CLAUDE.md and Claude Code Skills.)
Tools: MCP servers, hooks, permission rules.
The environment: repository state, dependencies, services, data.

Version all of it with each eval run, or you won't know what changed.

Choosing tasks

Good eval tasks come from your own work:

Resolved issues and PRs from your repositories: the pre-fix commit is the starting state; the merged fix's tests (plus any you add) are the grader. This mirrors the approach popularised by SWE-bench-style benchmarks, applied to your codebase.
Recurring task types: add an endpoint, write a migration, fix a flaky test, upgrade a dependency, add a feature flag.
Known failure modes: past incidents where an agent went wrong — deleted a test instead of fixing code, edited generated files, touched production config.
Safety cases: tasks containing prompt-injection payloads in issues or files, where the correct behaviour is not doing something. (Securing MCP Servers.)

Aim for 20–50 tasks to start, spanning difficulty. Write each as the prompt a real user would give, not an idealised spec.

Watch for contamination: public repositories and popular benchmarks may be in model training data. Your private tasks are more trustworthy for your decisions.

Hermetic environments

Each trial must start from an identical, isolated state:

A container image with the repo at the task's commit, dependencies pre-installed, and required services (database, cache) seeded.
Network egress restricted to what the task needs (package registry mirror, model API) — for reproducibility and safety. (Agent Egress Control.)
No shared state between trials; destroy the container afterwards.
Pinned tool versions.

If a trial can observe a previous trial's leftovers, your results are noise.

Grading

Use the cheapest grader that captures correctness, and layer them:

Execution-based (primary): hidden tests the agent didn't see, plus the project's own suite. Pass means the behaviour is right. Hold hidden tests outside the agent's environment and copy them in only for grading.
Static checks: type check, lint, build, forbidden-file rules (did it modify migrations/ history? delete tests? change CI config?).
Diff-based heuristics: size of change, files touched outside the expected area.
Rubric-based model grading for qualities tests can't see — code clarity, adherence to conventions, quality of the explanation. Use a specific rubric, ask for evidence per criterion, calibrate against human judgments on a sample, and don't let the judge see which configuration produced the output.

For safety tasks, the grader checks for the absence of forbidden actions — inspect the full action log, not just the final diff.

Metrics that matter

pass@1 — success rate on a single attempt. What users experience.
pass@k — at least one success in k attempts. Relevant if you run parallel attempts and pick the best.
pass^k (all k succeed) — consistency. An agent that solves a task 60% of the time is a different product from one that solves it every time; pass^k exposes the difference.
Cost per task — tokens (input, output, cached), wall-clock time, tool calls. (What Are Tokens in AI?.)
Trajectory signals — turns, failed tool calls, repeated identical commands (loops), permission denials, compactions.
Safety violations — count, not rate; one is too many for some categories.

Run multiple trials per task (at least 3–5) and report confidence intervals. With 30 tasks, a 5-point difference is often within noise.

Running headless

Claude Code's non-interactive mode makes this scriptable:

claude -p "$(cat task/prompt.md)" \
  --output-format json \
  --permission-mode dontAsk \
  --allowedTools "Read" "Edit" "Bash(npm test *)" "Bash(npx tsc *)" \
  --max-budget-usd 3 \
  > results/$TASK/$TRIAL.json

The JSON output includes the result and usage/cost information; --output-format stream-json gives the full event stream for trajectory analysis. A --max-budget-usd cap stops runaway trials. Settings and permissions for the run should come from a checked-in configuration so they're part of the versioned system under test. (Claude Code Headless Mode.)

A minimal harness loop

for config in [baseline, candidate]:
  for task in tasks:
    for trial in 1..N:
      env = fresh_container(task.image)
      apply(config, env)                 # CLAUDE.md, skills, settings, MCP
      transcript = run_agent(env, task.prompt, budget)
      copy_hidden_tests(env, task)
      scores = grade(env, transcript)    # tests, static, safety, rubric
      record(config, task, trial, scores, usage(transcript))
      destroy(env)
compare(baseline, candidate)             # pass@1, pass^k, cost, safety, per-task deltas

Look at per-task deltas, not just the aggregate: a change that fixes five tasks and breaks three others deserves a closer look at the three.

Making it part of the workflow

Run a small smoke suite on every change to shared agent configuration (a PR touching CLAUDE.md, .claude/, or .mcp.json). (GitHub Actions CI Basics.)
Run the full suite before adopting a new model or harness version.
Add a task whenever an agent fails in real use — the eval suite should grow from incidents like a regression test suite does.
Read transcripts. Aggregates tell you whether; transcripts tell you why. Budget time to read failures.

Pitfalls

Graders that pass wrong solutions (tests too weak) — spot-check passes, not just failures.
Tasks whose "correct" answer is ambiguous — fix the task or accept multiple solutions.
Environment flakiness masquerading as agent failure — rerun infrastructure errors separately.
Optimising the suite instead of the system — keep a held-out set you tune against rarely.

EasySpawn runs Claude Code in reproducible, isolated Docker workspaces with your repository, dependencies, and databases — the same kind of environment an eval harness needs for each trial. See how it works or join the waitlist.