What I Learned Building Evaluations for Coding Agents
In April of this year, I set out to build an evaluation framework to move away from “vibe-based” assumptions about models and towards evidence-backed assessments. I wanted to answer a question that none of the existing benchmarks could answer for me:
“How does this model perform in my codebase?"
I spent most of my time making agent runs observable, but less time tackling the hardest problem: evaluating the quality and correctness of their implementations.
Existing benchmarks
I’d been reading agentic coding benchmarks like SWE-bench and looking at aggregated results on Artificial Analysis. I understood that higher scores generally meant better performance on a particular benchmark, but that didn’t translate into anything particularly useful for me. Whilst both were helpful for getting a general sense of what models were capable of, they never reflected my work.
Replaying pull requests
This led me to build ShipTest in April 2026. The fundamental idea was that you could install an npm package into your codebase and run evaluations against your own code using your own pull requests.
The principles were simple:
- Agents wouldn’t have previously seen your codebase, reducing the chance that they’d already encountered the problems being evaluated.
- You know how code should be written in your own codebase because you’ve got firsthand experience building it.
- The evaluation environment would always be reset, preventing agents from accessing Git history that could reveal the original implementation.
- The evaluation environment should reflect a real development environment, using an agent harness that developers could actually use.
My focus was on developer experience. I didn’t want people to fight my framework; I just wanted it to work. You could use AI to generate evaluation tasks from Git history or merged pull requests, or write them by hand.
The instructions in the task.md file were passed directly to the model, as if the user had typed them themselves.
An example task prompt in ShipTest.
The underlying harness used to orchestrate the agents was PI. I chose PI because it’s a minimal harness that isn’t proprietary to any of the AI labs.
This gave me a more consistent environment for comparing different models, whether they were from OpenAI, Anthropic or Google. Rather than using a harness designed specifically for a benchmark, I wanted to use one that reflected how developers could actually work with these models.
Task execution revolved around Git and Git worktrees. I used Git to check out the repository at specific commits, typically the commit immediately before the change I wanted to evaluate was merged. Worktrees allowed me to run multiple evaluations concurrently.
I developed ShipTest to the point where I was using it to evaluate itself. I’d take pull requests I’d previously merged, run different models against the same tasks, and compare their implementations using the built-in pull request viewer, alongside information such as tool calls and token usage.
The candidate diff view presents a model’s proposed changes alongside its evaluation evidence.
The outcome matrix brings model attempts together for comparison.
Tackling AI evaluations professionally
Around this time, I interviewed for and secured a position on the AI Engineering team at my current workplace. I was brought onto the team to evaluate our coding agents, with the goal of improving their performance and increasing their impact across the wider tech department. Since I’d now be working on the same problem professionally, I stopped developing ShipTest in my spare time.
Evaluating agents at work
We were already using the Mastra framework in a few places, mainly for conversational AI and triaging support tickets. It felt like the natural choice for evaluations because other teams were already familiar with it.
Mastra provides functionality and APIs for both evaluations and orchestration, including access to tool calls, token usage and scorers (evaluations).
I put together a small dataset of reproducible tasks that would run against our API, ranging in difficulty from very easy to very hard.
For the scorers, I defined metrics that I thought would give useful indications of model behaviour. These used a combination of LLM-as-a-judge evaluations (using an LLM to assess another LLM’s output) and deterministic tests.
Mastra’s UI is quite information-dense, so I built a custom interface on top of it to make the evaluation results easier to understand. This included a pull request viewer that presented the models’ proposed changes in a familiar format.
Retrospective
Looking back at both projects, I built the infrastructure needed to run evaluations and observe model behaviour and implementations. However, what both systems lacked was an automated way to verify whether an implementation was actually correct. That’s arguably one of the most difficult parts of evaluating coding agents.
I had a few ideas about how to tackle this, but none of them felt quite right.
One approach was similar to SWE-bench: using tests from the original merged pull request to validate the model’s implementation.
The problem is that these tests might be too closely coupled to the original implementation. For example, a test might assert that a particular field has a specific name. If the model used a different name that still satisfied the original requirements, the test would fail despite the implementation potentially being valid.
One way around this would be to make the task instructions extremely specific about the expected implementation. But I’ve got mixed feelings about that, because you’re essentially nudging the model towards the “approved” answer. That doesn’t necessarily reflect real development work either.
I think a possible answer is to create specific tests for each evaluation task and grade the implementation rather than relying on a binary pass or fail.
Instead of failing the entire evaluation because one field name doesn’t match, you could deduct points from the model’s overall score.
The caveat is that you’d need some form of weighted scoring system to account for the severity of different problems. A minor naming discrepancy shouldn’t necessarily carry the same penalty as an implementation that fundamentally doesn’t work.
If I was starting again, I would spend more time working out the requirements and how to evaluate the results the model produced. Instead of spending the time on behavioural observations.


