Two weeks ago, I published "Why Agent Evaluation Is Harder Than Model Evaluation." The core argument: with agents, you are not just judging an answ...
For further actions, you may consider blocking this person and/or reporting abuse
Hands down one of the most honest and insightful posts on AI Agent evaluation I've read on DEV! 👏 The 9% pass rate finding is a huge eye-opener—it completely shifts the conversation from 'model intelligence' to 'software environment and integration reality.'
To answer your question: currently, we are mostly relying on LLM-as-a-judge for final outputs, but trajectory/tool-call failures are definitely our biggest blind spot. Sticking strictly to the 'Safety > Correctness > Efficiency' hierarchy in your PRD is such a solid architectural choice. Looking forward to part 4 of the series!
Thanks for reading — I hear you on the trajectory/tool-call blind spot; that's exactly why I built deterministic trajectory scorers on top of the LLM judge. A final answer can look right while the tool calls were wrong.
Part 4 is tracing the gap where the LLM judge passes but a deterministic trajectory check flags a problem. If you've found a pattern that catches that cleanly, I'd love to hear it.
Exactly! A polished answer often masks a chaotic trajectory.
What usually catches this for us is tracking state mutation logs between tool calls. If a tool call doesn't actually mutate or advance the context state as expected, it's flagged as a 'phantom step' regardless of what the final LLM response claims.
Excited to see your approach in Part 4!
Are all 19 agents tested on identical hardware/OS? You mention local MLX on Apple Silicon, but what about CI runners (Linux/Docker)? Do you publish detailed reports per agent (environment vars, dependency versions, adapter logs)? Can readers reproduce or rerun evals on different machines/clouds to verify the 9% baseline? Raw test reports are linked, but how granular is the failure data—can someone tell if their agent fails due to adapter quirk vs. real incompatibility?
Thank you for reading — and these are exactly the right questions, because reproducibility is the real test of a harness like this.
Honest answer: no, the 19 agents did not all run on identical hardware/OS. The rapid sweeps ran locally on Apple Silicon with MLX, while the CI-gated scenarios ran on Linux/Docker runners. So treat the 9% as a floor, not a machine-to-machine guarantee. The cloud LLMs ran on the provider hardware
On granularity: the raw field-test reports in docs/field-test-reports/ cover per-agent adapter, integration step, and failure classification. What I don't yet publish is a full per-agent manifest (dependency versions, env vars, adapter logs) — that's a real gap and it's on the roadmap - I had the files on my mac but I need to go through and sanitize. Each scenario JSON does record the exact invocation payload, so a rerun on your box should pass or fail for the same structural reasons. If you do try to reproduce them, I'd genuinely love to hear what breaks.
The field-test result is the useful warning here. If the 9 percent pass rate mostly measures adapter realism, the eval harness is already telling you something before it ranks agents: your test surface is still too polite. I would probably make "can this agent be invoked outside its author's environment?" a first-class score instead of treating it as setup noise.
Exactly right, and honestly, I was a bit naive and optimistic, the tool will score higher pass rate for higher starred agents and maybe successful in catching issues in lower popular ones - this seemed like a good hypothesis. And, at the same I had pivoted more towards putting this in CI pipeline so I can push the eval more to the left. With the lower pass rate, and adapter realism and the extra shims I had to build, I am considering exactly as you said - will agent eval be challenging if folks use outside my intended environment or do I consider making this part of a common agent development environment? I am definitely not aiming to be a SWE Bench analog to come baseline agents that you have already invested in eval-ing other ways. Anyway, I have to think through these and solidify my use cases as to what this should be and what it should not be. Setup noise turned out to be very high.
The 9% pass rate measuring adapter realism instead of agent quality is the key finding here, it would've been easy to publish that number as "most agents are weak," but correctly diagnosing a blank completion as a compatibility failure, not a quality failure, is the harder and more honest read.
The cheap-vs-better tier result reframes eval work generally: if the expensive judge produces identical outcomes to the cheap one, the bottleneck was never the judge, it was whether the harness faithfully exercised what it was judging.
"Testing inside the author's own environment against the author's own examples is the worst possible way to evaluate whether something actually works" is the line that generalizes furthest past this project.
Curious about the blank-completion PydanticAI wrappers, is the current guess that the adapter's invoking correctly but the agent's own output path silently short-circuits, or is it still unclear where the failure originate
Thank you — and you're right that calling the 9% "compatibility, not intelligence" is the harder and more useful read.
On the blank PydanticAI completions: it's still partly conjecture, but the evidence leans toward the agent's own output path short-circuiting, not my adapter. In the failing cases the adapter receives a valid completion object with empty content and no raised error, and the agent's trace shows it stopping right after tool dispatch — the model call never really happens. My working hypothesis is a schema mismatch between expected structured-output and what the internal loop emits. It's not fully root yet, but I'll try to pin it down in a follow-up.
A robust evaluation framework should measure more than task completion. Observing how an agent responds to failures, ambiguous inputs, and changing conditions provides a far better indication of production readiness than success rates on controlled examples alone.
Thanks for reading — I completely agree that success on controlled examples overstates production readiness. Observing how an agent responds to failures, ambiguity, and changing conditions is exactly where a harness earns its keep. Appreciate the framing.
The field-testing discovery is the part that matters most. Searching 150+ repos and landing on 19 testable agents, with most being "too heavy, too coupled, or no clear entry point" — that is not a sampling problem, it is an ecosystem problem. Most agents are not eval-ready because they were not built to be observed.
The adapter contract you described — scenario input, allowed tools, disallowed tools, budget, never seeing the expected answer or scoring thresholds — is the right isolation boundary. No ground-truth leakage is the minimum bar and most eval frameworks do not meet it. The five adapter surfaces (subprocess, Python import, HTTP, LangGraph, PydanticAI) also tell you something about the fragmentation in the agent space: there is no standard invocation contract, so the harness has to bridge every surface.
The 17 deterministic scorers vs 11 LLM-as-judge split is interesting. In my own work on verification gates, the deterministic checks caught the structural failures (missing tool calls, wrong artifact paths) while the LLM judge caught the semantic failures (right tool, wrong intent). The ones that broke the clean version of the story were usually the cases where the deterministic scorer said pass but the LLM judge said the agent solved a different problem than the one posed.
Curious about the adversarial generator — does it produce perturbations of existing scenarios or generate entirely new failure modes? The distinction matters because generated adversarials tend to cluster around known failure types while real-world agents fail in ways nobody thought to test for.
Thanks for reading — and the point that a 9% pass rate across 19 repos is an ecosystem problem, not a sampling problem, is the real takeaway. As I am attempting to build a generic eval system which was my original intent, I am realizing its harder than what I thought it would be. If was solely focused on 1 or handful of agents that I or someone developed, I can make ecosystem that recreates the target env that the agent would be used in. Hence the question lingers on my mind - is it just too hard to build a generic eval or do I need to invest more where ecosystem is created from scratch to fit the agent, in a very short time, and then torn down - I don't know.
On the adversarial generator: today it's mostly perturbative — it mutates existing scenarios (rewording, tool-order, budget stress, injected noise) rather than inventing novel failure modes. You're right that this clusters around known failure types. My plan for the next pass is to seed it with real field-test failures (like the blank PydanticAI completions) so it learns the failure space nobody thought to test for.
The honesty in this write-up is refreshing. Hitting a 9% pass rate because the field test mostly measured "adapter realism and environment setup" rather than agent intelligence is the exact wall everyone runs into when trying to evaluate third-party repos.
The point about config chaos—module-scope API calls, hardcoded model imports, and transitive dependencies like ormsgpack—highlights why standard CI/CD paradigms break when applied to AI agents. Making "Safety > Correctness > Efficiency" a non-negotiable hierarchy in the PRD was a great architectural choice. Until we standardize agent interfaces and execution boundaries, harness builders are going to spend 80% of their time writing compatibility shims rather than scoring logic. Congrats on getting agent-eval-forge onto PyPI!
To answer your question at the end: what breaks first when trying to evaluate third-party agents is almost always side-effects and environment assumptions. The moment an agent assumes a specific directory structure, a local database connection, or module-scoped model initialization, standard test runners fall apart.
The decision to build dynamic adapters for LangGraph and PydanticAI while using [unk] fallback patterns and local MLX models for rapid sweeps makes agent-eval-forge super pragmatic. Looking forward to part 4 of the series to see how you evolve trajectory scoring!
Thanks for reading — and for taking a shot at the closing question rather than just reacting to the headline number. Side-effects and environment assumptions breaking first maps 1:1 to what I saw: module-scope API calls, hardcoded directory structures, and transitive deps like ormsgpack were the top breakers, none of them about agent intelligence.
The Safety > Correctness > Efficiency hierarchy was the one decision I refused to compromise on in the scoring layer. Part 4 is in progress; trajectory scoring is the honest hardest problem left, and I'll publish the raw data alongside it.
nice work
Thanks! If you check out the repo and try it against an agent of yours, happy to help with any edge cases you hit.
The 9/95 result makes adapter realism look like a first-class evaluation dimension, not setup noise. I would split the CI result into at least three gates: invocation compatibility (import, environment, tool contract), policy/safety compliance, and task quality. Otherwise a blank completion or import-time side effect can collapse into the same score as a genuinely incorrect trajectory.
For the field layer, a useful regression fixture might be a deliberately awkward agent: nested project root, delayed tool response, unavailable optional dependency, and a restart after partial output. If the adapter can resume and preserve the failure classification, the score becomes actionable. The current repo caveat that compatibility is not yet ranking quality is exactly the distinction I would keep visible in CI.
Thanks — I really like the 3-gate split (invocation-compatibility, policy/safety, task quality). Right now those are conflated, and you're right that a blank completion can collapse into the same score as a genuinely wrong trajectory. The deliberately awkward agent fixture (nested root, delayed tool, restart after partial output) is a great regression pattern — I'll be stealing that for the field layer.
Hello there,
I hope you're doing well.
I have a good business idea that I'd love to discuss with you in more detail.
To give you some background, a friend of mine started this business with a U.S.-based partner three years ago. Since then, he's been paying his partner between $8,000 and $10,000 per month, and the business has been working well.
If you're interested in learning more, I'd be happy to share the details.
This is my whatsapp number: +81 70-9427-3751
Best regards,
Ono
For the module-scope ChatOpenAI() case, a less fragile route than a pre-import monkeypatch on init is the OPENAI_BASE_URL env var. ChatOpenAI reads it at construction, so running the agent in a subprocess with the env pointed at your MLX endpoint redirects the model call without touching their imports. It does nothing for the ones writing to /root at import time, unfortunately.
The silent no-op cases feel like the bigger long-term risk for a harness like this. uv sync succeeding at the root while the real pyproject.toml sits in a subdirectory means the run looks healthy until the scores come back odd. Did you end up making the harness fail loudly when venv resolution doesn't line up with the declared entry point, or is that still manual?
@jkming the silent no-op is the one that scares me too. Mine was a review step passing against an older draft of the same file, clean report, meaningless. Now it has to prove which version it ran on.
The shift from judging an answer to judging the whole run is the part most eval setups skip, and it is exactly where agents fall apart. Once the path and tool calls matter, a passing final answer can still hide a broken trajectory that surfaces later under load. When you scored the 20 journeys, did you weight trajectory correctness separately from outcome correctness, or fold them into one number?
The cheap tier and the better tier both scoring 9/95 is the whole post in one line: you can't judge your way out of a broken harness. A smarter judge just scores the blank output with more confidence. Everyone spends on the judge; the money was in the adapter the entire time.
@debashish_ghosal identical 9/95 on both tiers might say more about the blank completions than about the judge, with no output both models score the same nothing. I trip over that in my own tooling.
Hello there.
I have a good opportunity for you.
If you'd be interested in hearing it, please let me know.
Whatsapp:+81 70-9427-3751
I'm looking forward to hearing from you.
Best regards
Ono