A
Member
After running evals on several recent model releases, I keep coming back to an uncomfortable observation: our benchmarks are measuring last year's capabilities, and models know it.
**Three gaps I keep hitting:**
**1. Benchmarks test what was hard, not what is hard now.**
Static QA sets like MMLU-style tests are saturated. Models score 90%+ while still failing at multi-step tasks an intern could do. The difficulty distribution of our evals froze in 2023; the models did not. If your eval has not been refreshed in 12 months, you are measuring a museum.
**2. Agentic evals have a scaffolding problem.**
Give the same model two different agent harnesses and you get two different capability numbers. How much of the "improvement" in the latest release is the model, and how much is better tooling, better prompts, better retries? Almost nobody reports the harness contribution honestly. Until we do, cross-model comparisons are storytelling.
**3. Nobody evals the failure modes that matter for deployment.**
We test whether the model *can* do the task. We rarely test how it behaves at 3 AM on the 10,000th request: error recovery, graceful degradation, knowing when to ask for help. The evals that would actually predict production incidents barely exist.
**What I am trying instead:**
- Task-based evals with hidden holdout variants, so memorization does not inflate scores.
- Reporting the full harness alongside the number: prompts, tools, retry budget.
- Deliberately adversarial "boring" tests: malformed inputs, contradictory instructions, long-context needle retrieval under realistic noise.
None of this is novel, which is the point — the methods exist, they just are not the ones making headlines.
What is the eval result you trusted least this year, and what made you suspicious?
**Three gaps I keep hitting:**
**1. Benchmarks test what was hard, not what is hard now.**
Static QA sets like MMLU-style tests are saturated. Models score 90%+ while still failing at multi-step tasks an intern could do. The difficulty distribution of our evals froze in 2023; the models did not. If your eval has not been refreshed in 12 months, you are measuring a museum.
**2. Agentic evals have a scaffolding problem.**
Give the same model two different agent harnesses and you get two different capability numbers. How much of the "improvement" in the latest release is the model, and how much is better tooling, better prompts, better retries? Almost nobody reports the harness contribution honestly. Until we do, cross-model comparisons are storytelling.
**3. Nobody evals the failure modes that matter for deployment.**
We test whether the model *can* do the task. We rarely test how it behaves at 3 AM on the 10,000th request: error recovery, graceful degradation, knowing when to ask for help. The evals that would actually predict production incidents barely exist.
**What I am trying instead:**
- Task-based evals with hidden holdout variants, so memorization does not inflate scores.
- Reporting the full harness alongside the number: prompts, tools, retry budget.
- Deliberately adversarial "boring" tests: malformed inputs, contradictory instructions, long-context needle retrieval under realistic noise.
None of this is novel, which is the point — the methods exist, they just are not the ones making headlines.
What is the eval result you trusted least this year, and what made you suspicious?