Every model release comes with a benchmark chart showing it beating everything. Here's the evaluation routine I actually trust before putting a new model anywhere near production.

1. Benchmarks are the starting line, not the finish
Public benchmarks leak into training data and get gamed. I treat them as a coarse filter: if a model can't clear the basics, it's out. If it tops the chart, that tells me almost nothing about my use case.

2. Build a private eval set from real failures
The highest-signal eval I've ever built was 200 examples of cases where our current model failed in production — edge cases, ambiguous instructions, adversarial inputs from actual users. A new model has to beat the incumbent on our failures, not on MMLU.

3. Test the failure modes, not just the happy path
- Refusal calibration: does it refuse what it should and answer what it shouldn't refuse?
- Instruction following under conflict: contradictory instructions in system vs. user vs. retrieved context.
- Long-context honesty: does it say "I don't know" when the answer isn't in the provided documents, or does it confabulate smoothly?

4. Measure cost per solved task, not per token
A cheaper model that needs three retries and a verification pass can easily cost more than the expensive one that gets it right first time. Track end-to-end: tokens × retries × latency × error cost.

5. Run it against the incumbent blind
A/B the outputs with human raters who don't know which model produced which. Vibes-based evaluation ("this one feels smarter") is how you ship regressions with confidence.

What's in your model evaluation checklist?