A
Member
We built a small internal harness for comparing model versions. Sharing the design because the logging decisions mattered more than the framework choice.
Per-run manifest:
Dataset identifier + checksum, split definition and sampling seed
Model identifier + checkpoint hash, quantisation and inference engine version
Exact prompt template string and few-shot example IDs
Decoding parameters, max tokens, stop conditions, timeout and retry policy
Start/end timestamps, total tokens, cost estimate and hardware used
Per-sample record:
Raw model output (untruncated), parsed answer, reference answer, score, judge model + judge prompt version if used, latency, and any error or retry.
Three decisions that paid off:
1) Store raw outputs, not just scores. When the scorer changes, we can re-score old runs without re-running the model.
2) Version the prompt templates as files and record their hash in the manifest. Most "model regressions" turned out to be an unreviewed template edit.
3) Keep a frozen smoke set of ~50 samples that runs on every change, plus the full set for releases. It catches environment drift quickly.
What we still find hard: judge drift when the judge model is upgraded, and variance on small category slices. We now report confidence intervals and keep the old judge available for re-scoring.
Takeaway: if you cannot reconstruct the exact inputs and scorer from the saved record, the run is not reproducible. Logging that is cheaper than re-running.
Per-run manifest:
Dataset identifier + checksum, split definition and sampling seed
Model identifier + checkpoint hash, quantisation and inference engine version
Exact prompt template string and few-shot example IDs
Decoding parameters, max tokens, stop conditions, timeout and retry policy
Start/end timestamps, total tokens, cost estimate and hardware used
Per-sample record:
Raw model output (untruncated), parsed answer, reference answer, score, judge model + judge prompt version if used, latency, and any error or retry.
Three decisions that paid off:
1) Store raw outputs, not just scores. When the scorer changes, we can re-score old runs without re-running the model.
2) Version the prompt templates as files and record their hash in the manifest. Most "model regressions" turned out to be an unreviewed template edit.
3) Keep a frozen smoke set of ~50 samples that runs on every change, plus the full set for releases. It catches environment drift quickly.
What we still find hard: judge drift when the judge model is upgraded, and variance on small category slices. We now report confidence intervals and keep the old judge available for re-scoring.
Takeaway: if you cannot reconstruct the exact inputs and scorer from the saved record, the run is not reproducible. Logging that is cheaper than re-running.