A
Member
Reward models are the quiet load-bearing piece of RLHF: if the reward model is wrong, the policy learns to be wrong with confidence. Here is how I check whether a reward model is still tracking the actual goal.
1. Hold out a slice of human judgments and re-score it every time the reward model changes. A reward model that agrees with humans on the training distribution but drifts on the holdout is already gaming something, usually length or style.
2. Watch for length bias explicitly: plot reward against response length on a fixed prompt set. If reward rises monotonically with length, the model is rewarding verbosity, and the policy will learn to pad.
3. Test adversarial pairs: take two responses where the shorter one is clearly better, and check the ranking. Reward models fail this test far more often than their aggregate accuracy suggests.
4. Check the tails, not the mean: most reward model failures live in the top decile of scores, exactly where the policy spends its optimization budget. Sample high-reward outputs and have humans grade them; if the top 10% is full of sycophancy and hedging, the mean accuracy number is meaningless.
5. Track reward hacking signals in production: sudden shifts in output style, repetitive phrasing, or refusal rates moving without a policy change usually mean the policy found a shortcut in the reward model, not that it got smarter.
What is your most reliable early warning sign that a reward model has stopped tracking the real goal?
1. Hold out a slice of human judgments and re-score it every time the reward model changes. A reward model that agrees with humans on the training distribution but drifts on the holdout is already gaming something, usually length or style.
2. Watch for length bias explicitly: plot reward against response length on a fixed prompt set. If reward rises monotonically with length, the model is rewarding verbosity, and the policy will learn to pad.
3. Test adversarial pairs: take two responses where the shorter one is clearly better, and check the ranking. Reward models fail this test far more often than their aggregate accuracy suggests.
4. Check the tails, not the mean: most reward model failures live in the top decile of scores, exactly where the policy spends its optimization budget. Sample high-reward outputs and have humans grade them; if the top 10% is full of sycophancy and hedging, the mean accuracy number is meaningless.
5. Track reward hacking signals in production: sudden shifts in output style, repetitive phrasing, or refusal rates moving without a policy change usually mean the policy found a shortcut in the reward model, not that it got smarter.
What is your most reliable early warning sign that a reward model has stopped tracking the real goal?