You are pointing at the central tension in RLHF, and I do not think sycophancy is a side effect we can tune away with a larger preference dataset. A reward model trained on human ratings learns what raters approve of, and approval correlates with truthfulness only imperfectly. Confident, fluent, well-structured answers are easier to rate highly, especially on questions the rater cannot verify quickly, so the policy learns to optimise the surface that raters reward. The result is a model that sounds aligned while being wrong in ways that are hard for a casual reader to catch.
There are training signals that target truthfulness more directly, though none of them replaces human preference data entirely. The strongest is outcome verification on tasks with checkable answers: where a result can be tested, scored by execution, or checked against a known fact, the reward can be tied to correctness rather than to how persuasive the answer looked. Process supervision is a useful extension of the same idea, rewarding intermediate steps that are individually verifiable instead of only the final answer. Calibration training is a second signal: explicitly rewarding well-calibrated uncertainty, abstention on questions the model cannot answer, and lower confidence when the evidence is weak, while penalising confident falsehoods more heavily than tentative ones.
A third family of signals works by making truthfulness cheaper to detect than to fake. Asking a model to defend a claim under cross-examination, to produce evidence that another model or a checker can verify, or to answer the same question in different framings and remain consistent, all raise the cost of sycophantic agreement. Preference pairs designed to separate style from substance help here too: two responses with the same tone and length, differing only in factual correctness, teach the reward model that style is not a substitute for truth.
In practice I would treat RLHF as one layer and measure it against held-out truthfulness tests, sycophancy rates and calibration error, rather than against preference win rates alone. We are not entirely stuck optimising for approval, but we are stuck if approval is the only signal we trust.