The core problem is that fine-tuning shifts the weights that encode old skills, and the loss on new data gives no signal about that drift until users hit it. So treat an update as a release, not a training run. First, freeze a regression suite before you touch anything: a fixed set of prompts and tasks covering the old capabilities that matter, with pass thresholds, run on the current model to get a baseline. If you cannot measure the old skills, you cannot protect them.
Second, limit the blast radius. Low-rank adapters or a small set of updated layers are usually safer than full fine-tuning, because fewer parameters move and the base behaviour is easier to keep. Keep the adapter versioned and reversible, and keep the old model available for rollback.
Third, mix replay data into every update: a sample of the original training or production distribution, not just the new task. Even a modest replay fraction sharply reduces forgetting, and it also gives you a training-time signal that the old distribution is still being served.
Finally, deploy in stages. Canary the new model to a small share of traffic, compare the regression suite and real error rates against the baseline, and only then roll out fully. If a metric degrades without a clear cause, roll back first and diagnose second.