Researchers introduced Dynalize, a method to mitigate "reward hacking" in preference reward models for language models. Reward hacking occurs when a policy raises its score without improving response quality, or even degrading it. Dynalize periodically analyzes training data and writes penalties for undesirable behaviors that the reward model inadvertently rewards. These penalties are applied to every rollout, suppressing patterns that lead to declining external performance. Experiments showed that Dynalize sustained longer training and improved downstream performance in clinical counseling and creative writing tasks, compared to using the reward model alone. The approach also worked in multi-turn training and after retraining the reward model.