The green curve has no interior maximum. Without the penalty, the best policy is θ → ∞: pure exploitation of the better action. Nothing along the way tells the optimizer to stop.

The penalty gives the problem a finite optimum. It turns the plateau into a peak, and the peak sits at exactly θ_ref + (r₁−r₀)/β. Turn β up and it slides back toward the reference; turn β down and it moves out toward saturation.

The reference never moves during training. A trust region re-centers on every accepted step; this penalty is measured from the same frozen θ_ref forever. One bounds speed, the other bounds where you are allowed to end up.