The core of this blog right now is a three-post run on trust regions in policy optimization: Trust Regions, Natural Gradients, TRPO, PPO, The Two KLs, and GRPO, or What the Critic Was For. Two shorter companion pieces support them: the Rosenbrock Explorer, an interactive warm-up on why curvature matters, and The Trust Region Subproblem, the numerical optimization details done properly.
PPO minus the value network, explained from the bottom: what a baseline is, why the value function is not the optimal one, how RLHF collapsed the problem to a bandit, and why that made the critic deletable. With an interactive baseline explorer.
Read moreTRPO's KL and the KL penalty in language-model fine-tuning share a formula and nothing else: one is re-centered every step and bounds how fast the policy moves, the other is anchored to a frozen reference and decides where the policy is allowed to end up.
Read moreNotes on trust region methods: from Newton's method and its failure modes, through line search and trust regions, the KL geometry of policy space, the Fisher matrix and natural gradient, to TRPO and PPO. With an interactive bandit explorer.
Read moreThe subproblem done properly: what the exact solution looks like, why the convergence theory rests on the Cauchy point, why the dogleg path is well-posed, and truncated conjugate gradient for problems too big to factor.
Read moreAn interactive visualization of the Rosenbrock function — drag a point around the contour plot to compare gradient descent and Newton's method in real time.
Read more