Baseline Explorer

Baseline explorer

ĝ = ∇_θ log π(a) · (r(a) − b) · one sample, exact distribution

Sampling distribution · variance parabola

Drag the point along the parabola to move the baseline b. The mean above never moves

Policy & rewards

θ (logit)0.85

r₁ (action 1)1.00

r₀ (action 0)0.00

π(a=1) = σ(θ)0.700

Baseline

b0.00

value V = E[r]0.700

optimal b*0.300

One-sample estimate ĝ

if a=1 (prob σ)0.300

if a=0 (prob 1−σ)0.000

mean = true gradient0.210

Var(b)0.0189

Variance comparison

Var at b = 00.0189

Var at b = V0.0336

Var at b = b*0.0000

mean (true gradient)

possible estimates

value baseline V

optimal baseline b*

The mean never moves. Slide b anywhere: the two possible estimates shift and their spread changes, but the probability-weighted mean stays pinned to the true gradient. The score has mean zero, so any constant subtracts for free.

There is a zero-variance point. At b = b* the two possible estimates merge onto the mean: a single sample returns the exact gradient. Two outcomes and one adjustable constant make that possible only in the bandit, but the parabola is general.

The value baseline is not the optimal one. V weights rewards by their probabilities; b* flips the weights. They agree only at σ = ½. Push θ toward saturation and watch V drift away from b* — sometimes past the point where it beats no baseline at all.

From GRPO, or What the Critic Was For, part 5 of a series on trust regions.