AIML Rounds RoadmapMocksJobs
Concept20 minBefore this:SFT and Instruction Tuning

Policy Gradients for LLMs

How can a reward that isn't differentiable train an LLM, and why do you need a baseline?

The situation

Your model writes SQL, and you can check each query by running it. But pass or fail isn't differentiable, so you can't simply backpropagate through it. You also notice training gets very noisy after someone adds a bonus to every reward. The interviewer asks:

How can a reward that isn't differentiable train an LLM, and why do you need a baseline?

Why it matters

The RLHF and reasoning model lessons use PPO and GRPO. This lesson explains what they're doing underneath. Interviewers like to ask why you need reinforcement learning at all, and why GRPO doesn't need a critic.

The idea

Think of a coach who can't see a player's technique, only the score of each match. After a good match, the coach says to do more of what you did. After a bad one, to do less. If the coach praises every match, because every score is above zero, the player learns nothing. What helps is praise relative to the usual score.

Policy gradientWeight log-probsby rewardNo reward gradientBaselineSubtract theusual rewardLess noiseClippingLimit how farthe policy movesReuse samples
PPO learns its baseline. GRPO uses the group's mean.

Play

Try to break it. Choose no baseline, with 10 added to every reward. Then try the group mean.

Train with a baseline

Same reward, less noise
chance of the right answer at the end
–
runs stuck on a wrong answer
–
gradient noise at the start
–

Simulation with one prompt and four possible answers, one right. Reward: 1 if right, 0 if wrong, plus the added amount. 150 updates at a learning rate of 0.5, averaged over 40 fixed runs. Grey shows no baseline. A run counts as stuck if it ends with over 90% on a wrong answer.

Rewards of 11 or 10 carry the same information as 1 or 0. But with 8 answers per update, and no baseline, the right answer ends at only 50%, and 19 of 40 runs get stuck on a wrong one. Everything gets pushed up, so whichever answer happens to be sampled first runs away. The group mean removes the added 10 entirely: 99%, with none stuck, and about 20 times less gradient noise (0.08 against 1.68). Now set the added amount to 0: no baseline works fine, because a reward of 0 pushes nothing. Bigger groups also help: at G = 64, even no baseline reaches 99%, though each update then costs 64 answers.

Now say why: why does adding a constant hurt when there's no baseline?

Maths

In one breath

RLHF and GRPO both train a model from a score it gets after answering. That score comes from a reward model or a checker, so you can't backpropagate through it. The policy gradient gets around this: sample an answer, then raise the probability of its tokens in proportion to its reward. Good answers get more likely, and bad ones less. Alone, this is very noisy, so we subtract a baseline, an estimate of the usual reward. PPO learns the baseline with a value model, and clips each update. GRPO uses the average reward of the group as its baseline instead.

Text as a game

RL wordIn an LLM
PolicyThe model's next-token distribution
StateThe prompt and the tokens so far
ActionThe next token
EpisodeOne whole answer
RewardUsually one number at the end, from a reward model or a checker

Why not use ordinary training?

SFT needs the right answer, token by token. Here we only have a score for an answer the model wrote itself. We can't differentiate through "did the tests pass?", and sampling a token isn't differentiable either. The policy gradient needs neither: it only uses the gradient of the model's own log-probabilities.

Why a baseline?

Say every answer scores between 10 and 11. Without a baseline, every answer gets pushed up, the good ones only slightly more than the bad, and the signal is lost in noise. Subtracting the usual reward fixes this: above-average answers go up, and below-average ones go down. On average the gradient is unchanged, because the baseline term has an expected value of zero. Only the noise falls.

Three algorithms

REINFORCEPPOGRPO
BaselineNone, or a simple oneLearned value modelMean of the group's rewards
Extra modelsNoneValue model, and often reward and referenceReference only
Reusing samplesOne updateSeveral updates, kept safe by clippingClipped objective. DeepSeekMath used one update

Why PPO clips

Samples come from the old policy. As the policy changes, those samples describe it less well. PPO measures this with the ratio r of new to old probability, and stops giving credit once r leaves a small range, usually 1 ± 0.1 to 0.3. That makes it safe to take several gradient steps on one batch.

What InstructGPT and GRPO did

InstructGPT used PPO, with the value model initialised from the reward model. GRPO keeps PPO's clipped form, but drops the value model. DeepSeekMath sampled 64 answers per question, used a KL coefficient of 0.04, and made one update after each sampling round.

The maths

    The policy gradient with a baseline and the PPO clipped objective, in the form given in OpenAI’s Spinning Up. GRPO replaces A with the group-normalised reward.

    In practice

    In practice

    • One reward, many tokens: each token in an answer gets the answer's advantage, whether it helped or not. That's a noisy signal, so large groups help.
    • Keep it on-policy: samples must come from a recent policy. Fresh samples are expensive, as generation is slow, so RL training spends much of its time generating.
    • Keep it close: a KL term to the reference stops the policy drifting into odd text, which reward hacking exploits.
    • Watch for no signal: if every answer in a group gets the same reward, the advantages are all 0. Choose problems the model sometimes solves.
    • Check the reward first: the algorithm will find any weakness in it. Test the reward on known good and bad answers before a long run.

    The answer that sounds right

    "A positive reward means the model is doing well, so it should be reinforced"

    It sounds right, as positive sounds good. But only reward relative to the usual level carries information. If every answer scores about 10, a reward of 10 is merely average, and reinforcing it adds noise. The advantage, reward minus baseline, is what tells the model which answers were better.

    Say it

    How can a reward that isn't differentiable train an LLM, and why do you need a baseline?

    Say your answer out loud

    Aim for 30 seconds. It feels different from reading it, and that's the point.

    Tick what you covered

    The interviewer pushes further

    Follow-up 1 of 3

    Why doesn't subtracting a baseline bias the gradient?

    Answer it out loud, then check.

    The expected value of the gradient of log π, given the state, is zero. So a baseline that depends only on the state multiplies something with zero mean, and adds nothing on average. It only changes the variance.

    Follow-up 2 of 3

    A prompt gets rewards of 11, 10, 10 and 10. What are the weights, with and without the group mean?

    Answer it out loud, then check.

    Without a baseline, all four are positive, so every answer gets pushed up. With the group mean of 10.25, the good answer gets +0.75 and the others get −0.25, which is the signal we want.

    Follow-up 3 of 3

    Why can PPO reuse a batch, when plain policy gradient can't?

    Answer it out loud, then check.

    The plain gradient only holds for the policy that produced the samples. PPO corrects for the change with a probability ratio, and clips it so that the policy can't gain by moving far from where the samples are reliable.

    We use reinforcement learning when we only get a score for a whole answer, such as from a reward model or a test suite, and can't backpropagate through it. The policy gradient samples an answer and raises its tokens' log-probabilities in proportion to its reward, so the reward itself never needs a gradient. That's noisy, so we subtract a baseline. It doesn't change the expected gradient, but above-average answers go up and below-average ones go down. PPO learns the baseline with a value model, and clips the probability ratio so it can safely reuse samples. GRPO keeps the clipping, but uses the group's mean reward as the baseline, so it doesn't need a critic.

    Key terms and sources

    How reinforcement learning maps onto text, the policy gradient, why baselines reduce noise, PPO's clipping, value models, and how GRPO replaces them.

    Sources

    Explanations are ours, built from these sources. Lab numbers are illustrative, and worked examples were checked by calculation.