LLM Practice 2: Post-training and Generation

Answer: the model never learned to emit the end-of-turn token, or inference doesn't stop on it

1. Check the training data: is the end-of-turn token in every assistant turn, and is it inside the loss mask (it must be trained)?
2. Check inference: same chat template as training, and the end-of-turn token set as a stop token.
3. Writing the user's next message is the giveaway: the model is continuing the conversation format because nothing taught it to stop.

Follow-up: Why mask the user turns at all? Otherwise most of the loss is spent learning to imitate customers, and the model learns to write user messages.

Answer: temperature 0 (greedy) plus constrained / structured decoding against a JSON schema

1. Extraction has one right answer, so sampling only adds variance: use temperature 0.
2. Constrained decoding masks any token that would break the schema, so every output parses.
3. Keep a small labelled set to measure field-level accuracy after the change.

Follow-up: Is temperature 0 perfectly deterministic in production? Not always: batching and floating-point order can still change near-ties.

Answer: reward hacking: the policy is exploiting the reward model's length preference

1. The reward model is a proxy; optimising it hard finds its blind spots (here, longer answers score higher regardless of quality).
2. Stop at an earlier checkpoint chosen by human or held-out evaluation, not RM score.
3. Raise the KL penalty, add a length penalty or length-normalised reward, retrain the RM on fresh comparisons that include long-but-worse answers, and report length-controlled win rates.

Follow-up: How would you spot this automatically? Track response length and RM score against a small human-rated set every few hundred steps; diverging curves are the alarm.

Answer: margin = 0.1 × (2 − (−3)) = 0.5; loss = −ln σ(0.5) ≈ 0.474; gradient weight σ(−0.5) ≈ 0.38

1. Implicit rewards: 0.1 × 2 = 0.2 (chosen) and 0.1 × (−3) = −0.3 (rejected); margin 0.5.
2. Loss = −ln σ(0.5) = ln(1 + e^(−0.5)) ≈ 0.474 (it starts at ln 2 ≈ 0.693).
3. The gradient is scaled by σ(−margin) ≈ 0.38: the pair is partly learned and still contributes.

Follow-up: If both log-ratios drop (chosen −4, rejected −9), what happens to the loss? Same margin, same loss: DPO only cares about the difference, so both answers can become less likely (likelihood displacement).

Answer: correct answer +1.73, each wrong answer −0.58; all-correct groups give 0 advantage and no learning signal

1. Mean = 0.25; std = √(0.25 × 0.75) ≈ 0.433.
2. Correct: (1 − 0.25) / 0.433 ≈ +1.73. Wrong: (0 − 0.25) / 0.433 ≈ −0.58.
3. All four correct (or all wrong): every reward equals the mean, so every advantage is 0: filter such prompts or resample (as DAPO does).

Follow-up: Why no value model? The group mean is the baseline, saving a whole network's memory and training.

Answer: ground answers in the policy documents, allow and reward abstention, verify, and measure

1. Retrieve the relevant policy sections (RAG) and instruct the model to answer only from them, citing the section; escalate to a human if nothing relevant is found.
2. Teach abstention: SFT and preference data with "I'm not sure, let me check with an agent" examples; score abstentions above wrong answers in evals.
3. Use tools for anything computable (refund amounts, order status) instead of free-text generation.
4. Add a check: verify each policy claim in the draft against the retrieved text (NLI model or LLM judge) before sending.
5. Measure hallucination, citation accuracy and escalation rate on a labelled set of real questions; review samples weekly.

Follow-up: Why not just lower the temperature? If the model doesn't know the policy, its most likely answer is still a confident invention; grounding and abstention fix the cause.

Answer: not yet: the setup has position bias, and even without it the margin is about ±5.5 points

1. The new answer was always first, so position bias alone could produce a >50% win rate. Re-run with both orders and count only consistent verdicts (inconsistent = tie).
2. Margin of error at n = 300, p = 0.62: 1.96 × √(0.62 × 0.38 / 300) ≈ 0.055, so roughly 56–68% if the judge were unbiased.
3. Check length: if new answers are longer, report a length-controlled win rate.
4. Validate the judge on ~200 human-labelled pairs from your domain and use a different model family from the one being tested.

Follow-up: What result would convince you? An order-balanced, length-controlled win rate whose interval stays above 50%, from a judge with good agreement with human labels, plus no regression on policy-accuracy checks.