Full Recipes from Open Labs

Theory chapters give you parts; this one gives you assembled machines. Five published pipelines, end to end — what each lab actually ran, why they chose what they chose, and the patterns that repeat once you see them side by side.

Llama 3 (Meta, July 2024)

Shape: iterative rounds of (RM → rejection sampling → SFT → DPO), ×6.

1. Train a Bradley–Terry reward model on human preference pairs (used for data filtering, not PPO).
2. Rejection sampling: sample K = 10–30 candidates per prompt from the best current checkpoints; keep the RM's favorites.
3. SFT on that harvest plus curated human/synthetic data across capability areas (code, math, multilingual, tool use, long context, safety).
4. DPO on fresh preference pairs — sampled on-policy from recent checkpoints — at lr 1e-5 with β = 0.1, plus two practical tweaks: mask special formatting tokens (headers, turn-ends) out of the loss, and add an auxiliary NLL loss on the chosen response with coefficient 0.2 to hold its likelihood up (chapter 6's displacement issue, patched in production).
5. Loop the whole thing six times, refreshing preference data as the policy improves; model averaging (weight merging) across runs along the way.

Why no PPO? The paper says it plainly: DPO + rejection sampling was more stable, less compute-hungry, and easier to scale to 405B. A 2024-era answer — and the industry's last major flag for the "you may not need online RL" position before the reasoning era mooted it.

Tulu 3 (Ai2, November 2024)

Shape: SFT → on-policy DPO → RLVR. The most reproducible recipe in existence — data, code, and ablations all public.

1. SFT: ~939k prompts, skills-organized (chapter 3), heavily decontaminated.
2. DPO: ~270k on-policy pairs (generate from the SFT model, judge with GPT-4o), length-normalized DPO after ablating against PPO, SimPO, and vanilla DPO.
3. RLVR: PPO-style training against pure verifier rewards on GSM8K/MATH/IFEval — the stage that named the technique. Modest gains at 8B–70B, but proof that verifier-only RL slots cleanly onto a preference-tuned model; scaled up in Tulu 3 405B (2025), where RLVR beat DPO-only convincingly.

The meta-contribution: every stage ships with its evaluation regime (unseen-suite discipline from chapter 11) and negative results — what didn't work is documented, which is rarer and more valuable than what did.

OLMo 2 (Ai2, late 2024 / 2025)

Shape: the Tulu 3 recipe, applied to a fully-open model. Same three stages (SFT → on-policy DPO → RLVR) on OLMo 2 7B/13B/32B — the paper reports "all of the stages in the Tülu 3 Recipe easily translate," and extends the final stage into multi-stage RLVR over a released mixed dataset (7,473 GSM8K + 7,500 MATH + 14,973 IFEval-style constraint prompts). This completes the only stack where pre-training data, post-training data (including the ~378k-pair preference mixes), code, and weights are all open. If you want to study post-training with nothing hidden, this is the artifact.

Qwen 2.5 → Qwen 3 (Alibaba, 2024 → 2025)

Shape (Qwen 2.5): SFT → "offline RL" (DPO) → "online RL" (GRPO). Over 1M SFT examples (2 epochs at 32k context); then ~150k DPO pairs built without human annotators — resample the SFT model, run automated checks (code execution, answer matching), and use pass/fail as chosen/rejected — explicitly targeting skills a reward model judges poorly (reasoning, factuality); then GRPO with 8 samples per query against a reward model, prioritizing queries whose sampled responses have high score variance (a mild form of chapter 7's dynamic-sampling idea, a year early). Note the naming: Qwen's "offline RL" stage is DPO — chapter 5's offline/online framing, used as a section heading in a real tech report.

Shape (Qwen 3): a four-stage reasoning-first pipeline, GSPO-era.

1. Long-CoT cold start — SFT on curated reasoning traces.
2. Reasoning RL — large-scale RL with verifiable rewards (GRPO-lineage; the Qwen team's GSPO paper — chapter 7 — describes the algorithm built for these runs, motivated by MoE stability).
3. Thinking-mode fusion — SFT that merges "thinking" and "non-thinking" behavior into one model, with a user-controllable thinking budget (chapter 8's length control, productized).
4. General RL — broad RLHF over 20+ task domains with reward models, for instruction following, agent/tool use, and format robustness.

Notable: the small Qwen 3 models are distilled from the flagship (strong-to-weak), echoing R1's distillation asymmetry — RL discovers at scale, SFT copies down.

DeepSeek V3 → R1 (2024 → January 2025)

Shape: covered in full in chapter 8 — cold-start SFT → reasoning GRPO → rejection-sample 800k → SFT → all-scenarios RL; R1-Zero as the pure-RL ablation; distillation to 1.5B–70B. Included here to complete the pattern table below.

The patterns, once you line them up

Recurring moves worth internalizing:

• Everyone iterates. No pipeline is one pass; data is regenerated from the improving policy (chapter 3's rejection-sampling loop is the engine inside every recipe).
• On-policy preference data beat static datasets everywhere it was ablated.
• SFT after RL, not just before. R1 and Qwen 3 both harvest their RL checkpoints into SFT data — RL discovers, SFT consolidates and broadens.
• The 2024→2025 shift: the preference stage (DPO/RLHF) went from being the destination to being a stage; verifiable-reward RL became where the capability gains live; and one flagship's expensive RL now feeds a whole family via distillation.

A starter recipe for your own project

Scaled honestly to a small team with a single 8-GPU node and an open base model:

1. SFT on a strong public mix (e.g. Tulu 3's), subsetted to your domain, plus a few thousand examples of your task written or edited by you. Decontaminate against your evals. This alone is 80% of the value.
2. On-policy DPO: generate 4–8 responses per prompt from your SFT model, judge with the best model you can afford (position-randomized, chapter 4), train length-normalized DPO. Cheap, stable, reliable lift.
3. RLVR only if you have a verifier: if your task has checkable outcomes (tests, schemas, exact answers), run GRPO/RLOO with DAPO's tricks via TRL, verl, or Unsloth (chapter 10). If it doesn't, stop at DPO — a judge-rewarded RL loop is where small teams drown in reward hacking.
4. Eval battery from chapter 11 before and after every stage, plus twenty transcripts read by a human. Ship the checkpoint that wins the battery, not the one with the best training curve.

References

1. Grattafiori et al., "The Llama 3 Herd of Models", 2024 — §post-training.
2. Lambert et al., "Tülu 3: Pushing Frontiers in Open Language Model Post-Training", 2024.
3. OLMo Team, "2 OLMo 2 Furious", 2024/25.
4. Qwen Team, "Qwen3 Technical Report", 2025; "Qwen2.5 Technical Report", 2024.
5. DeepSeek-AI, DeepSeek-R1, 2025; DeepSeek-V3, 2024.