Twenty-eight papers and reports that explain how OpenAI thinks about scaling, post-training, reasoning, evaluation and safety, with what to take from each. Five are marked read first: together they cover the questions an OpenAI interviewer is most likely to take three levels deep.
Note
Links were checked in September 2026. All arXiv items are OpenAI-authored or co-authored. System cards for the newest models appear on openai.com first; check the model release notes for anything newer than this list.
For each paper, write four lines in journal/: the claim, the key figure or equation, the number you would quote, and one experiment you could run on 8 GB to test it. Module 00 describes the loop. If a "what to extract" line asks you to derive something, derive it on paper before you read the derivation.
Mark
Meaning
Read first
Core to the interview; know it cold
(8 GB)
Has a small-scale experiment you can run; see projects.md
WebText (outbound Reddit links with at least 3 karma) as a quality filter; byte-level BPE that blocks merges across character categories, the ancestor of the pre-tokenizer you build in lab 04; zero-shot task transfer as a function of model size.
Loss as power laws in non-embedding parameters \(N\), data \(D\) and compute \(C\) (\(\alpha_N \approx 0.076\), \(\alpha_D \approx 0.095\)). The compute-optimal allocation they found (\(N \propto C^{0.73}\)) and why Chinchilla later found roughly \(N \propto C^{0.5}\) (learning-rate schedules not matched to run length). Be able to fit one yourself: lab 08. (8 GB)
In-context learning improving with scale; the 175B model trained on about 300B tokens; the data-contamination analysis (n-gram overlap between training data and benchmarks), which is still how many contamination checks work.
The "predictable scaling" section: final loss and a HumanEval metric predicted from much smaller runs before training. That is the pretraining team's core discipline. Also note what the report does not disclose, and why.
The Bradley–Terry reward model over pairs of trajectory segments, \(P(a \succ b) = \sigma(r_a - r_b)\); asking humans about under 1% of the agent's interactions. Every RLHF system since descends from it.
The clipped surrogate objective and when its gradient is exactly zero (advantage positive, ratio above \(1+\epsilon\)). You implement this in lab 12; GRPO is PPO without a value network.
The full recipe at small scale: SFT, a reward model from comparisons, PPO with a KL penalty to the SFT policy. The figure where optimizing the reward model too hard makes true quality fall.
The three steps (SFT, RM, PPO); why a 1.3B InstructGPT was preferred over 175B GPT-3; the "alignment tax" and the PPO-ptx fix (mixing in pretraining gradients); how comparisons from one prompt are batched in the RM loss to avoid overfitting. Derive the RM loss \(-\log \sigma(r(x,y_w) - r(x,y_l))\) and connect it to DPO in lab 11.
The unbiased pass@k estimator \(1 - \binom{n-c}{k} / \binom{n}{k}\) and why \(1-(1-\hat p)^k\) is biased; HumanEval's 164 hand-written problems; Codex solved 28.8% with one sample and 70.2% with 100 samples per problem, the first clear evidence that sampling plus a verifier multiplies capability. Implemented in lab 15. (8 GB)
Process supervision (a reward per reasoning step) beats outcome supervision for best-of-\(N\) on MATH; the process reward model solves about 78% of a representative MATH test subset; PRM800K (800K step labels) is public; active learning to pick convincing wrong answers. Know the trade-off: process labels cost more but give denser credit assignment. (8 GB)
Performance rising smoothly with both RL training compute and test-time thinking compute. From the system card: how a reasoning model is evaluated before launch (jailbreaks, hallucination, CoT deception monitoring, Preparedness categories).
A specialized system with hand-engineered test-time strategies (o1-ioi) versus a general model (o3) that reached IOI gold-medal level without them. The lesson OpenAI draws: scale general RL rather than hand-built inference pipelines.
A unified system with a router between a fast model and a thinking model; "safe-completions" instead of hard refusals; how sycophancy, deception and hallucination are measured; precautionary High-capability treatment in biology.
4,326 short questions collected adversarially against GPT-4, graded correct / incorrect / not attempted; calibration measured from stated confidence and answer frequency. Small enough to evaluate a local model with CIs. (8 GB)
Three agentic benchmarks from the Preparedness side: 75 Kaggle competitions; replicating 20 ICML 2024 papers graded by author-agreed rubrics; 1,400+ freelance tasks worth $1M in payouts. Extract how each handles grading, contamination and cost.
1,266 questions that are hard to find but easy to verify: the asymmetry that makes browsing agents measurable. Useful template for your own agent evals.
5,000 multi-turn conversations graded against physician-written rubrics; rubric-based grading with a model grader and the checks on its agreement with physicians (compare Cohen's kappa in lab 15).
Tasks from 44 occupations, graded by blinded expert comparison with human deliverables. Shows where evaluation is heading: economic value, not exam questions.
Pretraining hallucination as a consequence of classification error; post-training evaluations that give zero credit for "I don't know" reward guessing. The fix: state a confidence threshold in the task and penalize wrong answers accordingly. A good interview topic that joins statistics and incentives. (8 GB)
The superalignment analogy: a small model supervises a large one. Performance gap recovered, \(\mathrm{PGR} = \frac{\text{weak-to-strong} - \text{weak}}{\text{strong ceiling} - \text{weak}}\); the auxiliary confidence loss; why reward modeling recovers less than NLP tasks. Code (archived). (8 GB)
Training a reasoning model to recall and reason over the written safety spec in its chain of thought, without human-written CoTs. The follow-up (with Apollo Research) measures covert actions and shows large reductions that are partly confounded by the model recognizing it is being evaluated.
A weaker model reading a frontier reasoning model's chain of thought catches reward hacking in coding tasks far better than one watching actions alone. Penalizing "bad thoughts" during RL makes the agent keep hacking while hiding intent. The "monitorability tax" argument follows. Reproducible in miniature: projects.md project 3. (8 GB)
The multi-lab position paper on why CoT is monitorable (hard tasks force externalized reasoning) and what would break it (outcome-only RL pressure, CoT supervision, latent-space reasoning architectures). The follow-up gives three evaluation archetypes and a metric; longer CoTs tend to be more monitorable.
Narrow fine-tuning on bad data (insecure code, bad advice) causes broad misalignment; model diffing with SAEs finds a "toxic persona" latent that predicts and steers it; a small amount of benign fine-tuning restores behavior. Connects interpretability to a training-time safety problem.
TopK SAEs set the sparsity \(L_0 = k\) directly (no L1 penalty, no shrinkage); clean scaling laws in latents and \(k\); dead-latent fixes (encoder initialized as decoder transpose, an auxiliary loss); evaluation beyond MSE (downstream loss, probes, ablation sparsity). The 2025 paper trains models whose weights are mostly zero so circuits are small by construction. Extend the ReLU SAE from lab 16 to TopK. (8 GB)
Mixture-of-experts with about 5.1B (120b) and 3.6B (20b) active parameters; MXFP4 expert weights so 120b fits one 80 GB GPU and 20b about 16 GB; the harmony response format; adjustable reasoning effort; the adversarial fine-tuning study used to judge open-weight release risk. Connect MXFP4 to lab 13.