Library: visual guides and definitive papers¶
A curated reading library for the hardest topics in modern AI: 19 topics, 90 items, 2 to 5 per topic. For each topic you get the best visual explanation, the definitive paper, and course slides or notes where they add something, each with a reason it made the cut and the exact sections worth your time.
Everything is also listed in machine-readable form in manifest.json; 59 items have a PDF that tools/fetch_library.py can download for offline study.
Note
This library complements curriculum/papers.md, which lists many papers per module. Here the goal is the opposite: the few items per topic that give the most understanding per hour, and which pages to read. Links were last reviewed in September 2026. If one breaks, the arXiv ID or title will find it.
Contents¶
- How to use this library
- Start here: a 10-item path
- Topics:
- Attention and transformers
- RoPE and context extension
- Tokenization
- GPU architecture, CUDA and Triton
- FlashAttention
- KV cache, PagedAttention and serving
- Scaling laws
- Distributed training (DP, FSDP/ZeRO, TP, PP)
- Mixed precision and FP8
- Optimizers (Adam, AdamW, Muon)
- RLHF, DPO and GRPO
- Reasoning and test-time compute
- Quantization
- Speculative decoding
- Mixture of experts
- State-space models (Mamba)
- Mechanistic interpretability
- Evaluation and statistics
- RAG and agents
- How to download the PDFs
- Licensing and fair use
- Adding an item
How to use this library¶
- Picture first. Read the visual explainer (article or video) before the paper; the paper then reads much faster.
- Read only the focus sections. The Focus on column names the sections, figures or algorithms that carry the idea. Read those closely and skim the rest.
- Close the loop in a lab. Each topic links the lab or curriculum module that uses it. Implement the mechanism and predict a number before you measure it.
- Write one paragraph. After each item, write down the claim, the evidence and one thing you would test next (the format used in
papers.md).
Levels: intro needs only basic deep learning; intermediate assumes you have built a small transformer; advanced assumes the intermediate items in the same topic.
Start here: a 10-item path¶
If you read nothing else, read these in order. Together they cover the stack from the transformer to evaluation, and each one sets up a lab. At one to two evenings per item, this is about a month of reading.
| # | Item | Why now | Then do |
|---|---|---|---|
| 1 | The Illustrated Transformer (article, intro) | The mechanism everything else builds on. | Lab 05 |
| 2 | Let's build the GPT Tokenizer (video, intro) | Tokenization explains many model quirks; you will build one next. | Lab 04 |
| 3 | Making Deep Learning Go Brrrr From First Principles (article, intro) | The performance model you need before any kernel or systems work. | Lab 03 |
| 4 | Transformer Inference Arithmetic (article, intro) | Turns that model into numbers for LLM inference. | Lab 03, then lab 07 |
| 5 | FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (paper, advanced) | The best example of IO-aware algorithm design. Read the online-softmax paper alongside it. | Lab 06 |
| 6 | Training Compute-Optimal Large Language Models (paper, intermediate) | How to size a model for a compute budget. | Lab 08 |
| 7 | The Ultra-Scale Playbook: Training LLMs on GPU Clusters (article, intermediate) | How training is spread across many GPUs, and what each scheme costs. | Lab 09 |
| 8 | Direct Preference Optimization: Your Language Model is Secretly a Reward Model (paper, intermediate) | Preference optimisation from a derivation you can check by hand. | Lab 11 |
| 9 | DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (paper, intermediate) | How reasoning models are trained with RL on verifiable rewards. | Lab 12 |
| 10 | Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations (paper, intermediate) | How to tell whether a result is real. | Lab 15 |
Tip
If you are aiming at interpretability, swap item 10 for A Mathematical Framework for Transformer Circuits and do lab 16. If you are aiming at inference, add PagedAttention and speculative decoding after item 5.
Attention and transformers¶
Get the mechanism from pictures first, then the specification, then what changed after 2017. Pairs with lab 05, module 04.
| Item | Type | Why it's the best | Focus on | Level |
|---|---|---|---|---|
| The Illustrated Transformer Jay Alammar, 2018 |
article | The most widely used visual introduction. It builds attention one word-vector at a time before showing the matrix form, which is the order intuition needs. | The self-attention walkthrough (query, key and value vectors for one word, then the matrix form) and multi-head attention. Skim the encoder-decoder parts. | intro |
| CS224N note: Self-Attention and Transformers (draft) (PDF) John Hewitt (Stanford CS224N), 2023 |
notes | Short, rigorous notes that introduce each transformer component as the fix for a concrete failure of bare self-attention, instead of presenting the architecture as given. | The sequence of fixes: position representations, element-wise nonlinearities, causal masking. Then the full block with residuals and layer norm. | intermediate |
| Attention Is All You Need · PDF Vaswani et al., 2017 |
paper | The primary source, and still the cleanest specification of the architecture. | Section 3.2 (scaled dot-product and multi-head attention, and the footnote on why scores are divided by sqrt(d_k)), Figure 2, and Table 1 (per-layer cost vs recurrence and convolution). | intro |
| CS336 lecture 3: architectures and hyperparameters · PDF Percy Liang, Tatsunori Hashimoto (Stanford CS336), 2025 |
slides | Covers what the 2017 paper cannot: the design choices modern LLMs converged on, argued from many published models. | Pre-norm vs post-norm, RMSNorm, gated MLPs, RoPE, attention variants (GQA/MQA), and the hyperparameter ratios that recur across models. | intermediate |
| The Transformer Family Version 2.0 Lilian Weng, 2023 |
article | The best single index of transformer variants, each with its core equation and citation. A reference to return to, not a first read. | Positional encodings, long-context and efficient-attention variants. Read the section you need when a paper assumes it. | advanced |
3Blue1Brown's neural-network video series (the chapters on transformers and attention) is the best first visual intuition if you have not seen attention before.
RoPE and context extension¶
RoPE is a rotation of queries and keys; context extension is mostly about which rotation frequencies you rescale. Pairs with lab 05, module 04.
| Item | Type | Why it's the best | Focus on | Level |
|---|---|---|---|---|
| You could have designed state of the art positional encoding Christopher Fleetwood (Hugging Face), 2024 |
article | Derives RoPE step by step from the properties a positional encoding should have, with plots at every step. The best intuition-first route. | The progression from integer positions to sinusoids to rotations, and why a relative encoding should act on queries and keys rather than on the embeddings. | intro |
| Rotary Embeddings: A Relative Revolution EleutherAI, 2021 |
article | Written by EleutherAI, who adopted RoPE early (GPT-J, GPT-NeoX); a short, correct derivation with the complex-number view and early experiments. | The derivation that rotating q and k makes their dot product depend only on relative position, and the implementation notes. | intermediate |
| RoFormer: Enhanced Transformer with Rotary Position Embedding · PDF Su et al., 2021 |
paper | The primary source for RoPE. | Section 3.2 (the 2D case, then the general block-diagonal rotation matrix) and the long-term-decay property of the rotated inner product. | intermediate |
| Extending Context Window of Large Language Models via Position Interpolation · PDF Chen et al., 2023 |
paper | The first simple, effective way to extend a trained RoPE model's context, with the clearest explanation of why naive extrapolation fails. | The bound showing attention scores can blow up at unseen positions, and why squeezing positions into the trained range plus a short fine-tune works. | advanced |
| YaRN: Efficient Context Window Extension of Large Language Models · PDF Peng et al., 2023 |
paper | The context-extension method adopted by several open model families, including DeepSeek-V3; its per-frequency view underlies later methods. | The frequency-dependent view: interpolate low-frequency dimensions, leave high-frequency ones alone (NTK-by-parts), plus the attention temperature correction. | advanced |
Tokenization¶
Many strange model failures are tokenization bugs; build a tokenizer once and you will recognise them. Pairs with lab 04, module 04.
| Item | Type | Why it's the best | Focus on | Level |
|---|---|---|---|---|
| Let's build the GPT Tokenizer Andrej Karpathy, 2024 |
video | Builds a working byte-level BPE tokenizer and then traces real model failures back to tokenization. Nothing else covers both the algorithm and its consequences as well. | The BPE training loop, GPT-2/GPT-4 regex pre-tokenization, special tokens, and the closing list of tokenization-caused failures (spelling, arithmetic, non-English text, trailing spaces). | intro |
| Neural Machine Translation of Rare Words with Subword Units · PDF Sennrich, Haddow, Birch, 2015 |
paper | The paper that brought BPE to neural NLP. The whole merge-learning algorithm fits in about a dozen lines. | Section 3.2 and its Python listing (Algorithm 1). Implement it before reading anything else about BPE. | intro |
| Language Model Tokenizers Introduce Unfairness Between Languages · PDF Petrov et al., 2023 |
paper | Quantifies how much more the same content costs in some languages. Essential if you build for users who do not write English. | The tokenization-premium measurements across languages and their consequences: higher cost, higher latency and less usable context. | intermediate |
Companion code for the video: karpathy/minbpe.
GPU architecture, CUDA and Triton¶
Performance work starts with knowing whether you are limited by compute, memory bandwidth or overhead. Pairs with lab 06, lab 03, module 02.
| Item | Type | Why it's the best | Focus on | Level |
|---|---|---|---|---|
| Making Deep Learning Go Brrrr From First Principles Horace He, 2022 |
article | The best mental model of GPU performance in one article, with a way to diagnose which regime you are in. | The three regimes (compute-bound, memory-bandwidth-bound, overhead-bound), operator fusion, and how to tell which regime a workload is in. | intro |
| How to Think About GPUs (How to Scale Your Model) Jacob Austin et al. (Google DeepMind), 2025 |
article | Quantitative: every hardware feature becomes a number you can plug into a roofline estimate, with GPUs and TPUs compared side by side. | SMs, tensor cores, the memory hierarchy and NVLink/InfiniBand networking, each expressed as numbers for roofline estimates. | intermediate |
| CS336 lecture 5: GPUs · PDF Percy Liang, Tatsunori Hashimoto (Stanford CS336), 2025 |
slides | University lecture slides that tie GPU hardware directly to LLM workloads. | The execution model, the memory hierarchy, and why tiling, fusion and matmul shapes decide performance. | intermediate |
| How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance: a Worklog Simon Boehm, 2022 |
article | Shows with measurements which optimisations make a matmul kernel fast and why. The best bridge from CUDA basics to real kernels. | The kernel-by-kernel progression (coalescing, shared-memory tiling, 1D and 2D block tiling, vectorised loads, warp tiling) and the throughput gained at each step. | intermediate |
| Triton: an intermediate language and compiler for tiled neural network computations Tillet, Kung, Cox, 2019 |
paper | The primary source for the tile-based programming model used in lab 06. | The core idea: program at the level of tiles and let the compiler handle scheduling and memory coalescing inside a tile. Then work through the official Triton tutorials (vector add, fused softmax, matmul). | advanced |
Also useful: the Modal GPU Glossary (CC BY 4.0) for decoding terms, and the rooflines chapter of How to Scale Your Model as the prerequisite for every estimate.
FlashAttention¶
Online softmax makes tiled attention exact; tiling makes it IO-efficient. Read in this order. Pairs with lab 06, module 07.
| Item | Type | Why it's the best | Focus on | Level |
|---|---|---|---|---|
| Online normalizer calculation for softmax · PDF Milakov, Gimelshein, 2018 |
paper | The key algorithmic ingredient of FlashAttention, isolated in a short paper. | The one-pass algorithm that keeps a running maximum and rescales the running sum. Prove to yourself that it is exact. | intermediate |
| FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness · PDF Dao et al., 2022 |
paper | The primary source, and a model of analysing an algorithm by memory traffic instead of FLOPs. | Figure 1 (tiling across HBM and SRAM), Algorithm 1, and the IO-complexity analysis in Section 3.2. Then the backward pass with recomputation in the appendix. | advanced |
| FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning · PDF Tri Dao, 2023 |
paper | Explains the engineering that turned the idea into the default attention kernel. | Section 3: fewer non-matmul FLOPs, parallelising over sequence length, and splitting work across warps to cut shared-memory traffic. | advanced |
| FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision · PDF Shah et al., 2024 |
paper | Shows how new hardware features (asynchronous copies, FP8 tensor cores) change kernel design. | Warp specialisation with asynchronous loads, overlapping softmax with the matmuls, and FP8 with incoherent processing to limit quantization error. | advanced |
KV cache, PagedAttention and serving¶
Decoding is usually memory-bandwidth-bound; serving systems are mostly about managing KV cache memory and batching. Pairs with lab 07, module 07.
| Item | Type | Why it's the best | Focus on | Level |
|---|---|---|---|---|
| Transformer Inference Arithmetic kipply, 2022 |
article | Short, numeric and practical: teaches you to estimate inference memory and latency on paper. | KV cache bytes per token, why small-batch decoding is memory-bandwidth-bound, and the batch size where it turns compute-bound. Redo each number for Llama 3 8B. | intro |
| Continuous batching from first principles Hugging Face, 2025 |
article | Builds serving concepts in the right order with clear diagrams. | The sections on the KV cache, chunked prefill and continuous batching: how requests are packed into one batch and scheduled. | intro |
| All About Transformer Inference (How to Scale Your Model) Jacob Austin et al. (Google DeepMind), 2025 |
article | The most complete quantitative treatment of inference, from sampling one token to a sharded serving system. | Prefill vs decode rooflines, KV cache memory, the latency-throughput trade-off, and how to shard a model for serving. | intermediate |
| Efficient Memory Management for Large Language Model Serving with PagedAttention · PDF Kwon et al., 2023 |
paper | The primary source for PagedAttention, now standard in serving engines. Its memory-waste analysis explains why. | Section 3 (KV cache waste: reservation, internal and external fragmentation) and Section 4 (block tables; copy-on-write sharing for parallel sampling and beam search). | intermediate |
| Efficiently Scaling Transformer Inference · PDF Pope et al., 2022 |
paper | The paper behind most modern inference cost models; the scaling-book inference chapter builds on it. | The cost model for latency vs utilisation, the partitioning layouts for feedforward and attention layers, and why multiquery attention matters at long context. | advanced |
Going further: Lilian Weng's Large Transformer Model Inference Optimization.
Scaling laws¶
Learn to read, fit and distrust power laws; every model-sizing decision rests on them. Pairs with lab 08, module 05.
| Item | Type | Why it's the best | Focus on | Level |
|---|---|---|---|---|
| Scaling Laws for Neural Language Models · PDF Kaplan et al., 2020 |
paper | The paper that established loss as a smooth power law in parameters, data and compute. | The summary of results in Section 1 and Figure 1. Note which allocation conclusion Chinchilla later revised, and why (the learning-rate schedule was not matched to training length). | intermediate |
| Training Compute-Optimal Large Language Models · PDF Hoffmann, Borgeaud, Mensch et al., 2022 |
paper | The compute-optimal result every model-sizing decision still starts from. | Section 3: three estimation approaches (fixed model sizes, IsoFLOP profiles, a parametric loss fit L(N, D) = E + A/N^alpha + B/D^beta), and the result that parameters and tokens should grow in equal proportion with compute. | intermediate |
| Chinchilla Scaling: A replication attempt · PDF Besiroglu et al., 2024 |
paper | A worked lesson in fitting scaling laws carefully and checking published fits. | How refitting the parametric approach on data reconstructed from the paper's figure gives estimates consistent with the other two approaches, and the implausibly narrow original confidence intervals. | advanced |
| CS336 lecture 9: scaling laws basics · PDF Percy Liang, Tatsunori Hashimoto (Stanford CS336), 2025 |
slides | A course-level synthesis of the scaling-law literature with the practical questions it answers. | Data and model scaling, how to run small sweeps and extrapolate, and the pitfalls in fitting. | intermediate |
| CS336 lecture 11: scaling details · PDF Percy Liang, Tatsunori Hashimoto (Stanford CS336), 2025 |
slides | How labs actually use scaling experiments, which the papers rarely spell out. | Case studies from published model reports, and hyperparameter transfer across scale (muP, batch size and learning-rate scaling). | advanced |
Distributed training (DP, FSDP/ZeRO, TP, PP)¶
Each parallelism scheme trades memory for communication; learn to compute both. Pairs with lab 09, module 05.
| Item | Type | Why it's the best | Focus on | Level |
|---|---|---|---|---|
| The Ultra-Scale Playbook: Training LLMs on GPU Clusters Hugging Face (Nanotron team), 2025 |
article | The most complete visual treatment of LLM training parallelism, with interactive memory calculators and results measured on real clusters. | The memory breakdown (weights, gradients, optimizer states, activations), then DP, ZeRO stages 1-3, TP, sequence and context parallelism, pipeline schedules and expert parallelism. | intermediate |
| How to Parallelize a Transformer for Training (How to Scale Your Model) Jacob Austin et al. (Google DeepMind), 2025 |
article | Derives, for each scheme, exactly when communication becomes the bottleneck. The analytical complement to the playbook. | For DP, FSDP, TP and PP: the communication cost per step and the batch size or model size at which each becomes communication-bound. | advanced |
| CS336 lecture 7: parallelism basics · PDF Percy Liang, Tatsunori Hashimoto (Stanford CS336), 2025 |
slides | Compact lecture slides that cover the whole design space in one sitting. | Collective operations and their costs, then data, tensor and pipeline parallelism with the memory and communication each saves or adds. | intermediate |
| ZeRO: Memory Optimizations Toward Training Trillion Parameter Models · PDF Rajbhandari et al., 2019 |
paper | The primary source for sharded data parallelism (the idea behind FSDP) and the clearest memory accounting for mixed-precision Adam. | Figure 1 and Section 5: the 2 + 2 + 12 bytes-per-parameter accounting, and how stages 1, 2 and 3 partition optimizer states, gradients and parameters. | intermediate |
| Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism · PDF Shoeybi et al., 2019 |
paper | The primary source for tensor parallelism as used in practice. | Section 3 and Figure 3: column-parallel then row-parallel splits of the MLP and attention, so each block needs one all-reduce forward and one backward. | intermediate |
Going further: Lilian Weng's How to Train Really Large Models on Many GPUs? (survey), PTD-P (pipeline schedules at cluster scale) and PyTorch FSDP (the implementation).
Mixed precision and FP8¶
Low precision is a range and resolution budget; scaling is how you stay inside it. Pairs with module 02, module 03.
| Item | Type | Why it's the best | Focus on | Level |
|---|---|---|---|---|
| Mixed Precision Training · PDF Micikevicius, Narang, Alben et al., 2017 |
paper | The three techniques every half-precision training recipe still uses. | Section 3: an FP32 master copy of weights, loss scaling (and the gradient histogram that motivates it), and FP32 accumulation. | intro |
| FP8 Formats for Deep Learning · PDF Micikevicius, Stosic et al., 2022 |
paper | The specification of the two FP8 formats that current accelerators implement. | The E4M3 and E5M2 encodings, why E4M3 gives up infinities to extend its range, and which tensors use which format (weights and activations vs gradients). | intermediate |
| Microscaling Data Formats for Deep Learning · PDF Rouhani et al., 2023 |
paper | Block scaling is how FP8 and FP4 are made usable; this paper defines the MX formats. | The block structure (32 elements share one power-of-two scale) and the experiments on which element formats hold up for inference and for training. | advanced |
| DeepSeek-V3 Technical Report · PDF DeepSeek-AI, 2024 |
paper | The most detailed public account of FP8 training at frontier scale. | Section 3.3 (fine-grained tile- and block-wise scaling, and promoting partial sums to higher precision). Also Section 2 for multi-head latent attention and auxiliary-loss-free MoE load balancing. | advanced |
Optimizers (Adam, AdamW, Muon)¶
Know Adam's update line by line, what decoupled weight decay changes, and why Muon orthogonalises its updates. Pairs with lab 02, module 03.
| Item | Type | Why it's the best | Focus on | Level |
|---|---|---|---|---|
| Why Momentum Really Works Gabriel Goh, 2017 |
article | Interactive figures that make optimisation dynamics visible; the best intuition for step sizes, curvature and momentum. | The eigen-decomposition of the error on a quadratic, why the largest curvature limits the step size, and how momentum changes the convergence rate. | intermediate |
| Adam: A Method for Stochastic Optimization · PDF Kingma, Ba, 2014 |
paper | The primary source for the default optimizer. | Algorithm 1 and Section 3 (bias correction). Skip the convergence analysis, which was later shown to be flawed. | intro |
| Decoupled Weight Decay Regularization · PDF Loshchilov, Hutter, 2017 |
paper | Explains the one-line difference between Adam with L2 and AdamW, which matters in every LLM training run. | Algorithm 2 (Adam with L2 regularisation vs decoupled weight decay, differences highlighted) and why L2 is not weight decay under adaptive methods. | intro |
| Muon: An optimizer for hidden layers in neural networks Keller Jordan, 2024 |
article | The original description of Muon by its author, with the speedrun evidence that made it popular. | The update (momentum, then approximate orthogonalisation with Newton-Schulz iterations), which parameters use Muon and which stay on AdamW, and the empirical results. | intermediate |
| Muon is Scalable for LLM Training · PDF Liu, Su, Yao et al. (Moonshot AI), 2025 |
paper | Shows what Muon needs to work at LLM scale. | The two changes: add weight decay, and rescale each matrix's update to match AdamW's update RMS so AdamW hyperparameters transfer. | advanced |
RLHF, DPO and GRPO¶
Post-training is a KL-regularised optimisation against a learned or verifiable reward; the algorithms differ in how they estimate it. Pairs with lab 11, lab 12, module 06.
| Item | Type | Why it's the best | Focus on | Level |
|---|---|---|---|---|
| Illustrating Reinforcement Learning from Human Feedback (RLHF) Nathan Lambert et al., 2022 |
article | The standard visual overview of the RLHF pipeline. | The three-stage diagram: pretrained model, reward model trained on comparisons, RL fine-tuning with a KL penalty to the reference model. | intro |
| Training language models to follow instructions with human feedback · PDF Ouyang, Wu, Jiang et al., 2022 |
paper | The definitive RLHF paper for language models. | Figure 2 (SFT, reward model, PPO) and Section 3.5 (reward-model loss and the PPO-ptx objective). Note that outputs of the 1.3B tuned model were preferred to those of 175B GPT-3. | intermediate |
| Direct Preference Optimization: Your Language Model is Secretly a Reward Model · PDF Rafailov et al., 2023 |
paper | Removes the reward model and the RL loop with a short derivation you can check by hand. | Section 4: from the optimum of the KL-constrained objective to the DPO loss, and what the DPO gradient does. Then Section 5 on the reward reparameterisation. | intermediate |
| DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models · PDF Shao, Wang, Zhu et al., 2024 |
paper | The primary source for GRPO, the RL algorithm behind most open reasoning models. | Section 4 (GRPO: group-relative advantages instead of a value network, and the KL estimator) with its PPO-vs-GRPO diagram, and the unified view of SFT, rejection sampling, DPO, PPO and GRPO. | advanced |
| Reinforcement Learning from Human Feedback (the RLHF Book) · PDF Nathan Lambert, 2025 |
notes | A free, book-length reference with consistent notation across reward modelling, policy gradients and direct alignment. | Use as the reference text: reward models, regularisation, policy-gradient algorithms and direct alignment algorithms. The web version is at rlhfbook.com. | intermediate |
Going further: CS336 lecture 15 slides on RLHF and alignment.
Reasoning and test-time compute¶
Reasoning models spend more compute at inference, and are trained with RL to use it well. Pairs with lab 12, module 06.
| Item | Type | Why it's the best | Focus on | Level |
|---|---|---|---|---|
| Understanding Reasoning LLMs Sebastian Raschka, 2025 |
article | The clearest map of how reasoning models are built, organised around the DeepSeek-R1 pipeline. | The four approaches: inference-time scaling, pure RL, SFT plus RL, and SFT with distillation, and where each shows up in the R1 recipe. | intro |
| Why We Think Lilian Weng, 2025 |
article | The most thorough survey of test-time compute, with citations for every claim. | Parallel sampling vs sequential revision, RL for reasoning, and whether chains of thought are faithful. | intermediate |
| DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning · PDF DeepSeek-AI, 2025 |
paper | The open recipe that showed reasoning can be trained with RL on verifiable rewards. | R1-Zero (RL with rule-based rewards and no SFT; response length grows during training), the multi-stage R1 recipe, distillation into small models, and the unsuccessful attempts (process reward models, MCTS). | intermediate |
| Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters · PDF Snell et al., 2024 |
paper | The careful study of how to spend inference compute, and when it substitutes for a bigger model. | How the best strategy (sequential revisions vs search against a process reward model) depends on question difficulty, and the compute-optimal allocation that is more than 4x more efficient than best-of-N. | advanced |
| CS336 lecture 16: RL from verifiable rewards · PDF Percy Liang, Tatsunori Hashimoto (Stanford CS336), 2025 |
slides | Lecture slides that connect policy-gradient basics to current reasoning-model recipes. | Policy gradients to GRPO, and the published reasoning-model training recipes. | advanced |
Going further: Let's Verify Step by Step (process vs outcome reward models) and s1 (budget forcing with 1,000 examples; a good small-budget reproduction).
Quantization¶
Quantization error is dominated by outliers; each method is a different way of handling them. Pairs with lab 13, module 07.
| Item | Type | Why it's the best | Focus on | Level |
|---|---|---|---|---|
| A Visual Guide to Quantization Maarten Grootendorst, 2024 |
article | The best visual introduction, with more than 50 custom figures. | Number formats, symmetric vs asymmetric quantization, outliers and clipping, then post-training methods (GPTQ, GGUF) and quantization-aware training. | intro |
| MIT 6.5940 TinyML and Efficient Deep Learning Computing (Fall 2024) Song Han (MIT), 2024 |
slides | The best university course on efficient ML, taught by the group behind SmoothQuant and AWQ. | The two quantization lectures: linear and k-means quantization, granularity, calibration, and quantization-aware training. Slides and recordings are linked from the course page. | intermediate |
| LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale · PDF Dettmers et al., 2022 |
paper | Discovered why naive INT8 breaks large models: emergent outlier features. | The emergence of large-magnitude outlier features with scale, and mixed-precision decomposition (outlier dimensions in FP16, the rest in INT8). | intermediate |
| SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models · PDF Xiao et al., 2022 |
paper | A mathematically equivalent rescaling that makes W8A8 quantization work; implemented in lab 13. | The per-channel smoothing factor that moves quantization difficulty from activations to weights, controlled by alpha, and the figure of activation vs weight magnitudes before and after. | intermediate |
| GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers · PDF Frantar et al., 2022 |
paper | The standard second-order method for 3-4 bit weight quantization; implemented in lab 13. | The algorithm box: quantize columns in order and push each column's error onto the remaining columns using the inverse Hessian (via Cholesky), with lazy batched updates. | advanced |
Going further: AWQ (activation-aware scaling that protects salient weights).
Speculative decoding¶
A cheap draft proposes, the target verifies in one pass, and rejection sampling keeps the output distribution exact. Pairs with lab 14, module 07.
| Item | Type | Why it's the best | Focus on | Level |
|---|---|---|---|---|
| Fast Inference from Transformers via Speculative Decoding · PDF Leviathan, Kalman, Matias, 2022 |
paper | The primary source, with the proof that the output distribution is unchanged and a clean speedup analysis. | Speculative sampling and why it is exact, then the expected tokens per step and walltime improvement as functions of the acceptance rate alpha and draft length gamma. | intermediate |
| Accelerating Large Language Model Decoding with Speculative Sampling · PDF Chen et al., 2023 |
paper | Independent concurrent work; short, with the rejection-sampling rule stated compactly. | The modified rejection-sampling algorithm and its proof, and the measured speedups on a 70B model. | intermediate |
| Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads · PDF Cai et al., 2024 |
paper | Removes the separate draft model, and its tree attention verifies many candidates in one pass. | Extra decoding heads on the target model, and tree attention to verify many candidate continuations in one forward pass. | advanced |
| EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty · PDF Li et al., 2024 |
paper | The feature-level drafting idea behind EAGLE-2 and EAGLE-3, which vLLM and SGLang support. | Drafting at the feature level (the second-to-top layer) instead of the token level, and why feeding in the sampled token resolves the uncertainty. | advanced |
Mixture of experts¶
MoE decouples parameters from FLOPs per token; routing and load balancing are where it gets hard. Pairs with module 04.
| Item | Type | Why it's the best | Focus on | Level |
|---|---|---|---|---|
| Mixture of Experts Explained Omar Sanseviero et al., 2023 |
article | The best single overview, covering training, fine-tuning and serving trade-offs. | Routing, load balancing and capacity factor, why MoEs are harder to fine-tune, and the serving trade-off (every expert must be in memory). | intro |
| CS336 lecture 4: mixture of experts · PDF Percy Liang, Tatsunori Hashimoto (Stanford CS336), 2025 |
slides | Lecture slides that compare the routing and balancing choices of recent open MoE models. | Routing functions, balancing losses, and the design choices of recent open MoE models. | intermediate |
| Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer · PDF Shazeer et al., 2017 |
paper | The origin of modern MoE routing. | Noisy top-k gating and the auxiliary losses that keep experts balanced. | intermediate |
| Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity · PDF Fedus, Zoph, Shazeer, 2021 |
paper | Made MoE simple enough to train reliably; its load-balancing loss is still the standard reference. | Top-1 routing, the capacity factor and dropped tokens, the auxiliary load-balancing loss, and selective precision for router stability. | intermediate |
| DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models · PDF Dai, Deng, Zhao et al., 2024 |
paper | The architecture that DeepSeek-V2 and V3 build on, and the source of the fine-grained, shared-expert design. | Fine-grained expert segmentation plus always-active shared experts, and the evidence that experts specialise more. | advanced |
State-space models (Mamba)¶
SSMs are linear recurrences that can also run as convolutions or as a form of attention; Mamba makes them input-dependent. Pairs with module 04.
| Item | Type | Why it's the best | Focus on | Level |
|---|---|---|---|---|
| The Annotated S4 Sasha Rush, Sidd Karamcheti, 2022 |
article | Line-by-line code for the model, which makes the math concrete. | The continuous-time SSM, discretisation, and the recurrent vs convolutional views of the same model. | advanced |
| State Space Duality (Mamba-2), Part I: The Model Albert Gu, Tri Dao, 2024 |
article | The authors' own four-part explanation, clearer than the paper. | How a selective SSM and a form of masked linear attention compute the same thing. Read part I (the model) and part III (the algorithm). | advanced |
| Efficiently Modeling Long Sequences with Structured State Spaces · PDF Gu, Goel, Ré, 2021 |
paper | The paper that made SSMs competitive on long sequences. | Sections 1-2: SSMs as both recurrences and convolutions, and why naive parameterisations fail. The structured-kernel math is optional. | advanced |
| Mamba: Linear-Time Sequence Modeling with Selective State Spaces · PDF Gu, Dao, 2023 |
paper | The primary source for selective SSMs. | Section 3: why input-dependent (selective) parameters break the convolution trick, the hardware-aware parallel scan, and the synthetic tasks (selective copying, induction heads) that motivate it. | advanced |
| Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality · PDF Dao, Gu, 2024 |
paper | Unifies SSMs and attention, and gives a matmul-based algorithm that uses tensor cores. | The semiseparable-matrix view, and the chunked SSD algorithm. | advanced |
Mechanistic interpretability¶
Start with the circuits vocabulary, then superposition, then sparse autoencoders and attribution graphs. Pairs with lab 16, module 09.
| Item | Type | Why it's the best | Focus on | Level |
|---|---|---|---|---|
| Zoom In: An Introduction to Circuits Olah et al., 2020 |
article | The clearest statement of the research programme, with vision-model examples you can see. | The three claims (features, circuits, universality) and the curve-detector examples. | intro |
| A Mathematical Framework for Transformer Circuits Elhage, Nanda, Olsson et al. (Anthropic), 2021 |
article | The foundation for reasoning about transformers as circuits; most later work uses its vocabulary. | The residual stream as a communication channel, QK and OV circuits, and the zero-, one- and two-layer attention-only analyses that lead to induction heads. | intermediate |
| Toy Models of Superposition · PDF Elhage, Hume, Olsson et al., 2022 |
paper | Explains why individual neurons are hard to interpret, which motivates sparse autoencoders. | The small ReLU toy model, the phase diagram of when features are stored in superposition, and feature geometry. The HTML version on transformer-circuits.pub is the canonical one. | intermediate |
| Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet Templeton, Conerly, Marcus et al. (Anthropic), 2024 |
article | Sparse autoencoders applied to a production model, with the methods and evaluation described in detail. | How the sparse autoencoders are trained and evaluated, the feature examples, and feature steering. | advanced |
| On the Biology of a Large Language Model Lindsey, Gurnee, Ameisen et al. (Anthropic), 2025 |
article | The current frontier of circuit-level analysis on a production model. | Case studies with attribution graphs (multi-step reasoning, planning ahead when writing poetry) and the stated limitations of the method. | advanced |
Going further: In-context Learning and Induction Heads, the mechanism you build in lab 16.
Evaluation and statistics¶
An eval score is an estimate with an error bar; most published comparisons ignore that. Pairs with lab 15, module 08.
| Item | Type | Why it's the best | Focus on | Level |
|---|---|---|---|---|
| Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations · PDF Evan Miller, 2024 |
paper | The reference for doing statistics on eval results correctly; lab 15 implements it. | The formulas: standard error of an eval score, clustered standard errors for grouped questions, paired differences between two models, and sample-size planning. | intermediate |
| Evaluating Large Language Models Trained on Code · PDF Chen, Tworek, Jun et al., 2021 |
paper | Defines the unbiased pass@k estimator used by every code benchmark. | Section 2.1: the unbiased pass@k estimator, why plugging the empirical pass rate into 1-(1-p)^k is biased, and the numerically stable implementation. | intro |
| Lessons from the Trenches on Reproducible Evaluation of Language Models · PDF Biderman, Schoelkopf, Sutawika et al., 2024 |
paper | Written by the maintainers of lm-evaluation-harness; explains why reported scores for the same model disagree. | The ways evaluations silently differ (prompt format, answer extraction, normalisation, few-shot selection) and the recommended practices. | intermediate |
| Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference · PDF Chiang, Zheng, Sheng et al., 2024 |
paper | The statistics behind pairwise-preference leaderboards. | The Bradley-Terry model, confidence intervals, and how model pairs are sampled. | intermediate |
| The LLM Evaluation Guidebook Clémentine Fourrier (Hugging Face), 2024 |
notes | Practical advice from people who ran the Open LLM Leaderboard. | The chapters on automatic benchmarks, human and model-as-judge evaluation, and troubleshooting. As of late 2025 the repository points to a maintained successor. | intro |
RAG and agents¶
Retrieval quality and simple, observable control flow matter more than framework choice. Pairs with lab 17, module 10.
| Item | Type | Why it's the best | Focus on | Level |
|---|---|---|---|---|
| Building effective agents Erik Schluntz, Barry Zhang (Anthropic), 2024 |
article | The most useful practical guide: simple composable patterns first, autonomous agents only when needed. | The workflow patterns (prompt chaining, routing, parallelisation, orchestrator-workers, evaluator-optimiser) vs agents, and when not to build an agent. | intro |
| LLM Powered Autonomous Agents Lilian Weng, 2023 |
article | Still the clearest map of the agent design space. | The planning, memory and tool-use decomposition, with the key papers for each. | intro |
| Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks · PDF Lewis, Perez, Piktus et al., 2020 |
paper | The paper that named and defined RAG. | Figure 1 and Section 2: a retriever and generator trained end to end, and RAG-Sequence vs RAG-Token marginalisation. | intermediate |
| ReAct: Synergizing Reasoning and Acting in Language Models · PDF Yao et al., 2022 |
paper | The thought-action-observation loop that most agent frameworks still use. | Figure 1 (reasoning-only vs acting-only vs interleaved traces) and the error analysis. | intro |
| Lost in the Middle: How Language Models Use Long Contexts · PDF Liu et al., 2023 |
paper | A simple, robust finding that changes how you order retrieved context. | The U-shaped accuracy curve over the position of the relevant document, and what it implies for placing retrieved chunks. | intro |
How to download the PDFs¶
The downloader uses only the Python standard library, so it runs anywhere Python 3.10+ does, including Windows without WSL.
python tools/fetch_library.py --list # every item, with topic, kind, level and whether a PDF exists
python tools/fetch_library.py --list --topic quantization # one topic
python tools/fetch_library.py --dry-run # show what would be downloaded, and where
python tools/fetch_library.py --topic flashattention --topic kv-cache-serving
python tools/fetch_library.py # everything with a PDF
python tools/fetch_library.py --id chinchilla --force # re-download one item
On Windows use py instead of python if python is not on your PATH.
What it does:
- Saves each file to
library/pdfs/<topic>/<id>.pdf. That folder is listed in.gitignore. - Skips files that already exist unless you pass
--force. - Uses a browser-like User-Agent, a 60-second timeout and up to 3 attempts with exponential backoff, and waits 1 second between downloads to be polite to arXiv.
- Checks that each response body starts with
%PDFand refuses to save anything else (for example an HTML error or login page). - Prints a summary table (ok, skipped, failed) and exits with status 1 if any download failed.
Tip
If a download fails behind a corporate or campus proxy, open the item's landing page in a browser instead. Some hosts rate-limit scripts; re-running later usually works because finished files are skipped.
Licensing and fair use¶
Info
The PDFs are for personal study only. They are downloaded to your machine and never committed to this repository: library/pdfs/ is git-ignored, and it should stay that way. Do not re-host or redistribute them.
- This repository links; it does not copy. Titles, links and the short notes above are the only things stored here.
- arXiv papers keep the license their authors chose, shown on each abstract page. Many use arXiv's default non-exclusive license, which lets arXiv distribute the paper but gives you no right to redistribute it.
- Articles and blogs (distill.pub, transformer-circuits.pub, Hugging Face, personal blogs) are read online. Distill articles are CC BY 4.0.
- Course slides (Stanford CS336) come from the course's public GitHub repository, which is MIT-licensed; cite the course if you reuse a figure.
- The
licensefield inmanifest.jsonis filled in only where it is known. When it is missing, assume all rights are reserved and check the landing page.
Adding an item¶
- The bar: it must be the best available item for its topic at its level, not merely good. Replace an item rather than growing a topic past five.
- Open the landing page and the PDF link yourself. Prefer stable URLs:
arxiv.org/abs/<id>andarxiv.org/pdf/<id>, official course sites, and authors' own pages. - Add an object to
manifest.jsonwithid(unique, kebab-case),topic,title,authors,year,kind(paper,notes,slides,articleorvideo),pdf_url(only if a PDF exists),landing_url,license(if known),focusandlevel(intro,intermediateoradvanced). - Add the matching table row here, then run
python tools/fetch_library.py --listto check that the manifest parses.