Twenty-eight papers and documents that explain how Meta's AI organizations build models and the software under them: the Llama lineage, the current Muse models, FAIR's architecture research, efficiency work, and PyTorch's compiler and distributed stack.
Each entry says what to extract, meaning what an interviewer on that team would expect you to know, and which lab in this repo lets you verify it.
Tip
Read the five marked Read first before anything else. They cover pretraining at scale, the training platform, the compiler, post-training and safety, and the current architecture. Together they are most of what a Meta core-AI interview will probe.
Trained on public data only. Small models trained on far more tokens than Chinchilla-optimal because inference cost matters (7B on 1T tokens). The pre-norm RMSNorm + SwiGLU + RoPE recipe that became standard. Verify: lab 08 inference-aware sizing, lab 05.
2T tokens, 4K context, GQA in the larger models. Post-training: SFT quality over quantity, two reward models (helpfulness, safety), rejection sampling then PPO, the margin term in the reward-model loss, Ghost Attention for multi-turn system prompts. Verify: lab 11 for the preference-loss family.
Long context via continued pretraining on longer sequences with a larger RoPE base frequency, rather than training long from scratch. Why the base matters: lower rotation frequencies keep distant positions distinguishable.
Code specialization from a general base; fill-in-the-middle training; long-context fine-tuning with RoPE base raised from 10,000 to 1,000,000. Useful as a worked example of domain continued-pretraining.
405B dense on about 15.6T tokens; 128K vocabulary; RoPE base 500,000; staged context extension to 128K; annealing on high-quality data at the end; scaling laws that predict downstream benchmark performance, not just loss; 4D parallelism (TP, CP, PP, FSDP) and reported 38–43% BF16 MFU; 419 unexpected interruptions in a 54-day window; post-training with SFT, rejection sampling and DPO over several rounds (no PPO); contamination analysis and benchmark confidence intervals. Verify: lab 03 to recompute its compute budget.
First Llama mixture-of-experts models (Scout, Maverick, both about 17B active parameters); early-fusion multimodality; interleaved attention layers without positional encoding ("iRoPE") for long context; FP8 training; a larger Behemoth teacher used for codistillation. At release Meta published a launch post and model cards rather than a full paper. A third-party consolidation of the public details exists; treat it as secondary.
Dense 29.6B with a 1.8B perception encoder; 3:1 local/global attention with a 2,048 window and RoPE only on local layers; 32 query / 2 KV heads; gated attention; distilled from the closed Muse Spark; about 4-bit quantization to fit 24–32 GB; a block-diffusion speculative drafter (DFlash) with 3.1x reported speedup on an RTX 5090. Compute its KV cache by hand (the worked example in the guide).
1,000 carefully curated SFT examples on a 65B base; the "superficial alignment hypothesis" that pretraining provides the knowledge and SFT mostly teaches format. Know where later work disagrees (reasoning and RL-heavy post-training).
The model judges its own samples (LLM-as-a-judge prompt) to build preference pairs, then iterative DPO. Failure mode to discuss: reward hacking and judge drift when the judge and policy are the same model. Verify: lab 11.
A safety classifier built as an instruction-tuned LLM, with the risk taxonomy in the prompt so policies can change without retraining; separate input and output classification. Relevant to system-level safety design questions.
n output heads on a shared trunk each predict a future token; gains grow with model size and are largest on code; the extra heads enable self-speculative decoding. Note the memory trick: compute each head's forward and backward sequentially so logits for all heads are never held at once. Verify: lab 14.
Images tokenized into discrete tokens and modeled with text in one sequence. The stability section is the part to know: mixed-modality training diverged, and QK-norm plus changed norm placement fixed it. Verify: QK-norm in lab 05.
One transformer trained with next-token loss on text and a diffusion loss on continuous image patches, with bidirectional attention inside each image. Compare with Chameleon's discrete approach.
Modality-specific weights (FFN, attention projections, norms) with shared global self-attention; matches dense baselines at a fraction of the FLOPs. A sparsity pattern that is not token-routed MoE.
Tokenizer-free: bytes grouped into dynamic patches where a small byte model's next-byte entropy is high; FLOP-matched scaling studies show patches can scale better than tokens. Verify: lab 04 to see what BPE does to non-English scripts.
Autoregressive modeling in a sentence-embedding space instead of token space; diffusion-based variants. A research bet on higher-level representations.
Feed the last hidden state back as the next input embedding ("continuous thought") instead of decoding a token; a curriculum replaces chain-of-thought steps gradually. Discuss what is lost: interpretability and monitorability of reasoning.
Layer dropout rising with depth plus an early-exit loss so early layers can predict tokens; self-speculative decoding with early layers as the draft and the remaining layers as verifier, sharing compute and cache. Verify: lab 14.
Rotating weights and activations by an orthogonal matrix leaves the network's function unchanged but removes outliers; learning the rotation (instead of a random Hadamard) makes 4-bit weights, activations and KV cache usable. Verify: lab 13, then add a random rotation before INT4 and measure the error.
For sub-billion models, deep-and-thin beats wide at equal parameters; embedding sharing, GQA and block-wise weight sharing. Useful for any on-device (ExecuTorch) discussion.
Dynamo hooks CPython frame evaluation to capture FX graphs from bytecode, with guards that trigger recompilation and graph breaks where Python can't be traced; AOTAutograd traces the backward ahead of time; Inductor generates Triton (GPU) and C++ (CPU) with fusion. Practice: run TORCH_LOGS=graph_breaks,recompiles on lab 05's train.py --compile.
Sharding parameters, gradients and optimizer state; all-gather before forward and backward, reduce-scatter after; prefetching and rate limiting to overlap communication; hybrid sharding. The paper describes the original flat-parameter design; FSDP2 (used in torchtitan) shards each parameter as a DTensor instead. Know why that matters: per-parameter dtype and freezing, and cleaner composition with tensor parallelism. Verify: lab 09 ZeRO memory.
One library for quantization-aware training, float8 and MX training, weight-only int4/int8 inference and sparsity, designed to compose with torch.compile and FSDP2. The code is at pytorch/ao.
Contrastive vision-language training alone yields general features, but the best ones sit in intermediate layers; alignment steps bring them to the output. This family is the vision encoder in Muse Glimmer.
Predict masked video in representation space rather than pixels; an action-conditioned variant plans robot actions with little robot data. The core of FAIR's world-model agenda.
Embedding tables for sparse features plus MLPs, with explicit pairwise feature interactions; model-parallel embeddings and data-parallel MLPs. The basis of many ML system design questions at Meta.
Recommendation reframed as sequential transduction over user action histories with a modified attention; shows recommendation quality scaling with compute in a way classic DLRMs did not. Bridges LLM and ranking work.
Three passes. Abstract and figures; then method and main tables; then the appendix only for papers you will reproduce. The Llama 3 paper's infrastructure and evaluation sections deserve a full pass.
Extract numbers into your journal. Model sizes, token counts, context lengths, MFU, ablation deltas. Then recompute one of them with lab 03.
Write one paragraph per paper answering: what problem, what mechanism, what evidence, what would I test next. Use the journal templates.
Tie papers to code. For PyTorch items, read the matching source in pytorch/pytorch or pytorch/torchtitan; for model papers, find the corresponding config in torchtitan's model folder.
See also the repo-wide papers.md, and projects.md for how to turn several of these into portfolio work.