Reading list: Google DeepMind and Google's AI teams¶
Thirty primary sources, grouped by theme, that Google interviewers are likely to know well because their colleagues wrote them. For each: link, year, and what to extract so you can discuss it precisely.
Start with the five marked read first. Pair each with its lab in this repo; reading without building does not stick (see module 00).
Scaled dot-product attention and why the \(1/\sqrt{d_k}\) factor keeps logit variance near 1; multi-head attention as parallel subspaces; the encoder–decoder layout that decoder-only models later dropped.
Everything framed as text-to-text; span-corruption pretraining; the C4 dataset and its filtering heuristics; relative position biases; and above all the method: one controlled ablation per design choice. Use it as a template for your own write-ups.
Three independent estimation methods agreeing that parameters and tokens should scale roughly equally with compute; the ~20 tokens per parameter rule of thumb; Chinchilla (70B, 1.4T tokens) beating the much larger Gopher at the same compute. Know why inference cost pushes real models past the "optimal" token count.
540B dense model trained on 6144 TPU v4 chips across two pods with Pathways. Architecture: SwiGLU, parallel attention and MLP in each block, multi-query attention, RoPE, shared input/output embeddings, no biases. The paper defines model FLOPs utilization (MFU). Loss spikes were handled by restarting from an earlier checkpoint and skipping a few hundred batches.
Two instabilities reproduced in small models at high learning rates: attention-logit growth (fixed by QK-layernorm) and output-logit divergence (fixed by z-loss). A method you can run on an 8 GB GPU: measure LR sensitivity across sizes.
Where QK normalization entered mainstream large-model practice, plus parallel layers and omitted biases for efficiency. Read the stability section only.
A large GDM sweep over parameterizations (standard, μP and others) and optimizers; per-layer learning-rate prescriptions; the finding that Adam's epsilon matters at scale and the Adam-atan2 variant that removes it.
Critical arithmetic intensity (~240 FLOPs/byte on TPU v5e); sharded-matmul notation and when each collective (AllGather, ReduceScatter, AllReduce) is needed; FSDP vs tensor vs pipeline parallelism on TPU topologies; KV-cache and latency analysis for inference; the profiling chapter. Do the worked problems.
You annotate a few tensors with shardings and the XLA compiler propagates them and inserts collectives. This is what jax.jit with NamedSharding does today.
Single-controller vs multi-controller designs; gang-scheduling of computations across TPU pods; how one client program drives thousands of chips. Useful background for infra interviews.
How to choose partitioning layouts for inference by model size, batch and latency target; why multi-query attention pairs with sharding the batch for the KV cache; the prefill vs decode trade-off expressed in MFU and latency. The best single source for TPU serving questions.
The memory-bandwidth argument: in incremental decoding, reloading K and V for every head dominates; sharing one K/V head across all query heads cuts KV bytes by the head count. Be able to redo the ratio of memory access to arithmetic.
Groups of query heads share a K/V head, interpolating between MHA and MQA; existing MHA checkpoints can be uptrained to GQA with about 5% of the original pretraining compute by mean-pooling K/V heads.
The modified rejection-sampling rule that makes the output distribution identical to the target model's; the expected tokens per target call, \(\frac{1-\alpha^{\gamma+1}}{1-\alpha}\) for acceptance rate \(\alpha\) and \(\gamma\) draft tokens; how to choose \(\gamma\) given the draft/target cost ratio.
Top-1 routing; the capacity factor and what happens to overflow tokens; selective precision (router computed in float32) for stability; the load-balancing loss you should be able to write down.
Natively multimodal training from the start; the Ultra/Pro/Nano split; the infrastructure section on training across TPU pods (failure handling, determinism, silent data corruption).
Long-context evaluation methodology (needle-in-a-haystack across text, audio and video, up to millions of tokens) and its limits; an MoE model at the frontier.
How GDM describes "thinking" models and their evaluation; the cost/quality frontier across model sizes; safety and frontier-safety evaluation sections. For later Gemini generations, read the current model cards on the GDM site.
Interleaved local sliding-window and global attention; logit soft-capping; knowledge distillation from a larger teacher to train the smaller models; GQA. (The original Gemma report, 2024, is the baseline.)
Five local layers per global layer to shrink the KV cache at long context; QK-norm replacing soft-capping; 128K context (32K for the 1B model); a SigLIP vision encoder; distillation-based training; the 262K-entry vocabulary. Compute the KV-cache saving of the 5:1 layout yourself.
What changed from Gemma 3: dense and MoE variants (the 26B-A4B model has 25.2B total and 3.8B active parameters, with 8 of 128 experts active plus 1 shared); per-layer embeddings in the E2B/E4B models; unified keys and values and proportional RoPE on global layers; up to 256K context; Apache 2.0 licence. Read the architecture and post-training sections.
Scaling the number of instruction tasks and model size; including chain-of-thought data in fine-tuning to keep reasoning ability; how the gains were measured.
Few-shot reasoning traces help only above a certain model scale; the evaluation setup on arithmetic and symbolic tasks. Historical root of "thinking" models.
Two ways to spend inference compute (search against a verifier, revising the output distribution) and a "compute-optimal" policy that adapts to problem difficulty; the FLOPs-matched comparison with a much larger model.
JumpReLU SAEs trained on every layer of Gemma 2 2B and 9B; how they measure reconstruction vs sparsity; how to use released SAEs as a research substrate on a single GPU.
Three passes. Abstract, figures and conclusion first; then the method with a pen; then reproduce one number (a FLOP count, a memory saving, a speedup) with your own arithmetic or code.
One-paragraph notes in your journal per paper: the claim, the evidence, one thing you doubt, one experiment you could run on 8 GB.
Interview framing. For each read-first item, prepare a two-minute explanation and one follow-up you would try. Google interviewers often ask "what would you change?" rather than "what does it say?".