Core AI from first principles

Rebuild the modern LLM stack from scratch, one tested lab at a time.

Autograd → tokenizer → GPT → FlashAttention in Triton → KV cache → LoRA · DPO · GRPO → quantization → speculative decoding → evals → interpretability → retrieval → mixture of experts.
Every lab runs on a laptop CPU. The scale-up runs are sized for one 8 GB GPU or a free Colab T4.

Cheat sheets · Docs · Labs · Curriculum · Library · Companies · Roadmap · Setup

llm deep-learning pytorch triton transformers flash-attention rlhf grpo quantization interpretability from-scratch ai-engineering

Free print-ready interview cheat sheets

Dense, two-column PDFs written as interview answers: the mechanism, the derivation, the numbers, and the trap the interviewer is fishing for. Download, print, revise.

LLM & GenAI Engineer · 24 pages GPU math, pretraining, RLHF / DPO / GRPO, inference serving, KV cache, RAG, agents, from-scratch code, debugging, formulas, papers

Vision & Multimodal Engineer · 25 pages Geometry, CNNs, ViT / Swin, YOLOv1 to v11, DETR, SAM 2, MAE / DINOv2, CLIP / SigLIP, VLMs, diffusion, DiT, flow matching, 3DGS, edge deployment

Plus The life of a token (5 pages): the whole LLM stack told as one story, from tokenizer to sampled output. LaTeX sources and build steps are in cheatsheet/. If one helps you prepare, a ⭐ helps the next person find it.

Test-driven 229 reference tests, run in CI on Linux, Windows and macOS

From scratch you write the math; libraries only for plumbing

Measured predict FLOPs, memory and tokens/s, then check

Runs anywhere CPU, NVIDIA, Apple Silicon, Colab, Codespaces

Career-aimed company guides, research and founder tracks

Why Achilles

Every engineer has an Achilles heel: a layer of the stack they can name but cannot build. Most AI preparation makes it worse. It is reading: blog posts, paper summaries, question lists. You finish able to name FlashAttention and unable to write it, and interviews at the labs that matter find that gap within minutes.

Achilles trains the opposite skill. You implement every core mechanism yourself, and a test suite tells you when you are right. Every lab also asks you to predict a number (memory, FLOPs, tokens/s, loss) before you measure it, because the gap between prediction and measurement is where understanding comes from.

Info

The rule: you understand what you can build, measure and explain, and nothing else.

What's inside

Part	What you get
Labs	18 test-driven labs: a handout, a stub you implement, a test suite and a reference solution for each
Curriculum	11 modules from the math to applied systems: mechanisms, derivations, napkin math, traps and papers
Library	90 visual guides, papers and course notes across 19 hard topics, the sections worth your time in each, and a PDF fetcher
Companies	How to get into Anthropic, OpenAI, Google DeepMind, Meta and NVIDIA: teams, interview loops, reading lists and portfolio projects
Research-engineer track	Interview loops, a question bank with answers, LLM system design, coding drills and a portfolio plan
Founder track	Problem selection, eval-driven products, unit economics, moats, go-to-market, fundraising and team
Career	Résumé, stories, outreach, market intel, negotiation and visas
Roadmap	A 12-month plan with exit criteria for each stage and two decision gates

Start in five minutes

git clone https://github.com/Lourdhu02/achilles.git && cd achilles
python -m venv .venv && source .venv/bin/activate      # Windows: .venv\Scripts\Activate.ps1
pip install torch --index-url https://download.pytorch.org/whl/cpu   # NVIDIA GPU: .../whl/cu128
pip install -r requirements.txt
python -m pytest --impl=solution -q    # all 229 reference tests pass: your setup works
pytest labs/01_autograd                # 37 failing tests: your first job
python tools/progress.py               # your scoreboard across all 18 labs

Tip

Nothing to install: open the Colab quickstart (free T4 GPU) or a Codespace (CPU, preconfigured). On a Mac, install torch with a plain pip install torch. Per-platform instructions, including the RTX 50-series on Windows, are in SETUP.md.

Run it on what you have

You have	What you can do	Setup
Any laptop, no GPU	Every lab's tests, the napkin math, the Triton kernels in their CPU interpreter (Linux), a small pretraining run	CPU wheels
NVIDIA GPU with 8 GB or more (RTX 5060 class)	Everything above, plus pretraining on TinyStories, Triton on the GPU, LoRA and DPO on a 0.5B model, GRPO, quantization and speculative-decoding benchmarks	CUDA wheels
Apple Silicon Mac	Every test except the Triton ones; training on the Apple GPU (MPS)	Apple Silicon
Only a browser	Colab or Kaggle with a free T4 for the scale-up runs; Codespaces for a ready CPU environment	Cloud notebooks

python tools/measure_gpu.py measures your hardware's roofline and prints the train.py --preset that fits it. The tier table lists the minimum hardware for every experiment.

How a lab works

1. Read the handout: the mechanism, the derivation and worked numbers.
2. Predict the number that matters (memory, FLOPs, loss, tokens/s) and write it down.
3. Implement exercise.py until pytest labs/NN_name passes. Hints are in the stub.
4. Measure and compare against your prediction; scale up on a GPU if you have one.
5. Write it up in journal/. The gap you found is the lesson, and a write-up is what gets you interviews.

# labs/06_attention_kernels/exercise.py (abridged)
def flash_attention_forward(q, k, v, causal=False, block_q=16, block_k=16):
    """Tiled attention with online softmax. Returns O and the log-sum-exp of each row."""
    # HINT: for each query tile keep m (running max), l (running sum), acc (unnormalized output);
    # HINT: for each key tile: s = q k^T * scale, mask, m_new = max(m, rowmax(s)), p = exp(s - m_new),
    # HINT: rescale l and acc by exp(m - m_new), add p's contribution. Finally O = acc / l, LSE = m + log l.
    raise NotImplementedError("06_attention_kernels: implement flash_attention_forward")

The tests compare your code against PyTorch or a closed form, check invariants (speculative decoding must be exactly lossless, DPO must reach its analytic optimum), and include the edge case that catches the usual bug. Run pytest labs/NN_name --impl=solution to see the reference pass, and read solution.py only after an honest attempt.

The labs

	Lab	You build	Tests
	01 · Autograd	Reverse-mode autodiff in NumPy; train an MLP with it	37
	02 · Training core	AdamW matching PyTorch over 25 steps to 1e-10, Muon, WSD schedules, correct gradient accumulation	24
	03 · Napkin math	Parameters, FLOPs, KV cache, roofline, MFU, $/token; exact counts for GPT-2 and Llama 3	18
	04 · Tokenizer	Byte-level BPE with GPT-4 and o200k pre-tokenization, and why they treat Telugu differently	33
	05 · Transformer	A Llama-style GPT (RoPE, GQA, SwiGLU, QK-norm) plus a pretraining script for CPU, CUDA or MPS	17
	06 · Attention kernels	Online softmax, FlashAttention forward and backward, a Triton kernel	21
	07 · KV cache & sampling	KV cache, chunked prefill, top-k, top-p and min-p	13
	08 · Scaling laws	Power-law fits, Chinchilla-optimal and inference-aware model sizing	5
	09 · Parallelism	Ring all-reduce, Megatron tensor parallelism, ZeRO memory; real DDP on two CPU processes	7
	10 · LoRA	Adapters, merging, a direct test of the low-rank hypothesis	4
	11 · DPO	DPO, IPO and SimPO, and a numerical proof of DPO's closed-form optimum	5
	12 · GRPO	REINFORCE to GRPO with clipping and a k3 KL penalty; RL on a verifiable task	5
	13 · Quantization	INT8 and INT4, NF4, SmoothQuant, GPTQ	8
	14 · Speculative decoding	Exact rejection sampling, statistically verified to be lossless	3
	15 · Eval statistics	Confidence intervals, paired tests, clustered standard errors, pass@k, power, Bradley–Terry	6
	16 · Interpretability	Induction heads (and the positional shortcut that fakes them), activation patching, sparse autoencoders	8
	17 · Retrieval	BM25, RRF, MMR, nDCG, an IVF vector index	5
	18 · Mixture of experts	Top-k routing, load balancing, capacity and token dropping, total vs active parameters	10

All reference solutions pass in CI on Linux, Windows and macOS, with the Triton kernels running in Triton's CPU interpreter on Linux.

Learning path

flowchart LR
    A[Math] --> B[Compute & hardware] --> C[Deep learning] --> D[Transformers]
    D --> E[Pretraining & scaling]
    D --> F[Inference & kernels]
    E --> G[Post-training: SFT · DPO · GRPO]
    G --> H[Evaluation & research]
    F --> H
    H --> I[Interpretability & safety]
    D --> J[Applied LLM systems]
    H --> K((Research engineer))
    J --> L((AI founder))

Module	Covers	Labs
00 How to learn	The read, derive, build, measure, write loop; a weekly template, a prediction log, reading papers
01 Math	Linear algebra, matrix calculus, probability and information, optimization, statistics	01
02 Compute	The GPU in one picture, the roofline (attention included), number formats, FLOP, memory and communication accounting	03
03 Deep learning	Autodiff, initialization, normalization, optimizers, schedules, mixed precision, a debugging playbook	01, 02
04 Transformers	Attention, positional encodings, the block, parameter and FLOP accounting, MoE, beyond vanilla attention	04, 05, 18
05 Pretraining	Data, scaling laws, the standard recipe, distributed training, stability, long context, planning a 7B run	08, 09
06 Post-training	SFT, LoRA, reward models and PPO, the DPO derivation, GRPO, test-time compute, distillation	10–12
07 Inference	Prefill and decode, serving mechanics, speculative decoding, quantization, kernels, capacity planning	06, 07, 13, 14
08 Evaluation & research	Error bars, product evals, experiment discipline, reproducing papers, research taste, 8 GB projects	15
09 Interpretability & safety	Mechanistic interpretability, alignment failure modes, misuse and security, governance	16
10 Applied systems	RAG, agents, context engineering, structured outputs, LLMOps and cost control	17

The library: visual guides and papers

For each of 19 hard topics (FlashAttention, distributed training, RLHF and GRPO, mechanistic interpretability, and more) the library picks the best visual explainer, the definitive paper and the course notes worth reading, says why each made the cut, and names the sections to focus on. A 10-item "start here" path takes you from the transformer to evaluation.

python tools/fetch_library.py --list                     # the 90 items and which have PDFs
python tools/fetch_library.py --topic flashattention     # download PDFs to library/pdfs/ (git-ignored) for offline study

Get into a frontier lab

Each guide covers the teams where core-AI engineers work, what the interview loop tests, the papers to read, the portfolio projects that would impress that team, and a plan to get there. Volatile facts are dated, and each folder has a brief for refreshing them every quarter.

Organization	The guide covers	Also
Anthropic	Interpretability, alignment, RL, pretraining, inference; the values interview	Reading list · Projects
OpenAI	Research, post-training and reasoning, safety systems, scaling and inference	Reading list · Projects
Google DeepMind	Gemini, research engineering, JAX and TPUs, hiring committees and team matching	Reading list · Projects
Meta	Superintelligence Labs, FAIR, Llama, PyTorch; the E-level loop	Reading list · Projects
NVIDIA	CUDA and kernels, TensorRT-LLM, deep-learning software, research; performance interviews	Reading list · Projects
More labs	Microsoft, Apple, xAI, Mistral, DeepSeek, Qwen, Hugging Face, Cohere, Ai2, India-based labs, startups

Two tracks

Research engineer The core teams and what each one tests · interview loops · a question bank with answers · LLM-era system design · coding drills · a portfolio that proves depth

AI founder Problem selection · eval-driven product · unit economics · moats · go-to-market · fundraising · team

Repository layout

curriculum/   11 modules: mechanisms, derivations, napkin math, traps, papers
labs/         18 labs: handout · exercise.py · solution.py · tests
library/      90 visual guides and papers by topic, manifest.json
companies/    Anthropic, OpenAI, Google DeepMind, Meta, NVIDIA and more labs
tracks/       research-engineer/ and founder/
career/       résumé, stories, outreach, negotiation, visas
cheatsheet/   LLM (24 pp) and vision (25 pp) interview cheat sheets, 5-page story version (PDF + LaTeX)
journal/      templates for experiments, paper notes and reviews: your proof of work
notebooks/    Colab and Kaggle quickstart
tools/        stub generator · progress scoreboard · GPU roofline · PDF fetcher · link checker · docs build

Contributing

Found a wrong number, a test that accepts a wrong implementation, or a better explainer? That is the most valuable contribution there is. See CONTRIBUTING.md, and please follow the code of conduct.

Citation

@misc{raju2026achilles,
  author       = {Raju, Lourdu},
  title        = {Achilles: Core {AI} from First Principles},
  year         = {2026},
  howpublished = {\url{https://github.com/Lourdhu02/achilles}}
}

Acknowledgements

Inspired by Andrej Karpathy's Zero to Hero and nanoGPT, Stanford CS336, GPU MODE, ARENA, and the papers in curriculum/papers.md. Icons by Lucide (ISC, see assets/icons/LICENSE).

License

MIT. If Achilles helps you, star it so others can find it.