How LLMs Actually Work: Tokens to RAG

By the end

What you'll build

• Explain why LLMs read tokens, not words, and implement a toy byte-pair-encoding tokenizer that trains and encodes
• Reason about vocabulary size, special tokens, and token economics, including why non-English text costs more tokens
• Compute cosine similarity in NumPy and explain embeddings as the geometry of meaning
• Build a nearest-neighbour search over embedding vectors — the core of a retriever
• Describe context windows, and implement softmax with temperature and top-k/top-p truncation sampling
• Chunk documents with overlap and reason about the size/overlap/top-k trade-offs
• Implement TF-IDF retrieval by hand and explain how BM25 refines it, plus when dense retrieval wins
• Assemble an end-to-end RAG pipeline: chunk, embed, retrieve top-k, and build a grounded prompt
• Diagnose RAG retrieval failures using a failure taxonomy, then apply and verify the right fix
• Reproduce a production-shaped RAG on your own machine with real sentence-transformers embeddings and a FAISS index

Curriculum

What's inside

7 modules · 40 lessons

1. 01

   From Text to Tokens

   6 lessons
   • Start Here: What This Course Is (and Isn't)
   • Why Models Can't Read Words: The Tokenization Problem
   • Byte-Pair Encoding, From Scratch
   • Lab: Train a Toy BPE Tokenizerpython lab
   • Lab: Encode Words With Your Mergespython lab
   • + 1 more lesson
2. 02

   Tokens in the Real World

   4 lessons
   • Vocabularies, Special Tokens & Token Economics
   • Tokenization Gotchas: Numbers, Whitespace & Indian Scripts
   • Drill: Tokenization Vocabularyfill blank
   • Deck: Tokenization Essentialsflashcard sr
3. 03

   Embeddings as Vectors

   6 lessons
   • What Is an Embedding?
   • The Geometry of Meaning: Cosine Similarity
   • Lab: Cosine Similarity in NumPypython lab
   • Lab: Find Nearest Neighbours (Top-k by Cosine)python lab
   • Checkpoint: Embeddingsquiz
   • + 1 more lesson
4. 04

   Context, Sampling & Temperature

   6 lessons
   • The Context Window: What Fits and What Falls Off
   • From Logits to Probabilities: Softmax, Temperature & Truncation
   • Lab: Softmax, Temperature & Entropypython lab
   • Deck: Sampling & Decodingflashcard sr
   • Drill: Sampling Parametersfill blank
   • + 1 more lesson
5. 05

   Chunking Trade-offs

   5 lessons
   • Why We Chunk: Long Documents Meet Short Windows
   • Chunk Size, Overlap & the Boundary Problem
   • Lab: A Character Chunker With Overlappython lab
   • Lab: Tune Chunk Size & Top-krag lab
   • Deck: Chunking & Retrieval Termsflashcard sr
6. 06

   Retrieval & RAG End-to-End

   7 lessons
   • Retrieval 101: Sparse vs Dense, and BM25
   • Lab: TF-IDF Retrieval by Handpython lab
   • RAG: The Architecture
   • Lab: Retrieve the Answer Chunkrag lab
   • Error Analysis: When RAG Retrieves the Wrong Thing
   • + 2 more lessons
7. 07

   Build & Assess

   6 lessons
   • Lab: Assemble a Mini-RAGpython lab
   • From Browser-Scale to Production: What Actually Changes
   • Drill: RAG Pipeline Recallfill blank
   • Deck: RAG Architectureflashcard sr
   • Capstone: Build a Real RAG on Your Own Machine
   • + 1 more lesson

The shape of it

How this course works