How LLMs Actually Work: Tokens to RAG

By the end

What you'll build

• Explain why LLMs read tokens, not words, and implement a toy byte-pair-encoding tokenizer that trains and encodes
• Reason about vocabulary size, special tokens, and token economics, including why non-English text costs more tokens
• Compute cosine similarity in NumPy and explain embeddings as the geometry of meaning
• Build a nearest-neighbour search over embedding vectors — the core of a retriever
• Describe context windows, and implement softmax with temperature and top-k/top-p truncation sampling
• Chunk documents with overlap and reason about the size/overlap/top-k trade-offs
• Implement TF-IDF retrieval by hand and explain how BM25 refines it, plus when dense retrieval wins
• Assemble an end-to-end RAG pipeline: chunk, embed, retrieve top-k, and build a grounded prompt
• Diagnose RAG retrieval failures using a failure taxonomy, then apply and verify the right fix
• Reproduce a production-shaped RAG on your own machine with real sentence-transformers embeddings and a FAISS index

The shape of it

How this course works