Glossary
Key terms from the curriculum, defined in plain language.
A
- Attention
- The mechanism that allows each token in a transformer to query every other token and compute relevance weights, enabling information flow across the sequence. The attention matrix grows quadratically with sequence length.
- Related chapter
B
- BM25
- A term-frequency and inverse-document-frequency scoring function used in sparse retrieval. It handles exact matches and rare tokens without learned representations, making it robust to domain shift.
- Related chapter
- Byte-Pair Encoding
- A tokenization algorithm that starts from characters and iteratively merges the most frequent adjacent symbol pairs until reaching a target vocabulary size. Common words map to single tokens while rare words decompose into subword fragments.
- Related chapter
C
- Catastrophic Forgetting
- A failure mode in fine-tuning where weight updates on a narrow dataset overwrite the model's general capabilities. Parameter-efficient methods like LoRA reduce this risk by freezing base weights.
- Related chapter
- Chain of Thought
- A prompting technique that asks the model to externalize reasoning into explicit steps before committing to an answer. Improves multi-step accuracy on tasks with verifiable intermediate structure like arithmetic and logic.
- Related chapter
- Chunking
- The process of splitting documents into smaller segments for embedding and retrieval. A practical starting point is 256 to 512 tokens with 40 to 80 token overlap to prevent useful sentences from being stranded at boundaries.
- Related chapter
- Context Window
- The maximum number of tokens a model can process in a single forward pass. Limits are measured in tokens, not words, and vary by model from 8K to 2M tokens.
- Related chapter, Related chapter
- Continuous Batching
- A scheduling strategy that makes batching decisions at the token level rather than request level. When a request completes, its memory is freed immediately and a new request fills that slot on the next decode iteration.
- Related chapter
- Cross-Encoder Reranker
- A model that scores relevance by concatenating query and document into a single sequence and running full transformer attention over both. More accurate than bi-encoder dot products but requires one forward pass per candidate.
- Related chapter
D
- DPO
- Direct Preference Optimization eliminates the reward model from RLHF by optimizing the policy directly on preference pairs. More stable than PPO and requires only the policy and reference model in memory.
- Related chapter
E
- Embeddings
- High-dimensional vector representations where semantically related tokens end up close together. Transformer layers refine embeddings through context, making them contextual rather than static.
- Related chapter, Related chapter
- Episodic Memory
- Memory that stores what happened: past interactions, task trajectories, and session summaries. Typically implemented as a vector database with semantic search.
- Related chapter
F
- Few-Shot Prompting
- A technique that inserts worked examples before the actual query. The model generalizes from the pattern in the examples through in-context learning without weight updates.
- Related chapter
- Fine-Tuning
- Adapting a pretrained model to specific tasks by updating weights on task-specific data. Teaches models how to respond (format, tone, behavior), not what facts to recall.
- Related chapter
- FlashAttention
- An algorithm that avoids materializing the full attention score matrix by computing attention in tiles that fit in on-chip SRAM. Reduces memory bandwidth and enables longer context lengths.
- Related chapter, Related chapter
- Function Calling
- Structured generation where the model returns tool calls (function name plus typed arguments) rather than prose. Separates reasoning from execution and enables agents to interact with external systems.
- Related chapter, Related chapter
G
- GQA
- Grouped-Query Attention groups multiple query heads to share a single key-value head pair. Reduces KV cache memory by up to 8x with under 0.2% accuracy loss compared to Multi-Head Attention.
- Related chapter, Related chapter
H
- Hallucination
- When a model generates confident, plausible text that is factually incorrect. Occurs because the model predicts probable continuations even when training data contains no signal about a specific fact.
- Related chapter
- Hybrid Retrieval
- Combines dense (embedding) search and sparse (BM25) keyword search, merging results with Reciprocal Rank Fusion. Sparse catches exact terminology, dense catches semantic equivalence.
- Related chapter
I
- Inference
- The process of running a trained model to generate outputs. Splits into prefill (compute-bound, processes input tokens) and decode (memory-bound, generates output tokens one at a time).
- Related chapter
- Instruction Tuning
- Supervised fine-tuning on prompt-response pairs that converts a base model into an assistant. Quality matters more than quantity: 1,000 diverse expert-reviewed examples outperform 100,000 scraped pairs.
- Related chapter
J
- JSON Mode
- A serving feature that masks the vocabulary at each token position so only syntactically valid JSON tokens can be sampled. Guarantees schema conformance at the syntax level.
- Related chapter
K
- Knowledge Distillation
- Transfers behavior from a large teacher model to a smaller student model by training the student to match the teacher's full output distribution over all tokens, not just final answers.
- Related chapter
- KV Cache
- Stores attention keys and values for all prior tokens during autoregressive generation to avoid recomputation. Memory footprint scales linearly with context length and batch size.
- Related chapter, Related chapter
L
- LoRA
- Low-Rank Adaptation decomposes weight updates into two small matrices, training only 0.1 to 1 percent of parameters while freezing the base model. Reduces memory and catastrophic forgetting risk.
- Related chapter
- Lost in the Middle
- A phenomenon where models reliably use information near the start and end of long prompts but substantially underuse content buried in the middle. Worsens with context length.
- Related chapter, Related chapter
M
- Mixture of Experts
- An architecture that replaces the feed-forward network with multiple expert networks and a learned router. Reduces per-token compute but all expert weights must reside in memory simultaneously.
- Related chapter
- MQA
- Multi-Query Attention uses one key-value head shared across all query heads, reducing memory 64x compared to Multi-Head Attention but with a 2 to 3% accuracy penalty.
- Related chapter
P
- PagedAttention
- Applies virtual memory principles to KV cache management by dividing it into fixed-size blocks. Reduces memory waste below 4% and enables prefix sharing at zero copy cost.
- Related chapter
- Prefix Caching
- Reuses GPU-computed KV representations for static prompt portions across requests. Reduces input token cost by 50 to 90 percent on shared prefixes like system prompts.
- Related chapter, Related chapter
- Prompt Injection
- An attack where user-controlled input overwrites or extends the model's instructions. Mitigated by using XML delimiters to isolate untrusted input from trusted instructions.
- Related chapter
Q
- QLoRA
- Extends LoRA by quantizing the frozen base model to 4-bit using NF4 before training. Reduces memory by 4x compared to 16-bit LoRA, enabling 70B models on two A100s.
- Related chapter
- Quantization
- Reduces weight precision from 16-bit to 8-bit or 4-bit, cutting memory transfers and improving decode throughput. INT4 typically costs under 1 to 3 percent accuracy on benchmarks.
- Related chapter
R
- RAG
- Retrieval-Augmented Generation fixes knowledge gaps by finding relevant documents at query time and injecting them into the prompt. Replaces weight-encoded memory with retrieval from an external index.
- Related chapter
- ReAct
- A control loop pattern for agents that cycles through Think (LLM reasons about next action), Act (execute tool call), Observe (receive tool output), and Loop (feed observation back as context).
- Related chapter
- Reciprocal Rank Fusion
- A method for merging ranked lists from multiple retrieval systems. Each document gets a score of 1 divided by k plus rank in each list, then scores are summed. Requires no score normalization.
- Related chapter
- Reranking
- A second-stage retrieval step that uses a cross-encoder to score candidate chunks against the query with joint attention. More accurate than bi-encoder embeddings but requires one forward pass per candidate.
- Related chapter
- RLHF
- Reinforcement Learning from Human Feedback trains a reward model on preference pairs, then optimizes the policy with PPO using the reward model as signal. Operationally demanding and sensitive to hyperparameters.
- Related chapter
- RoPE
- Rotary Position Embedding encodes relative position by rotating query and key vectors before the dot product. Generalizes better to sequences longer than training length.
- Related chapter
S
- Self-Attention
- The operation where every token simultaneously asks every other token how relevant it is, computing weighted relationships. Enables long-range dependencies at the cost of quadratic memory scaling.
- Related chapter
- Self-Consistency
- Runs the same prompt multiple times at nonzero temperature, generating independent reasoning paths, then selects the most common final answer. Correct paths converge, wrong paths diverge.
- Related chapter
- Semantic Caching
- Uses vector search over past query-response pairs to return cached answers for semantically similar queries. Can cut costs by 30 to 70 percent at high volume.
- Related chapter
- Semantic Memory
- Memory that stores what is true: facts about entities, confirmed user preferences, and attributes. Requires exact retrieval via key-value or relational storage, not fuzzy vector search.
- Related chapter
- Speculative Decoding
- A small draft model generates K candidate tokens cheaply, then the large target model verifies all K predictions in a single parallel pass. Can achieve 2x to 3x speedup on low-temperature tasks.
- Related chapter
T
- Temperature
- A sampling parameter that controls output randomness. Lower values produce more deterministic outputs, higher values increase diversity. Set to 0 for greedy decoding.
- Related chapter
- Tokenization
- Breaking input text into discrete integer IDs that the model processes. Tokens are subword fragments, not words. Context limits are always measured in tokens.
- Related chapter
- Tool Schema
- A JSON Schema definition describing a tool's name, purpose, and typed parameters. The LLM uses the schema to decide which tool to call and what arguments to pass.
- Related chapter
- Top-p
- Nucleus sampling that selects from the smallest set of tokens whose cumulative probability exceeds p. Provides dynamic vocabulary cutoff that adapts to the probability distribution.
- Related chapter
- TPOT
- Time Per Output Token measures the delay between successive generated tokens. Dominated by decode time and determines streaming responsiveness.
- Related chapter
- Transformer
- A neural architecture built on self-attention that processes all tokens in a sequence simultaneously during training. Enabled scaling through parallelism that recurrent models could not achieve.
- Related chapter
- TTFT
- Time to First Token measures the delay between sending a request and receiving the first response token. Dominated by prefill time and critical for interactive applications.
- Related chapter
V
- Vector Database
- A database optimized for storing embeddings and performing approximate nearest-neighbor search. Used in RAG pipelines and episodic memory systems.
- Related chapter, Related chapter
Z
- Zero-Shot Prompting
- Asking the model to perform a task without providing examples. Works well for tasks with abundant training signal but fails when output format is unusual or precision is critical.
- Related chapter