Skip to content

Glossary

Key terms in LLM engineering. Each entry has a brief definition and a link to the course note where the concept is covered in depth.

A

Attention mechanism — The core operation in transformers. Each token computes a weighted sum of all other tokens in the context, where weights (attention scores) represent relevance. Enables the model to "focus" on relevant parts of the input regardless of distance. → Transformers and Attention

Autoregressive generation — How LLMs generate text: predict one token at a time, append it to the context, and repeat. Each token depends on all previous tokens. → How LLMs Generate Text

B

BM25 — A sparse retrieval algorithm based on term frequency and inverse document frequency. Excels at exact keyword matching; used in hybrid search alongside dense vector retrieval. → Hybrid Search

BitsAndBytes (bnb) — A library for quantizing model weights to 4-bit or 8-bit precision during training and inference, enabling large models to fit on smaller GPUs. → LoRA and QLoRA

C

Chain — A fixed sequence of LLM calls and operations. The execution path is predetermined at build time. Contrast with agent. → LangChain Overview

Chain-of-Thought (CoT) — A prompting technique where the model is instructed to generate intermediate reasoning steps before producing the final answer. Improves performance on multi-step reasoning tasks. → Chain-of-Thought

Chunking — Splitting a document into smaller text segments before embedding. Chunk size and overlap are key hyperparameters that affect retrieval quality. → Chunking Strategies

ChromaDB — An open-source vector database for storing and querying embeddings locally or in a server. Commonly used for RAG prototyping and small-scale production. → ChromaDB

Constitutional AI (CAI) — Anthropic's approach to training AI systems using AI-generated feedback based on a set of principles, rather than human labels for every example. → Responsible AI and Safety

Context window — The maximum number of tokens an LLM can process in a single call (input + output combined). Determines how much text the model can "see" at once. → Context Windows

Cosine similarity — A measure of the angle between two vectors. Returns 1.0 for identical direction, 0 for orthogonal, -1 for opposite. Used to compare embedding vectors in semantic search. → Cosine Similarity

Cross-encoder — A reranking model that processes the query and a candidate document together (with full attention between them), producing a relevance score. More accurate than bi-encoders but O(n) per query. → Reranking

D

Dense retrieval — Embedding-based retrieval where query and documents are encoded into continuous vector space and compared by cosine similarity. Captures semantic meaning. Contrast with sparse retrieval. → Vector Search

E

Embedding — A dense vector representation of text that encodes semantic meaning. Similar texts have similar embeddings (small cosine distance). Produced by embedding models like text-embedding-3-small. → What Are Embeddings

Embedding model — A model trained to produce embeddings. Distinct from a language model — embedding models produce fixed-size vectors, not text. → Embedding Models

F

Faithfulness — A RAGAS metric measuring whether the generated answer contains only claims that are supported by the retrieved context. A faithfulness score of 1.0 means every claim in the answer can be traced to the context. → RAGAS Framework

Few-shot prompting — Including 2–8 input/output examples in the prompt before the actual query. Helps the model understand the expected format and behaviour. → Zero-Shot and Few-Shot

Fine-tuning — Updating a pretrained model's weights on a task-specific dataset to improve performance on that task. Contrast with RAG (which adds knowledge at inference time) and prompting (which doesn't change weights). → Fine-Tuning Overview

Function calling — An API feature that lets you define JSON Schema tool definitions; the model returns structured tool invocations instead of free text. Also called "tool use." → Function Calling Overview

G

GGUF — A file format for quantized LLM weights, used by llama.cpp for CPU inference. Replaces the older GGML format. → llama.cpp

Grounding — Anchoring model outputs to a specific, verifiable source (documents, database records). RAG is a grounding technique. → What Is RAG

Guardrails — Input and output checks that enforce safety and quality constraints on LLM pipelines. Can be rule-based, classifier-based, or model-based. → Guardrails

H

Hallucination — When a model generates plausible-sounding but factually incorrect or unsupported content. The primary failure mode RAG is designed to mitigate. → Hallucination and Faithfulness

HNSW (Hierarchical Navigable Small World) — The graph-based index algorithm used by most vector databases for approximate nearest neighbour search. Provides fast retrieval with small accuracy loss vs brute force. → Indexing and Filtering

Human-in-the-loop — A design pattern where a human reviews and approves agent actions before they are executed. Implemented in LangGraph via interrupt_before. → Multi-Agent Orchestration

HyDE (Hypothetical Document Embeddings) — A retrieval technique that generates a hypothetical answer to the query and embeds that instead of the raw query. Bridges the query-document embedding gap. → HyDE

I

In-context learning — The ability of LLMs to perform new tasks given only examples in the prompt, without any weight updates. Demonstrated by GPT-3. → Zero-Shot and Few-Shot

J

JSON mode — An API parameter (response_format={"type": "json_object"}) that guarantees the model's output is valid JSON. Does not enforce a specific schema. → Structured Output

K

KV cache — A performance optimization in transformer inference. Key and value matrices from previous tokens are cached so they don't need to be recomputed for each new token. Reduces inference cost for long contexts. → Context Windows

L

LangChain — A framework for building LLM applications with composable chains, LCEL pipelines, memory, and integrations. → LangChain Fundamentals

LangGraph — A library for building stateful, graph-structured LLM applications with explicit state management, conditional routing, and checkpointing. → LangGraph

LangSmith — An observability platform for LLM applications. Captures traces, datasets, and evaluations for chains, agents, and RAG pipelines. → LangSmith

LCEL (LangChain Expression Language) — A declarative syntax for composing LangChain components using the pipe operator (|). Supports async, streaming, and parallel execution. → LCEL

LLM-as-judge — Using a capable LLM to evaluate the outputs of another LLM on a rubric (e.g., faithfulness 1–5, relevance 1–5). Enables automated evaluation without human annotation. → Human Evaluations

LLMOps — The operational practices for deploying, monitoring, and maintaining LLM-powered applications in production. → LLMOps

LoRA (Low-Rank Adaptation) — A PEFT method that freezes base model weights and adds trainable low-rank decomposition matrices to specific layers. Enables efficient fine-tuning with a small fraction of the parameters. → LoRA and QLoRA

M

MTEB (Massive Text Embedding Benchmark) — A benchmark evaluating embedding models across 56 tasks (retrieval, clustering, classification, etc.). The standard leaderboard for choosing embedding models. → Embedding Models

Multi-head attention — Running multiple attention operations in parallel, each with different learned weight matrices. The outputs are concatenated and projected. Allows the model to attend to different aspects of the input simultaneously. → Transformers and Attention

N

NF4 (Normal Float 4) — A 4-bit quantization format used in QLoRA. Bin boundaries are spaced to match the normal distribution of pretrained weights, making it more accurate than standard 4-bit integer quantization. → LoRA and QLoRA

O

Ollama — A tool for running open-source LLMs locally with an OpenAI-compatible REST API. Manages model downloads and serves inference. → Ollama

P

PEFT (Parameter-Efficient Fine-Tuning) — A family of methods for fine-tuning large models by training only a small subset of parameters. LoRA is the most widely used PEFT method. → PEFT

Perplexity — A measure of how well a language model predicts a held-out text. Lower perplexity = the model is less surprised by the text. Used to compare models on domain fit. → Evaluation

Prompt caching — An Anthropic feature that caches large prompt prefixes (system prompts, context) for 5 minutes, charging ~10% of normal input token price on cache hits. → Anthropic Messages API

Prompt injection — An attack where user input contains instructions that override or extend the system prompt, causing the model to behave in unintended ways. → Jailbreaks and Prompt Injection

Q

QLoRA — Quantized LoRA. Combines 4-bit NF4 quantization of the base model with LoRA adapter training in bfloat16. Enables fine-tuning 7B–70B models on a single GPU. → LoRA and QLoRA

Quantization — Reducing the numerical precision of model weights (e.g., from float32 to int4) to decrease memory requirements and speed up inference, at the cost of some accuracy. → Quantization

R

RAG (Retrieval-Augmented Generation) — A technique that retrieves relevant documents from an external knowledge base and includes them in the prompt before generation. Reduces hallucination and enables knowledge-grounded responses. → What Is RAG

RAGAS — A framework for evaluating RAG pipelines using automated metrics: faithfulness, answer relevancy, context recall, and context precision. → RAGAS Framework

ReAct — A prompting pattern that interleaves Reasoning (Thought) and Acting (Action/Observation) in an agent loop. The conceptual basis for most LLM agent frameworks. → The ReAct Loop

Recall@k — The fraction of relevant documents that appear in the top-k retrieval results. A primary metric for evaluating retrieval quality. → Evaluation Overview

Reducer — In LangGraph, a function that defines how to merge a new value with the existing state value for a key. operator.add appends to lists; default (no reducer) replaces. → Stateful Graphs

Reranking — A post-retrieval step that scores retrieved documents for relevance using a cross-encoder model. Improves retrieval precision at the cost of added latency. → Reranking

RLHF (Reinforcement Learning from Human Feedback) — A training technique that uses human preference judgments to train a reward model, which is then used to fine-tune the LLM via RL. Used to align models with human preferences. → Responsible AI and Safety

S

Self-consistency — A prompting technique that samples the model multiple times with temperature > 0 and takes a majority vote on the final answer. Improves accuracy at the cost of higher token usage. → Chain-of-Thought

Semantic cache — A cache that stores LLM responses indexed by embeddings. On a new query, checks cosine similarity to cached queries; returns a cached response if similarity exceeds a threshold. → Caching Strategies

SFTTrainer — The TRL library's supervised fine-tuning trainer. Handles dataset formatting, gradient accumulation, and PEFT integration. → PEFT

Sparse retrieval — Retrieval based on exact token matching (BM25, TF-IDF). Strong for queries with specific identifiers; weak for semantic similarity. → Hybrid Search

SSE (Server-Sent Events) — A one-way HTTP streaming protocol where the server pushes data: {content}\n\n events to the client. Used for streaming LLM token output. → Streaming Responses

StateGraph — The core LangGraph class. Defines nodes (functions), edges (transitions), and state schema. Compiled into a runnable graph. → Nodes and Edges

System prompt — Instructions passed to the model before the conversation begins. Defines persona, scope, and format constraints. Processed with higher priority than user messages in most models. → System Prompts

T

Temperature — A parameter that controls the randomness of token sampling. temperature=0 = deterministic (greedy), temperature=1 = sample from full distribution. → How LLMs Generate Text

tiktoken — OpenAI's tokenizer library. Used to count tokens before sending requests (for cost estimation) and to implement token-aware chunking. → Tokenization

Token — The basic unit of text that LLMs process. Roughly 0.75 words in English. Pricing, context limits, and rate limits are all measured in tokens. → Tokenization

Tool use — See function calling.

top_p (nucleus sampling) — Sample only from the smallest set of tokens whose cumulative probability reaches top_p. Alternative to temperature for controlling randomness. → How LLMs Generate Text

Transformer — The neural network architecture underlying all modern LLMs. Uses self-attention to process all tokens in parallel, enabling long-range dependencies. → Transformers and Attention

V

Vector database — A database optimised for storing and querying high-dimensional embedding vectors. Key capabilities: approximate nearest neighbour search, metadata filtering, CRUD operations. → Vector DB Overview

vLLM — A high-throughput LLM serving library with PagedAttention for efficient KV cache management. Used for self-hosted production inference. → Local LLMs

Z

Zero-shot prompting — Giving the model only instructions, with no examples. Works when the task is within the model's training distribution and the instructions are sufficiently clear. → Zero-Shot and Few-Shot