Reference: LLM Vocabulary

This is the single reference file that closes the vocabulary gap the corpus never quite filled: what a token actually is, how embeddings are compared, what attention is (briefly — the corpus has a whole series), why the context window misbehaves, what the sampling knobs do, and how the whole stack from “rewrite the prompt” to “train a new model” sits on one ladder of cost. Read top to bottom the first time; jump to the roster table thereafter.

Roster

Concept	One-line definition	Typical range / value	When to reach for it	Corpus article
Token	The atomic unit a language model actually reads; not a word, not a character.	1 token ≈ 4 English chars ≈ 0.75 words	Always — it is the unit everything below is denominated in.	this reference (tokenization gap)
Tokenization (BPE / WordPiece / SentencePiece)	The algorithm that splits raw text into tokens and maps each to an integer ID.	vocab 30k–200k	Before any model call; the tokenizer is the model’s vocabulary.	this reference
Embedding	A fixed-length vector representing a piece of text in a space where “similar” inputs are nearby.	384–4096 dims	Any time you need semantic similarity, search, or clustering of text.	What are embeddings)
Cosine similarity	Dot product of two L2-normalized vectors; range −1 to 1.	[−1, 1], 1 = identical direction	Comparing embeddings when magnitude is uninformative (default for text search).	What are embeddings)
Dot product (raw)	Σ aᵢ·bᵢ, unnormalized.	(−∞, ∞)	Only when vectors are already unit-normalized or magnitude carries signal.	What are embeddings)
Attention	A weighted average of a set of vectors, weights from a learned query–key similarity.	per head, O(n²) in seq length	Core of the Transformer; you pick a model that uses it, you don’t invoke it.	Attention from scratch)
Context window	Max tokens the model can attend to in one forward pass.	4k–200k (model-specific)	Determines how much text fits in the prompt before the start gets lost.	Why LLMs forget the middle)
Temperature	Scalar that divides logits before softmax; higher = flatter = more random.	0.0 (greedy) to ~2.0; 0.7 typical	Tuning creativity vs determinism of generation.	this reference
Top-k	Keep only the k highest-prob tokens, renormalize, sample.	k = 1 (greedy) to ~100	Capping the long tail with a fixed, predictable cutoff regardless of shape.	this reference
Top-p (nucleus)	Keep the smallest set of tokens whose cumulative prob ≥ p, renormalize.	p = 0.9–0.95 typical	Cutting the long tail adaptively — fits better than top-k when how peaked the distribution is varies step to step.	this reference
Repetition penalty	Discount tokens already present in the output.	1.0 (off) to ~1.3	Stopping loops in long-form generation.	this reference
Prompt engineering	Writing the prompt (instructions, examples, format) to steer the model.	$0 per call	Always the first lever; cheapest.	Prompt engineering patterns
In-context learning (ICL)	Putting worked examples in the prompt so the model infers the task without weight updates.	2–10 shots in the prompt	When the task is expressible in words + a few demos.	Fine-tuning vs prompt engineering vs ICL
Retrieval-augmented generation (RAG)	Fetch relevant chunks from a store and paste them into the prompt at call time.	k = 3–20 chunks of ~500 tokens	When the model needs knowledge it wasn’t trained on, or that changes.	Building your first RAG pipeline); Fine-tuning vs RAG; Vector databases compared)
Fine-tuning	Updating some or all of the model’s weights on your data.	$ to $$$	When the model needs a skill or style it can’t get from the prompt.	Fine-tuning vs RAG
LoRA / QLoRA	Fine-tune only small low-rank adapter matrices bolted onto the frozen base.	rank r = 8–64; <1% of params	When full fine-tuning is too expensive but you still need weight updates.	LoRA and QLoRA explained)
Hallucination	A fluent, confident, false statement; the model doing what it was trained to do when the most-likely continuation is wrong.	—	Sampling knobs do not fix this; the steering ladder does.	Why LLMs confidently make things up)

Which lever when — the steering-lever ladder

The most useful mental model in this entire area: everything from “rewrite the prompt” to “train a new model” lives on one ladder of cost and effort. Start at the bottom and only climb when the current rung provably cannot fix the failure — unless you already know the failure is missing knowledge from a large, private, or fast-changing corpus, in which case start at retrieval; nobody prompt-engineers their way to information the model was never shown.

Start with the prompt. Most problems are prompt problems. State the task, the format, the constraints, the audience, the expected output. Only move up if a better prompt cannot fix it.
- Failure mode it fixes: the model knows what to do but isn’t being told clearly.
Then in-context learning (ICL). If the task is “do X to this input,” paste 2–10 worked examples of X into the prompt. The model now does X without any weight changes. It’s usually the highest-leverage lever per dollar, because a few examples cost nothing to maintain and need no training infrastructure.
- Failure mode it fixes: the task is well-defined but unfamiliar; examples teach it.
- When to stop climbing here: if a handful of examples gets you 95% of the way there, stop adding more — the marginal example past ~10 starts to dilute attention.
Then RAG. If the failure is “the model doesn’t know Y” (Y is private, recent, large, or changing), retrieve Y from a store and paste the chunks into the prompt. RAG is a prompt technique — it just automates which text goes in.
- Failure mode it fixes: missing knowledge, not missing skill.
- When not to use it: when the failure is a style or format the model can’t imitate even with examples. Knowledge ≠ capability.
Then LoRA fine-tuning. If the failure is a skill or style the model cannot do even with examples and retrieved context (a niche output format, a house tone, a proprietary API-call pattern), fine-tune a small adapter. LoRA lets you do this on one or two GPUs.
- Failure mode it fixes: capability that prompting and RAG cannot elicit.
Then full fine-tuning. Reserved for when adapters provably cannot capture the change (rare), or you are building a base model for a domain (rare and expensive).
- Failure mode it fixes: global behaviour change that adapters leave leaking through the frozen weights.

A second, smaller decision — which sampling knobs?

Need determinism (eval, tests, extraction)? temperature=0.0 (equivalently top_k=1, greedy).
General chat / drafting? temperature=0.7, top_p=0.95.
Brainstorming / divergent ideation? temperature=0.9–1.0, top_p=0.9.
Long-form where the model keeps repeating itself? Add repetition_penalty=1.1–1.2.
Never crank temperature past ~1.2 unless you are deliberately generating noise.

A third — which tokenizer? You usually do not choose. The tokenizer is shipped with the model: gpt-4o uses o200k_base, Llama-3 uses its own SentencePiece, Claude its own. You can mix tokenizers across an LLM and its embedding model (embeddings have their own), but always budget context windows and costs in the model’s own tokenizer, not characters or words.

Tokenization — the gap no corpus article ever closed

A token is the atomic unit a language model reads. It is not a word and not a character. It is an entry in a fixed vocabulary (typically 30,000–200,000 entries) learned during pretraining. Roughly, for English, one token ≈ four characters ≈ 0.75 of a word — but the mapping is learned and uneven: common words like the are one token, rare words like tokenization may split into multiple pieces — under GPT-4o’s tokenizer it’s two: token + ization.

Why len(text) is not the token count. The model sees a sequence of integer IDs, not characters. Two strings of the same character length can have very different token counts; the same string can have a different token count under a different tokenizer. Everything that depends on “how much fits in the context window” or “how much does this call cost” is measured in tokens, not characters or words — and there’s no single fixed ratio to convert between them: confuse tokens with words and your context-window budgeting, your RAG chunk sizing, and your fine-tuning data estimates can all be off by a wide, content-dependent margin. Measure with the actual tokenizer rather than assuming a constant.

The three families of subword tokenizers you will meet:

Family	Idea	Used by
BPE (Byte-Pair Encoding)	Start from characters; merge the most frequent pair until the vocab is full.	GPT-2/3/4, Llama-2, Mistral
WordPiece	Like BPE but scores merges by likelihood gain rather than raw frequency.	BERT, DistilBERT
SentencePiece	Treats the input as a raw byte stream (no whitespace pre-split); language-agnostic.	Llama-3, T5, ALBERT, mBART

The practical difference for you: BPE and WordPiece pre-split on whitespace and then merge; SentencePiece treats the whole string as one stream and handles languages with no spaces (Japanese, Thai) without falling over.

# pip install tiktoken
import tiktoken

enc = tiktoken.get_encoding("o200k_base")          # GPT-4o family tokenizer
text = "The model tokenizes 'tokenization' oddly."

ids = enc.encode(text)
print(len(text),       "chars")    # 41 chars
print(len(text.split()), "words")  # 5 whitespace-split chunks
print(len(ids),        "tokens")   # 10 tokens -- the only number the model cares about

assert enc.decode(ids) == text      # byte-level BPE round-trips losslessly on valid UTF-8

- `tiktoken.get_encoding("o200k_base")` loads the BPE vocabulary used by the `gpt-4o` family — 200,019 entries. The vocab is a fixed dict mapping byte strings → integer IDs. - `enc.encode(text)` runs BPE: start from UTF-8 bytes, apply merges in priority order until none apply, emit the IDs. The number of IDs is the true "length" the model sees — 10 tokens here: `['The', ' model', ' token', 'izes', " '", 'token', 'ization', "'", ' oddly', '.']`. - `enc.decode(ids)` is the exact inverse *for this tokenizer* — byte-level BPE (what `tiktoken` implements) is a bijection on its own output, so `decode(encode(x)) == x` holds for any valid UTF-8 input. This is not true of every tokenizer family: WordPiece as shipped with BERT lowercases and strips accents before tokenizing, so it does not round-trip, and SentencePiece without byte-fallback can lose out-of-vocabulary characters entirely. - The character count and word count are *not* good proxies for the token count, and there's no single correction factor reliable enough to budget from — the gap varies by content and is worse on code, non-Latin scripts, and strings heavy in rare words. Measure with the tokenizer.

Why this matters for the rest of the stack. Every concept below — embeddings, attention, context window, RAG, fine-tuning — is denominated in tokens. Confuse tokens with words and your context-window budgeting, your RAG chunk sizing, and your fine-tuning data estimates all go wrong by a margin large enough to matter.

Embeddings — what they are, and cosine vs dot product

An embedding is a fixed-length vector (typically 384–4096 floats) representing a piece of text in a space where “similar” texts have nearby vectors. The vector is the output of a trained encoder model (e.g., text-embedding-3-small, bge-small-en, e5-small). The geometry is learned during training so that cos(vec("dog"), vec("puppy")) > cos(vec("dog"), vec("toaster")).

Two ways to compare embeddings:

Cosine similarity — the dot product after L2-normalizing both vectors. Range [−1, 1]; 1 = same direction, 0 = orthogonal, −1 = opposite. The default for text search because it ignores vector magnitude (usually an artifact of length, not meaning).
Raw dot product — Σ aᵢbᵢ, unnormalized, range (−∞, ∞). Appropriate only when the embedding model emits unit vectors (some do — check the model’s pipeline) or when magnitude is a signal you care about.

For two vectors $a, b \in \mathbb{R}^d$:

$\text{dot}(a, b) = \sum_{i=1}^{d} a_i b_i = a \cdot b$

$\text{cos}(a, b) = \frac{a \cdot b}{\|a\|\,\|b\|} = \frac{\sum_{i} a_i b_i}{\sqrt{\sum_i a_i^2}\,\sqrt{\sum_i b_i^2}}$

Plain English	Symbol	Python (numpy)
Dot product	$a \cdot b$	`np.dot(a, b)`
L2 norm of $a$	$\\|a\\|$	`np.linalg.norm(a)`
Cosine similarity	$\cos(a,b)$	`np.dot(a,b) / (np.linalg.norm(a) * np.linalg.norm(b))`
Cosine via normalization	—	`a/=np.linalg.norm(a); b/=np.linalg.norm(b); np.dot(a,b)`

When the embedding model emits unit vectors (so $\|a\|=\|b\|=1$ ), cosine and dot product coincide exactly. This is why many vector stores “just use dot product” — they assume the model has already normalized. Verify it rather than assume it: many embedding models, especially older ones, do not normalize internally, and their raw dot product is unbounded and dominated by vector length, not meaning.

# pip install sentence-transformers
from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer("all-MiniLM-L6-v2")   # 384-dim, small and good
embs = model.encode(["a dog barks", "a puppy barks", "a toaster toasts"])
a, b, c = embs

def cos(x, y):
    return float(np.dot(x, y) / (np.linalg.norm(x) * np.linalg.norm(y)))

print(cos(a, b))                      # 0.878   similar
print(cos(a, c))                      # 0.131   dissimilar
print(float(np.dot(a, b)))            # 0.878 -- equals cosine, because this model normalizes internally

- `SentenceTransformer("all-MiniLM-L6-v2")` loads a 384-dim encoder with its *own* WordPiece tokenizer (independent of any LLM tokenizer). Its pipeline is Transformer → mean-pooling → L2-**normalize** (`print(model)` shows all three modules) — every vector this model outputs already has unit length. - `model.encode([...])` returns a `(3, 384)` numpy array; each row is the *normalized* mean-pooled hidden state of the input. - That's why `np.dot(a, b)` above equals `cos(a, b)` exactly (both 0.878): normalization makes the two identical for this specific model. - **Common mistake:** assuming this holds for every encoder. Many embedding models — especially older `sentence-transformers` checkpoints — do *not* normalize internally, so their raw dot product is unbounded and dominated by vector length, which often correlates with text length, not meaning. Check the model's pipeline (`print(model)`) before trusting a raw dot product as a similarity score, or normalize yourself and use cosine explicitly.

The dimensionality tradeoff: larger embeddings (1024, 1536, 3072) capture more nuance and disambiguate near-duplicates better, but cost more storage and more RAM in your vector index. 384 dims is enough for most retrieval over English up to ~1M vectors; 1024+ starts to matter past 10M vectors or when you need fine-grained entity disambiguation. OpenAI’s text-embedding-3-* lets you truncate to a smaller dim at query time — a clean way to trade quality for speed.

Attention — recap, not a re-teach

Attention is the mechanism that lets a Transformer look at every token in the input when producing each output token, weighting them by learned relevance. The whole thing is one line: given a query matrix $Q$ , key matrix $K$ , value matrix $V$ ,

$\text{Attention}(Q,K,V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right) V$

The “magic” is that $Q$ , $K$ , $V$ are learned projections of the input, stacked across many “heads” and many layers. We do not re-teach it here — the corpus has a full series:

Attention from scratch: the attention mechanism finally explained without)

What you need for the rest of this reference: attention is $O(n^2)$ in sequence length $n$ , which is why context windows are finite and why “lost in the middle” exists.

Context window — and the lost-in-the-middle problem

The context window is the maximum number of tokens the model can attend to in a single forward pass. It is a hard architectural limit per model (4k for old GPT-3, 8k for GPT-4/4o, 128k for GPT-4 Turbo, 200k for Claude 3.5). Exceeding it is not “the model gets slower” — it is “the API returns an error” or “the system silently truncates the start.”

Crucially, fitting is not using. Models pay an attention tax on long inputs: information in the middle of a long prompt is retrieved less reliably than information at the start or end. This is the lost-in-the-middle effect.

Why LLMs forget the middle: understanding context windows)

Practical consequence: when you RAG, put retrieved chunks at the very start or very end of the prompt, not buried in the middle. When you must put something in the middle, restate the instruction after the chunks.

Sampling knobs — temperature, top-k, top-p, repetition penalty

Generation is a loop: the model emits a probability distribution over the vocabulary for the next token, you pick one (or sample one), append it, repeat. The knobs control how that picking happens.

Temperature $T$ : divide the logits by $T$ before softmax. $T \to 0$ → argmax (greedy); $T=1$ → the model’s own distribution; $T \gg 1$ → flatter, more random.
Top-k: keep only the $k$ highest-probability tokens, zero the rest, renormalize, sample. A fixed cap, regardless of how peaked or flat the distribution is.
Top-p (nucleus): keep the smallest set of tokens whose cumulative probability ≥ $p$ , zero the rest, renormalize, sample. Adaptive — widens or narrows automatically depending on how peaked the distribution is at each step.
Repetition penalty: divide the probability of any token already in the output by a factor (typically 1.1–1.3) before sampling. Stops loops.

Let $z \in \mathbb{R}^{|V|}$ be the logits for the next token.

$p_i^{(T)} = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)}$

As $T \to 0$ , $p^{(T)}$ collapses to a point mass on $\arg\max_i z_i$ (greedy decoding). As $T \to \infty$ , $p^{(T)}$ approaches the uniform distribution.

Plain English	Symbol	Python (torch)
Logits	$z$	`logits`
Temperature-scaled softmax	$p^{(T)}$	`torch.softmax(logits / T, dim=-1)`
Greedy decode	$\arg\max_i z_i$	`logits.argmax(-1)`
Top-k mask	—	`vals, idx = torch.topk(logits, k); mask.fill_(-inf); mask[idx] = vals`
Top-p (nucleus)	—	sort by prob, cumsum, mask where cumsum > p

Top-k and top-p are applied in series: top-k first (drop the long tail), then top-p on the survivors (drop the low-probability survivors), then sample. Setting both to “strict” values can collapse the distribution to a single token and silently turn your sampler into a greedy decoder.

Same prompt, three temperatures:

# Illustrative; assumes an OpenAI-compatible client `client` is configured.
prompt = "Write one sentence about a cat who learns to code."

for T in (0.0, 0.7, 1.0):
    r = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=T,
        top_p=0.95,
        seed=42,
        max_tokens=40,
    )
    print(f"T={T}: {r.choices[0].message.content.strip()}")

Plausible output:

T=0.0: A determined tabby named Whiskers spent her nights pawing at a laptop until she mastered Python, her first program simply printing "meow."
T=0.7: A curious cat named Miso spent weeks watching her owner type, until one morning she pawed out a working Python script that printed "purr."
T=1.0: A cat named Biscuit, tired of naps, commandeered a Raspberry Pi and accidentally shipped a bug to production — but the humans were too charmed to revert it.

- `temperature=0.0` (or any $T \to 0$) collapses the softmax to argmax on the logits — the model deterministically picks the most likely next token at each step, which is as close to reproducible as a hosted API gets (see the caveats below for where it still isn't guaranteed). Use this for eval, extraction, classification — anything where you want the least randomness available. - `temperature=0.7` is the de facto default for chat: a little randomness, but the top of the distribution still dominates. Most user-visible "creative" output sits here. - `temperature=1.0` is the model's *own* learned distribution. It feels more random than 0.7 because the long tail of the vocabulary is now reachable on every step. - `seed=42` only makes the run reproducible *at a fixed temperature* and *on the same backend*; OpenAI's seed is best-effort, not a guarantee. - `top_p=0.95` is a backstop: even at high temperature, you almost never want to sample from the full 200k-vocab tail. Nucleus sampling keeps the smallest set of tokens summing to 0.95 of probability and discards the rest. - **Common mistake:** setting `temperature=0` *and* `top_p=0.1` together. At T=0 the softmax is a point mass, so top-p is irrelevant. Don't over-constrain; pick one knob to drive and leave the other at its default.

Hallucination — the failure mode that ties it all together

A hallucination is a fluent, confident, false statement from an LLM. It is not a bug in the crash sense; it is the model doing exactly what it was trained to do — sample the most likely continuation — when the most likely continuation happens to be wrong. Sampling knobs do not fix hallucinations; they change which wrong answers you get. The real fixes live on the steering-lever ladder: RAG (give the model the right facts), ICL (show it the format), fine-tuning (teach it the calibration you want).

Why LLMs confidently make things up: understanding hallucinations)

Edge cases + common mistakes

len(text) != token count. Always count tokens with the model’s own tokenizer. Character-based budgets drift unpredictably even on English, and the divide-by-four heuristic breaks down further on code or non-Latin scripts — there’s no fixed correction factor reliable enough to budget from; measure with the model’s own tokenizer.
Mixing tokenizers. Your LLM, your embedding model, and your reranker each have their own tokenizer. len(prompt) in LLM-tokens is not len(prompt) in embedding-tokens. Budget each separately.
Cosine vs dot product mismatch. If your embedding model is not unit-normalized (older sentence-transformers models often aren’t), storing raw vectors and using dot product as similarity ranks long documents above short ones regardless of relevance. Normalize, or use cosine explicitly.
temperature=0 is not fully deterministic across hosted-API runs. It is deterministic within a single backend instance, but providers change backends. Pin seed, and don’t rely on it for production reproducibility — cache outputs instead.
Top-p and top-k interact. They are applied in series (top-k first, then top-p on the survivors). Setting both to strict values can collapse the distribution to a single token and silently turn your sampler into a greedy decoder.
Repetition penalty over 1.3 hurts quality. It also penalizes legitimate repetition (repeated function names in code, repeated list items). Use the smallest value that stops the loop.
Context window ≠ effective use. Just because a model accepts 128k tokens does not mean it reliably uses the middle 64k. Reorder prompts so the crucial instruction is at the start or end.
RAG is not “search.” RAG retrieves chunks by embedding similarity; chunks are not documents. Chunk size, overlap, and the embedding model dominate the result. See Building your first RAG pipeline).
LoRA is not free fine-tuning. It is cheap relative to full fine-tuning but still needs a dataset of hundreds-to-thousands of examples, a GPU, and an eval set. It is also not a substitute for RAG when the failure is “doesn’t know X.”
In-context learning caps out. Past ~5–10 examples the prompt gets long, attention dilutes, and the model stops improving. That is the signal to move to LoRA.
Choosing a tokenizer is choosing a vocabulary. You cannot swap a model’s tokenizer without retraining — the embedding matrix is sized to the vocab. This is why “upgrade the tokenizer” is never a quick fix.

Reference: LLM Vocabulary

Roster

Which lever when — the steering-lever ladder

Tokenization — the gap no corpus article ever closed

Embeddings — what they are, and cosine vs dot product

Attention — recap, not a re-teach

Context window — and the lost-in-the-middle problem

Sampling knobs — temperature, top-k, top-p, repetition penalty

Hallucination — the failure mode that ties it all together

Edge cases + common mistakes

Cross-references

Further reading

The Three Ways to Steer an LLM (And Why You Need to Pick One)

Fine-Tuning vs. RAG: How to Actually Decide

LoRA and QLoRA Explained: Fine-Tuning Big Models on Small Budgets

Building a Baseline in 10 Minutes: A Practical AutoML Workflow

Looking for something else?

Reference: LLM Vocabulary

Roster

Which lever when — the steering-lever ladder

Tokenization — the gap no corpus article ever closed

Embeddings — what they are, and cosine vs dot product

Attention — recap, not a re-teach

Context window — and the lost-in-the-middle problem

Sampling knobs — temperature, top-k, top-p, repetition penalty

Hallucination — the failure mode that ties it all together

Edge cases + common mistakes

Cross-references

Further reading

Related articles

The Three Ways to Steer an LLM (And Why You Need to Pick One)

Fine-Tuning vs. RAG: How to Actually Decide

LoRA and QLoRA Explained: Fine-Tuning Big Models on Small Budgets

Building a Baseline in 10 Minutes: A Practical AutoML Workflow

Looking for something else?