Building a Miniature Transformer From Scratch: The 'Lego' Approach to Deep Learning

Last time, Lina assembled attention, multi-head attention, and positional encoding. Now she’s ready to wire them into a working mini-transformer, testing on a tiny toy vocabulary before moving to real book-review text.

The Blueprint: Why We Build a ‘Mini’ Version First

Think of a Transformer as a complex Lego set. The 5,000-piece Millennium Falcon looks impossible at first glance. Start with a 50-piece starter kit, though. The whole thing is just the same few bricks, snapped together in clever ways.

Giant models like GPT-4 are scaled-up versions of this same structure. Billions of parameters, sure, but the logic is identical to what we are building today. We’re going to build a “MiniTransformer.” Small enough that your laptop doesn’t melt and you can see every calculation in real-time. The goal isn’t to beat Google. It’s to understand the flow of data from a raw word to a meaningful concept.

Let’s start by defining our “bricks.” In a real model, the embedding size might be 768 or 1024. For our Lego version, we’ll keep it tiny.

import torch
import torch.nn as nn
import torch.nn.functional as F
import math

# Hyperparameters for our 'Mini' model
VOCAB_SIZE = 20    # A tiny vocabulary for our toy task
D_MODEL = 32       # Each word is represented by 32 numbers (instead of 768)
NUM_HEADS = 4      # 4 'experts' looking at the data
NUM_LAYERS = 2     # How many Transformer blocks we stack
D_FF = 64          # The size of the internal 'thinking' layer
MAX_SEQ_LEN = 10   # The longest sentence we can handle

print(f"Just the four attention projections per layer add up to roughly {NUM_LAYERS * (D_MODEL**2 * 4)} parameters.")
# That's only the Q/K/V/output projection weights -- it omits biases, the embedding
# table, the feed-forward layers, and layer norms. We'll print the model's real total
# once MiniTransformer is fully built, below.

• import torch / import torch.nn as nn / import torch.nn.functional as F / import math — Imports PyTorch core, its module API (nn.Linear, nn.Module, etc.), its functional API (F.softmax), and the standard math module (for sqrt and log).
• VOCAB_SIZE = 20 — The total number of unique “words” in the toy vocabulary. Real models use 30,000+ tokens; this is deliberately tiny so training finishes in seconds.
• D_MODEL = 32 — The dimensionality of each word embedding vector. Real Transformers use 512–1024; 32 is small enough to inspect every number.
• NUM_HEADS = 4 — The number of attention heads, matching the multi-head concept from Part 8. Each head gets D_MODEL // NUM_HEADS = 8 dimensions.
• NUM_LAYERS = 2 — How many Transformer blocks to stack. Real models stack 6–96+ layers; 2 is enough to demonstrate depth without slowing training.
• D_FF = 64 — The hidden size of the feed-forward “thinking” layer inside each block. This is wider than D_MODEL so the network can learn richer intermediate representations before projecting back.
• MAX_SEQ_LEN = 10 — The longest sequence the model can accept, determined by the positional encoding buffer size.
• NUM_LAYERS * (D_MODEL**2 * 4) — Counts only the four attention projection weight matrices per layer (Q, K, V, output), each D_MODEL × D_MODEL. It omits biases, the embedding table, the feed-forward layers, and layer norms — which is why the model’s actual parameter count, printed later once MiniTransformer is built, comes out to roughly double this estimate. Also notice this formula has no NUM_HEADS term: splitting d_model into more heads changes how the projections are grouped, not how many numbers they hold, so head count alone never moves the parameter total.

The Foundation: Embeddings and Position (Where Words Live)

Computers don’t read words; they read coordinates. We need to turn a word like “sneaker” into a map location. In this map, “sneaker” and “boot” should be neighbors, while “sneaker” and “refrigerator” sit miles apart. That’s what an Embedding does.

Here’s the thing: as we saw in Part 8, the Transformer has no built-in sense of time. Feed it “the refund exceeded the charge,” and it sees a pile of words, not a sequence. So we manually add a “GPS signal” called Positional Encoding. This tells the model that “refund” comes several positions before “charge” — not just that both words appear somewhere.

class PositionalEncoding(nn.Module):
    def __init__(self, d_model, max_len=5000):
        super().__init__()
        # Create a matrix of [max_len, d_model]
        pe = torch.zeros(max_len, d_model)
        position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1)
        # The 'clock' signal using sine and cosine
        div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))
        pe[:, 0::2] = torch.sin(position * div_term)
        pe[:, 1::2] = torch.cos(position * div_term)
        self.register_buffer('pe', pe.unsqueeze(0))

    def forward(self, x):
        # Add the GPS signal to the word embeddings
        # x shape: [batch, seq_len, d_model]
        x = x + self.pe[:, :x.size(1)]
        return x

# Let's see it in action
test_emb = torch.zeros(1, 5, D_MODEL) # 5 words, empty embeddings
pe_layer = PositionalEncoding(D_MODEL)
output = pe_layer(test_emb)
print("Shape after adding GPS signal:", output.shape) 
# Interpretation: The shape stays the same, but the zeros are now unique 'barcodes'.

• class PositionalEncoding(nn.Module) — Defines a PyTorch module that adds positional signals to embeddings. Inheriting from nn.Module gives it parameter tracking and .to(device) support.
• pe = torch.zeros(max_len, d_model) — Pre-allocates the encoding matrix: one row per position (0 to max_len - 1), one column per embedding dimension.
• position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1) — Creates a column vector of shape [max_len, 1] containing position indices [0, 1, 2, ..., max_len-1]. unsqueeze(1) adds the second dimension so it can broadcast against div_term.
• div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model)) — Pre-computes the frequency divisor for each pair of dimensions. This is the same 1 / 10000^(2i/d_model) formula from Part 8, rewritten with exp(log(...)) for numerical stability. The result is a row vector of shape [d_model/2].
• pe[:, 0::2] = torch.sin(position * div_term) — Fills every even-indexed column (indices 0, 2, 4, …) with sine values. 0::2 means “start at index 0, step by 2.” The broadcasting of position (column) and div_term (row) produces a [max_len, d_model/2] matrix that fills the even columns.
• pe[:, 1::2] = torch.cos(position * div_term) — Fills every odd-indexed column (indices 1, 3, 5, …) with cosine values, the paired counterpart to the sine columns.
• self.register_buffer('pe', pe.unsqueeze(0)) — Stores the encoding as a non-learnable buffer (not a Parameter, so no gradients are computed for it). unsqueeze(0) adds a batch dimension so the shape becomes [1, max_len, d_model], ready to broadcast against inputs of shape [batch, seq_len, d_model].
• x = x + self.pe[:, :x.size(1)] — Element-wise adds the positional encoding to the word embeddings. The slice :x.size(1) ensures we only use the first seq_len rows of the buffer, matching the actual sequence length.
• test_emb = torch.zeros(1, 5, D_MODEL) — Creates a dummy input of 5 zero-embedding words to verify the encoding doesn’t change the tensor shape.
• output.shape — Prints [1, 5, 32], confirming the positional encoding is additive (shape-preserving) rather than concatenative.

The Engine Room: Multi-Head Attention

Picture a room full of people. Attention is the process of deciding who to listen to when someone says a specific word. “Multi-head” means several people are listening for different things. One head might track who’s doing the action (grammar), while another picks up on the mood of the sentence (emotion).

The model then builds a weighted summary of the whole sentence for every single word. It uses Queries (what I’m looking for), Keys (what I contain), and Values (the actual information).

class MultiHeadAttention(nn.Module):
    def __init__(self, d_model, num_heads):
        super().__init__()
        self.num_heads = num_heads
        self.head_dim = d_model // num_heads
        
        self.q_linear = nn.Linear(d_model, d_model)
        self.k_linear = nn.Linear(d_model, d_model)
        self.v_linear = nn.Linear(d_model, d_model)
        self.out_proj = nn.Linear(d_model, d_model)

    def forward(self, x):
        batch, seq_len, _ = x.shape
        # 1. Linear projection and split into heads
        q = self.q_linear(x).view(batch, seq_len, self.num_heads, self.head_dim).transpose(1, 2)
        k = self.k_linear(x).view(batch, seq_len, self.num_heads, self.head_dim).transpose(1, 2)
        v = self.v_linear(x).view(batch, seq_len, self.num_heads, self.head_dim).transpose(1, 2)

        # 2. Scaled Dot-Product Attention
        # We divide by sqrt(head_dim) to keep numbers from exploding
        scores = torch.matmul(q, k.transpose(-2, -1)) / math.sqrt(self.head_dim)
        attn = F.softmax(scores, dim=-1)
        
        # 3. Combine heads and project back
        context = torch.matmul(attn, v).transpose(1, 2).contiguous().view(batch, seq_len, -1)
        return self.out_proj(context)

• self.head_dim = d_model // num_heads — Each head receives an equal slice of the embedding dimensions. 32 // 4 = 8 in this model. Integer division ensures an exact split; a non-divisible d_model would raise an error.
• self.q_linear / self.k_linear / self.v_linear — Three separate nn.Linear(d_model, d_model) layers, each with its own learned weight matrix. These project the shared input into Query, Key, and Value spaces. Having separate weights lets the model learn different transformations for “what am I looking for” vs. “what do I contain” vs. “what information do I carry.”
• self.out_proj = nn.Linear(d_model, d_model) — The final output projection that merges the concatenated heads back into a single d_model-wide vector (the Concatenation Trick from Part 8).
• batch, seq_len, _ = x.shape — Unpacks the input tensor’s three dimensions. The third (d_model) is discarded with _ because the linear layers already know its size.
• .view(batch, seq_len, self.num_heads, self.head_dim) — Reshapes the projected tensor from [batch, seq_len, d_model] to [batch, seq_len, num_heads, head_dim], splitting the embedding into per-head chunks.
• .transpose(1, 2) — Swaps the sequence-length and head dimensions, producing [batch, num_heads, seq_len, head_dim]. This rearrangement lets torch.matmul compute attention scores for all heads simultaneously in one batched matrix multiply.
• scores = torch.matmul(q, k.transpose(-2, -1)) / math.sqrt(self.head_dim) — Computes the raw attention scores as Q × Kᵀ (dot product of every query with every key), then scales by 1/√(head_dim). The scaling prevents the dot products from growing large when head_dim is big, which would push the softmax into saturation (near-zero gradients).
• attn = F.softmax(scores, dim=-1) — Converts raw scores into a probability distribution along the key dimension. Each row now sums to 1, representing how much attention each query pays to each key.
• context = torch.matmul(attn, v) — Weighted sum of values: each query’s attention weights are multiplied by the value matrix, producing a head_dim-wide context vector per position.
• .transpose(1, 2).contiguous().view(batch, seq_len, -1) — Transposes back to [batch, seq_len, num_heads, head_dim], calls .contiguous() to ensure the tensor is in contiguous memory after the transpose, then reshapes to [batch, seq_len, d_model] — concatenating all heads side by side. The -1 lets PyTorch infer the last dimension.
• return self.out_proj(context) — Passes the concatenated multi-head output through the final linear layer, letting the heads “talk” to each other and merge their findings.

The Assembly Line: The Transformer Block

Now we snap the pieces together. A single Transformer Block combines the attention layer with a small “thinking” layer (Feed-Forward). Two techniques hold it together:

1. Residual Connections: Add the layer’s input back to its output. This skip-connection lets information flow through even when a layer gets confused.
2. Layer Normalization: Keep the numbers “polite” (centered around zero) so they don’t grow out of control.

class TransformerBlock(nn.Module):
    def __init__(self, d_model, num_heads, d_ff):
        super().__init__()
        self.attention = MultiHeadAttention(d_model, num_heads)
        self.norm1 = nn.LayerNorm(d_model)
        self.norm2 = nn.LayerNorm(d_model)
        self.feed_forward = nn.Sequential(
            nn.Linear(d_model, d_ff),
            nn.ReLU(),
            nn.Linear(d_ff, d_model)
        )

    def forward(self, x):
        # Step 1: Attention + Skip Connection + Norm
        x = self.norm1(x + self.attention(x))
        # Step 2: Thinking + Skip Connection + Norm
        x = self.norm2(x + self.feed_forward(x))
        return x

• self.attention = MultiHeadAttention(d_model, num_heads) — Instantiates the multi-head attention layer defined above, wiring it in as a sub-module so PyTorch registers its parameters.
• self.norm1 = nn.LayerNorm(d_model) / self.norm2 = nn.LayerNorm(d_model) — Two separate Layer Normalization layers. LayerNorm normalizes across the feature dimension (d_model) for each position independently, keeping activations centered and scaled. This differs from BatchNorm (Part 11), which normalizes across the batch dimension.
• self.feed_forward = nn.Sequential(...) — A two-layer MLP: Linear(d_model → d_ff) expands the representation, ReLU() introduces non-linearity, Linear(d_ff → d_model) projects back. This “position-wise” network processes each token’s vector independently — attention gathers context, then the feed-forward transforms it.
• x = self.norm1(x + self.attention(x)) — The residual (skip) connection: the block’s input x is added to the attention output before normalization. If the attention layer produces near-zero or unhelpful outputs, the original signal still flows through the + x, preventing gradient death in deep stacks.
• x = self.norm2(x + self.feed_forward(x)) — A second residual connection around the feed-forward layer, same principle: the post-attention representation is preserved alongside the feed-forward’s contribution.
• return x — The output of one Transformer block, ready to be passed to the next stacked block or to the final output layer.

The Final Reveal: Training Our Mini-Model

Time to test our Lego creation. We’ll give it a simple task: Sequence Reversal. Feed it [1, 2, 3, 4, 5], and it should learn to output [5, 4, 3, 2, 1]. Pulling that off means it grasps both the words (embeddings) and their order (position).

There’s a catch worth calling out: if we only ever train on one fixed input/output pair, the model doesn’t need to understand reversal at all. It can just memorize “position 0 always outputs token 5, position 1 always outputs token 4” — a lookup table with zero attention involved, since the input at each position never changes across training steps. To actually prove the attention mechanism is doing something, every training batch needs a different, randomly generated sequence. That way the only way to get an unseen sequence right is to genuinely attend from each output position back to the matching input position.

class MiniTransformer(nn.Module):
    def __init__(self, vocab_size, d_model, num_heads, num_layers, d_ff, max_len):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, d_model)
        self.pos_encoding = PositionalEncoding(d_model, max_len)
        self.blocks = nn.ModuleList([TransformerBlock(d_model, num_heads, d_ff) for _ in range(num_layers)])
        self.fc_out = nn.Linear(d_model, vocab_size)

    def forward(self, x):
        x = self.pos_encoding(self.embedding(x))
        for block in self.blocks:
            x = block(x)
        return self.fc_out(x)

With the model defined, build it and see what it actually costs — then write a data generator that hands it a fresh random reversal example on every call, so there’s nothing fixed to memorize.

# Setup training
import random
torch.manual_seed(0)
random.seed(0)

model = MiniTransformer(VOCAB_SIZE, D_MODEL, NUM_HEADS, NUM_LAYERS, D_FF, MAX_SEQ_LEN)
print(f"Actual total parameters: {sum(p.numel() for p in model.parameters())}")
# 18,388 -- about 0.017% of BERT-base's ~110 million parameters. Tiny, but real,
# and roughly double the attention-only estimate from the top of the article.
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

SEQ_LEN = 5  # actual words per sentence; the rest of MAX_SEQ_LEN is padding (id 0)

def make_batch(batch_size):
    # A fresh, randomly shuffled sequence every call -- so the model can't just
    # memorize one fixed input/output pair.
    src = torch.zeros(batch_size, MAX_SEQ_LEN, dtype=torch.long)
    tgt = torch.zeros(batch_size, MAX_SEQ_LEN, dtype=torch.long)
    for i in range(batch_size):
        seq = random.sample(range(1, VOCAB_SIZE), SEQ_LEN)
        src[i, :SEQ_LEN] = torch.tensor(seq)
        tgt[i, :SEQ_LEN] = torch.tensor(seq[::-1])
    return src, tgt

Now the training loop — 500 epochs, a brand-new random batch every time:

BATCH_SIZE = 32
for epoch in range(500):
    src, tgt = make_batch(BATCH_SIZE)
    optimizer.zero_grad()
    output = model(src) # Shape: [32, 10, 20]
    loss = criterion(output.reshape(-1, VOCAB_SIZE), tgt.reshape(-1))
    loss.backward()
    optimizer.step()
    if epoch % 100 == 0:
        print(f"Epoch {epoch}, Loss: {loss.item():.4f}")

Finally, the real test: sequences the model has never seen during training.

# Check generalization on sequences the model never trained on
model.eval()
with torch.no_grad():
    test_src, test_tgt = make_batch(200)
    test_out = model(test_src)
    predicted_ids = test_out.argmax(dim=-1)
    mask = test_tgt != 0
    accuracy = (predicted_ids[mask] == test_tgt[mask]).float().mean().item()
    print(f"Held-out per-token accuracy over 200 unseen sequences: {accuracy * 100:.1f}%")
    print(f"Example Target:    {test_tgt[0, :SEQ_LEN].tolist()}")
    print(f"Example Predicted: {predicted_ids[0, :SEQ_LEN].tolist()}")

• self.embedding = nn.Embedding(vocab_size, d_model) — A learned lookup table that maps integer word IDs (0–19) to dense d_model-dimensional vectors. These embeddings are trained from scratch — the model starts with random vectors and learns meaningful associations through backpropagation.
• self.pos_encoding = PositionalEncoding(d_model, max_len) — The sinusoidal positional encoding layer from above, added immediately after the embedding lookup.
• self.blocks = nn.ModuleList([...]) — A list of NUM_LAYERS Transformer blocks. nn.ModuleList (unlike a plain Python list) ensures PyTorch registers each block’s parameters for gradient computation and .to(device) transfers.
• self.fc_out = nn.Linear(d_model, vocab_size) — A final linear projection from d_model back to vocab_size, producing a logit for each word in the vocabulary. The highest logit at each position is the model’s predicted word.
• x = self.pos_encoding(self.embedding(x)) — Pipeline: raw integer IDs → learned embedding vectors → embedding + positional signal. The positional encoding is applied after (added to) the embedding, not concatenated.
• for block in self.blocks: x = block(x) — Passes the representation through each stacked Transformer block in sequence. Each block refines the representation by applying attention (context gathering) and feed-forward (transformation) with residual connections.
• sum(p.numel() for p in model.parameters()) — Sums the element count of every parameter tensor (weights and biases, across the embedding, both Transformer blocks, and the output layer) to get the model’s real total: 18,388. That’s more than double the 8,192 estimate from the top of the article, because that estimate only counted the attention projections.
• criterion = nn.CrossEntropyLoss() — Standard multi-class classification loss that internally applies log_softmax then computes negative log-likelihood. Used here because the model picks one word out of VOCAB_SIZE possibilities per position.
• optimizer = torch.optim.Adam(model.parameters(), lr=0.001) — Adam optimizer, a common default for Transformers. The 0.001 learning rate is typical for small models; larger models often use warmup schedules.
• make_batch(batch_size) — Builds a batch of random reversal examples: random.sample(range(1, VOCAB_SIZE), SEQ_LEN) draws 5 distinct token IDs without replacement, tgt is the same tokens in reverse order, and both are padded with 0 out to MAX_SEQ_LEN. Calling this fresh every epoch means the model sees a new sequence each time — the input at, say, position 0 is a different token on almost every call, so a model that ignores attention and just memorizes “position 0 → token X” cannot solve it.
• output = model(src) — Forward pass producing a tensor of shape [32, 10, 20] (batch=32, seq_len=10, vocab_size=20). Each of the 10 positions gets a distribution over 20 possible words.
• loss = criterion(output.reshape(-1, VOCAB_SIZE), tgt.reshape(-1)) — Reshapes output to [320, 20] and target to [320] (flattening batch and sequence dimensions) so CrossEntropyLoss can compare each position’s predicted distribution against the correct word ID.
• loss.backward() / optimizer.step() — Standard backprop + parameter update. Gradients flow through the residual connections, attention scores, and positional encoding back to the embedding table.
• model.eval() / torch.no_grad() — Switches off training-only behavior (not strictly necessary here since this model has no Dropout or BatchNorm, but it’s the right habit) and disables gradient tracking for the held-out check, since we’re only measuring accuracy, not training further.
• predicted_ids = test_out.argmax(dim=-1) — For each position, picks the vocabulary index with the highest logit — the model’s “best guess.”
• mask = test_tgt != 0 — Excludes the padding positions (id 0) from the accuracy calculation, so only the 5 real words per sequence count.
• accuracy = (predicted_ids[mask] == test_tgt[mask]).float().mean().item() — Per-token accuracy across 200 sequences the model has never seen during training — the actual generalization test.

What this actually means:

• A falling loss curve: The model’s weights are learning the general reversal pattern, not one fixed answer — the source sequence changes every batch, so there’s nothing fixed to memorize.
• Held-out accuracy near 100%: When the predicted sequence matches the target on sequences the model never trained on, that’s real evidence the MiniTransformer is using attention to look across the sentence — not just recalling a lookup table tied to position.

Building a Toy Transformer From Scratch vs. Starting with a Pretrained Model

Approach	What it does	Best for	Tradeoff
Build a tiny toy Transformer from scratch (this article)	Implement every component — embeddings, positional encoding, multi-head attention, residual connections, layer norm — by hand in a few dozen lines of PyTorch. Train on a toy task like sequence reversal.	Learning the mechanics; debugging; building the mental model that makes pretrained models debuggable instead of mysterious.	The resulting model is useless for real tasks. The toy vocabulary has 20 words; real language has tens of thousands. The embedding dimension is 32; production models use 512+. Scaling up would require enormous data and compute.
Jump straight to a pretrained model (e.g., HuggingFace)	Load a model already trained on billions of tokens, then fine-tune on your task.	Production; any real-world NLP task where you need actual performance, not understanding.	The internals are a black box. When training diverges, attention patterns misfire, or the model produces nonsensical output, you have no mental model of why — because you never built the pieces yourself or watched them learn.

What you gain from doing it from scratch once: Lina now understands exactly what happens when a token flows through a Transformer — which projection multiplies it, which normalization centers it, which attention weights mix it. When she later loads a pretrained model for BookSight’s purchase-prediction task, the “black box” becomes a transparent pipeline she can debug, not a mystery she can only shrug at.

Recap

What we covered in this part:

1. Embeddings & Position: Giving words a location and a sequence index.
2. Multi-Head Attention: Letting the model view a sentence through multiple “expert” lenses.
3. Transformer Blocks: Stacking layers with residual connections and normalization to keep learning stable.
4. Training: Using a simple task like sequence reversal — on randomly generated sequences, so success can only come from genuine attention — to prove the architecture works.

Next, Lina confirms her hunch: training a Transformer from scratch on BookSight’s real review data would take forever and cost a fortune. So instead of building bigger from scratch, she’s going to borrow a brain — a model someone else already spent thousands of GPU-hours training — and teach it only the one thing it doesn’t already know.

Check Your Understanding

Questions below move from simple recall up to open-ended design, roughly following Bloom’s Taxonomy.

Remember What are the two “vital tricks” used inside a Transformer Block, and what does each one do?

Understand In your own words, explain why the MiniTransformer still needs a PositionalEncoding layer even though it already has a MultiHeadAttention layer.

Apply Using the article’s head_dim = d_model // num_heads formula and the scaling factor sqrt(head_dim) from the attention score calculation, compute both values for this article’s MiniTransformer, where D_MODEL = 32 and NUM_HEADS = 4.

Analyze The TransformerBlock.forward method adds the block’s input back to its output (x = self.norm1(x + self.attention(x))) instead of just using self.attention(x) directly. Walk through what could go wrong during training of a multi-layer model like this one if that residual connection were removed.

Evaluate The article tests the MiniTransformer on Sequence Reversal, a task where the model must look at the last word to predict the first one. Critique this choice of task: what does successfully reversing a random 5-token sequence actually prove about the model’s use of attention and position, and what does it not prove that a real language task would?

Create Design a second toy task (not sequence reversal) that would specifically test whether the MiniTransformer’s Multi-Head Attention—not just its Positional Encoding—is doing useful work. What input/output pairs would make attention necessary?

---

Related articles

• Multi-Head Attention and Positional Encoding: Giving the Transformer a Sense of Direction)
• Borrowing Brains: A Beginner’s Guide to Transfer Learning)

References & Further reading

• Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). “Attention Is All You Need.” Advances in Neural Information Processing Systems (NeurIPS 2017). — The foundational paper that introduced the Transformer architecture: the encoder-decoder structure, scaled dot-product attention, multi-head attention, sinusoidal positional encoding, residual connections, and layer normalization — every component assembled in this article’s MiniTransformer.
• PyTorch Documentation: torch.nn.Module, torch.nn.LayerNorm, torch.nn.Embedding — Official library docs for the PyTorch primitives used throughout the MiniTransformer implementation.

---