The Attention Mechanism, Finally Explained (Without the Matrix Algebra)

In Part 6, Lina gave her recurrent networks a memory. LSTMs and GRUs solved the vanishing gradient problem by using gates to protect important information as it flows through time. Even with perfect memory, a network still has to process every single word in a sentence to understand it.

Imagine Lina is reviewing a 500-page-view customer session. Her LSTM has a perfect memory of every page. When she gets to page 437 and sees “the customer added a premium warranty,” she doesn’t need to re-read all 436 previous pages to understand why. She jumps straight to the parts that matter—maybe the moment they viewed the expensive product it protects. She attends to the relevant parts.

That’s what attention does. It lets a neural network skip the irrelevant stuff and focus on what matters right now. It works so well, in fact, that it’s become the foundation of every major language model today, from GPT to Claude. Let’s build it from scratch—no matrix algebra required.

1. The Library Search Problem

Think of a traditional neural network as a librarian with a terrible filing system. You ask for a book about “apple pie recipes.” Instead of pulling just the pie books, she tries to recall everything in the library at once—cookbooks, history books, novels—and then somehow figures out which parts matter for your question.

That’s exhausting and inefficient. What if the librarian had a smart search system instead? You type “apple pie recipes” into a search bar, and the system finds only the books whose titles and tags match. She pulls those. The rest she ignores.

Attention is exactly that search system. It lets the network ask which words it has seen so far are most relevant to what it is doing right now.

Start with the simplest version. Take the sentence “The manager approved the billing charge.” When the network processes the word “charge,” it needs to figure out whether this is a billing charge or a battery charge. The answer depends on nearby words like “manager” and “billing.”

The ideal version of attention is a simple lookup table.

# Our sentence: "The manager approved the billing charge."
# Let's represent each word as a simple number (in reality, it's a vector)
sentence = {
    'The': 0.1,
    'charge': 0.5,
    'manager': 0.7,
    'approved': 0.6,
    'the': 0.1,
    'billing': 0.8
}

# When we're processing 'charge', we ask: which words are most relevant?
# In a perfect world, we'd have a lookup table that tells us
relevance_to_charge = {
    'The': 0.0,        # Not relevant
    'charge': 1.0,     # Very relevant (it's the word itself)
    'manager': 0.9,    # Very relevant (tells us what kind of charge)
    'approved': 0.6,   # Somewhat relevant
    'the': 0.0,        # Not relevant
    'billing': 0.8     # Very relevant (financial context)
}

# Now we take a weighted average of all the words
attended_representation = 0.0
for word, relevance in relevance_to_charge.items():
    attended_representation += sentence[word] * relevance

print(f"Attended representation of 'charge': {attended_representation:.2f}")

• sentence = {...} — A dictionary mapping each word in the sentence to a scalar value. In a real system, each word would be a high-dimensional vector (e.g., 512 numbers), not a single float.
• relevance_to_charge = {...} — A hardcoded lookup of how relevant each word is to “charge.” In a real attention mechanism, these weights are learned from the data using Query–Key interactions, not handcrafted.
• for word, relevance in relevance_to_charge.items() — Iterates over each word–relevance pair in the dictionary.
• attended_representation += sentence[word] * relevance — Multiplies each word’s value by its relevance weight and accumulates the sum. This is the core “weighted average” that attention computes — the same operation as a dot product between the relevance weights and the word values.

What this actually means: We multiplied each word’s value by how relevant it is, then added them all up. The result is a new representation of “charge” enriched with information from the most relevant context. The words “manager” and “billing” pulled the meaning toward “billing charge” rather than “battery charge.”

One problem: there is no magic lookup table that tells us each word’s relevance. We have to learn it. That’s where queries, keys, and values come in.

2. Queries, Keys, and Values: The Three Musketeers

This is where most people get tripped up. Forget the terminology for a moment and think about what’s actually going on.

Say you’re on YouTube. You type “cat videos” into the search bar. The algorithm needs to:

1. Understand what you’re looking for (“cat videos”)
2. Check every video’s title and tags to see if they match
3. Return the actual videos that match

Attention works the same way:

1. Query (Q): What you’re looking for (“cat videos”)
2. Key (K): The video titles and tags (what the algorithm checks against)
3. Value (V): The actual video content (what you watch once you find a match)

The trick: all three come from the same source—the input sentence. We just transform it three different ways.

import numpy as np

# Let's say we have a word represented as a vector
# (In reality, this would be a 512-dimensional vector from a neural network,
# but we'll use 4 dimensions for simplicity)
word_embedding = np.array([0.5, 0.2, 0.8, 0.1])

# We create three different "views" of this word
# by multiplying it by three different learned weight matrices
# (These weights are trained by the network)

W_q = np.array([[0.1, 0.2, 0.3, 0.4],
                 [0.5, 0.6, 0.7, 0.8]])

W_k = np.array([[0.9, 0.8, 0.7, 0.6],
                 [0.5, 0.4, 0.3, 0.2]])

W_v = np.array([[0.1, 0.1, 0.1, 0.1],
                 [0.2, 0.2, 0.2, 0.2]])

# The Query: "What am I looking for?"
query = W_q @ word_embedding
print(f"Query: {query}")

# The Key: "What am I?"
key = W_k @ word_embedding
print(f"Key: {key}")

# The Value: "Here's my actual content"
value = W_v @ word_embedding
print(f"Value: {value}")

• word_embedding = np.array([0.5, 0.2, 0.8, 0.1]) — A 4-dimensional vector representing one word. Real systems use 512+ dimensions, but 4 is enough to see the mechanics.
• W_q, W_k, W_v — Three learned weight matrices, each shaped (2, 4). They transform the same 4D embedding into three different 2D “views”: Query, Key, and Value. These matrices are what the network trains.
• query = W_q @ word_embedding — Matrix multiplication: the (2, 4) matrix times the (4,) vector produces a (2,) vector. This is the “what am I looking for?” view.
• key = W_k @ word_embedding — Same operation with W_k, producing the “what am I?” view.
• value = W_v @ word_embedding — Same operation with W_v, producing the “here’s my actual content” view.
• The @ operator is NumPy’s matrix multiplication operator, equivalent to np.matmul().

What this means in practice: We took one word and created three different versions of it. The Query asks “What information do I need?” The Key says “Here’s what I contain.” The Value says “Here’s the actual data.”

The hard part is internalizing that Q, K, and V are just different perspectives on the same data. Not three separate things. The same word wearing three different masks. The network learns what each mask should look like so the Query can find the most relevant Keys and pull out their Values.

3. The Similarity Score: How Close Are We?

Now we have a Query (what we’re looking for) and a set of Keys (what each word contains). We need a way to measure: “How much does this Key match this Query?”

The math here is straightforward. Multiply the Query and the Key together. If they point in the same direction — both have large positive numbers in the same positions — the result is large. If they point in opposite directions, the result is small or negative.

Let’s see it in action.

import numpy as np

# Our Query: what we're looking for
query = np.array([1.0, 0.5])

# Keys from different words in the sentence
key_manager = np.array([0.9, 0.6])   # Similar to query
key_battery = np.array([0.1, 0.2])   # Different from query (the wrong meaning)
key_billing = np.array([0.95, 0.55]) # Very similar to query

# The similarity score is just the dot product (multiply and sum)
def similarity_score(query, key):
    return np.dot(query, key)

print(f"Similarity to 'manager': {similarity_score(query, key_manager):.2f}")
print(f"Similarity to 'battery': {similarity_score(query, key_battery):.2f}")
print(f"Similarity to 'billing': {similarity_score(query, key_billing):.2f}")

• query = np.array([1.0, 0.5]) — The query vector for the word “charge”: what it’s “looking for” in the rest of the sentence.
• key_manager, key_battery, key_billing — Key vectors for three candidate words. Each is a 2D vector; the closer it is to the query, the higher the similarity score will be.
• def similarity_score(query, key): return np.dot(query, key) — The dot product: multiply corresponding elements, then sum. This is the standard way to measure how much two vectors “point in the same direction.”
• np.dot([1.0, 0.5], [0.9, 0.6]) = 1.0×0.9 + 0.5×0.6 = 0.9 + 0.3 = 1.2 — The “manager” score is high because the key vector is similar to the query.
• np.dot([1.0, 0.5], [0.1, 0.2]) = 0.1 + 0.1 = 0.2 — The “battery” score is low because the key vector points in a very different direction.

What this actually means: The score for “manager” is 1.20. For “battery” it’s 0.20. For “billing” it’s 1.23. These numbers tell the model where to focus — the higher the score, the more relevant the word is to the query.

But these raw scores come in all different sizes, and comparing them directly is tricky. What if we had a sentence with 100 words? We’d get 100 different numbers — some huge, some tiny. The model would struggle to interpret them. That’s where softmax comes in.

4. The Softmax: Turning Scores into Percentages

Softmax takes any list of numbers and converts them into percentages that sum to 1.0 (or 100%). Think of it as a volume knob: it amplifies the big numbers and pushes the small ones toward the background.

Here’s the problem: raw similarity scores can be any size. One word might score 0.2, another 50. The network has no way to interpret that range. But convert them to percentages—“This word gets 5% of my attention, that one gets 60%“—and the picture sharpens.

import numpy as np

# Our similarity scores from the previous section
scores = np.array([1.2, 0.2, 1.225])

# Softmax formula: for each score, calculate e^score, then divide by the sum
def softmax(scores):
    # First, exponentiate each score
    exp_scores = np.exp(scores)
    # Then divide each by the sum of all exponentials
    return exp_scores / np.sum(exp_scores)

attention_weights = softmax(scores)
print(f"Attention weights: {attention_weights}")
print(f"Sum of weights: {np.sum(attention_weights):.4f}")
print(f"As percentages: {attention_weights * 100}")

• scores = np.array([1.2, 0.2, 1.225]) — Raw similarity scores from the dot products in the previous section. These are unbounded — they could be any positive or negative number.
• np.exp(scores) — Exponentiates each score: $e^{1.2}$, $e^{0.2}$, $e^{1.225}$. This is what makes softmax “soft”: bigger scores get exponentially bigger shares — but how much bigger depends on how far apart the raw scores already were (more on that below, where these particular scores turn out to be too close together for the gap to widen).
• exp_scores / np.sum(exp_scores) — Divides each exponentiated score by the total so all weights sum to exactly 1.0. This is what turns raw scores into “percentages of attention.”
• np.sum(attention_weights) — Verifies the weights sum to 1.0 by construction.
• attention_weights * 100 — Displays the weights as percentages for human readability (42% instead of 0.42).

What this actually means: The raw scores were [1.2, 0.2, 1.225]. After softmax, they became roughly [0.42, 0.15, 0.43]. Pay about 42% attention to “manager,” 15% to “battery,” and 43% to “billing.” They add up to 100%.

Notice the pattern: “manager” and “billing” started close together (1.2 vs. 1.225) and stayed close as weights (42% vs. 43%). Now look closer at “battery”: its raw score was 6× smaller than “manager“‘s (0.2 vs. 1.2), but after softmax it still holds 15% of the attention — only about 2.7× smaller than “manager“‘s 42%. That gap narrowed, it didn’t widen. Softmax isn’t a magnifying glass that automatically stretches every gap — how much it sharpens depends on how far apart the raw scores already are. When scores sit close together, like these three do, softmax stays gentle and keeps attention spread across all of them; it only sharpens hard once scores are separated by a few units or more. Here, that gentleness is doing real work: “battery” is genuinely the weak match, but it isn’t zeroed out. A small amount of attention stays on every word, which matters once a later sentence turns out more ambiguous than this one.

5. Putting It All Together: The Weighted Sum

Now we have attention weights for each word. We take those weights, multiply by the actual word values (the V’s from earlier), and sum them up. The result: a new representation of our original word, enriched with context.

Let’s run the full calculation end-to-end.

import numpy as np

# Step 1: Create Q, K, V for each word in our sentence
# (In reality, these come from a neural network. We'll use simple numbers.)

words = ['charge', 'manager', 'billing']

# Queries, Keys, and Values (simplified as 2D vectors)
queries = {
    'charge': np.array([1.0, 0.5]),
    'manager': np.array([0.8, 0.6]),
    'billing': np.array([0.9, 0.7])
}

keys = {
    'charge': np.array([0.95, 0.55]),
    'manager': np.array([0.9, 0.6]),
    'billing': np.array([0.92, 0.65])
}

values = {
    'charge': np.array([0.5, 0.2]),
    'manager': np.array([0.7, 0.4]),
    'billing': np.array([0.8, 0.5])
}

# Step 2: Calculate similarity scores
query_charge = queries['charge']
scores = []
for word in words:
    score = np.dot(query_charge, keys[word])
    scores.append(score)

scores = np.array(scores)
print(f"Raw similarity scores: {scores}")

# Step 3: Apply softmax to get attention weights
def softmax(x):
    return np.exp(x) / np.sum(np.exp(x))

attention_weights = softmax(scores)
print(f"Attention weights: {attention_weights}")

# Step 4: Weighted sum of values
attended_output = np.zeros(2)  # Same dimension as our values
for i, word in enumerate(words):
    attended_output += attention_weights[i] * values[word]

print(f"Original 'charge' value: {values['charge']}")
print(f"After attention: {attended_output}")
print(f"\nWhat happened: The word 'charge' got enriched with context from 'manager' and 'billing'.")
print(f"The output is closer to the meaning of 'billing charge' because those words dominated the attention.")

• words = ['charge', 'manager', 'billing'] — The three words in our toy sentence that we’ll run attention over.
• queries, keys, values — Three dictionaries, each mapping a word to its Q, K, or V vector (2D). In a real transformer, these come from learned linear projections of the input embeddings.
• query_charge = queries['charge'] — The query for the word “charge”: what it’s “looking for” in the rest of the sentence.
• for word in words: score = np.dot(query_charge, keys[word]) — Computes the dot-product similarity between the query and every key, producing one raw score per word.
• scores = np.array(scores) — Converts the list of scores into a NumPy array so we can apply softmax vectorized.
• attention_weights = softmax(scores) — Converts raw scores into percentages that sum to 1.0.
• attended_output = np.zeros(2) — Initializes the output as a zero vector with the same dimensionality as the value vectors (2D).
• for i, word in enumerate(words): attended_output += attention_weights[i] * values[word] — The weighted sum: each value is multiplied by its attention weight and accumulated. This is the final “attended” representation.
• enumerate(words) — Yields (index, word) pairs so we can index into attention_weights by position while also looking up values by name.

What this actually means: We started with “charge” as [0.5, 0.2]. After attention, it became roughly [0.67, 0.37]. The numbers shifted because we mixed in information from “manager” and “billing.” The attention weights here—about 33%, 33%, and 34%—are all fairly close. In this simplified example, the three key vectors happen to be similar to the query. A trained network would usually produce a sharper split once its queries and keys reflect real learned patterns. Put “charge” in a different context—“the phone’s battery charge was low”—and attention would shift toward “battery” instead. The output would be different.

This is the magic of attention: the same word gets a different representation depending on context. The network doesn’t need to memorize every meaning of “charge.” It just learns to blend the meanings of nearby words based on how relevant they are.

The Full Picture

Here’s what we’ve built, step by step:

1. Query: “What am I looking for?” (derived from the current word)
2. Keys: “What does each word contain?” (derived from all words in the sentence)
3. Similarity: “How much does each key match my query?” (dot product)
4. Softmax: “Turn those scores into percentages” (so they add up to 100%)
5. Weighted Sum: “Blend the values based on the percentages” (get the final output)

The whole thing is called Scaled Dot-Product Attention. The “scaled” part refers to a detail we skipped: in practice, we divide the similarity scores by the square root of the dimension before applying softmax. This keeps the numbers from getting too large, which would shrink gradients and stall training. The intuition stays the same, though.

Scaled Dot-Product Attention

The full attention operation, in one line:

$\text{Attention}(Q, K, V) = \text{softmax}\!\left(\frac{QK^T}{\sqrt{d_k}}\right)V$

Plain English	Statistical symbol	Python equivalent
Query (what am I looking for)	$Q$	query / queries
Key (what does each word contain)	$K$	key / keys
Value (the actual content to blend)	$V$	value / values
Raw similarity scores (dot products)	$QK^T$	np.dot(query, key)
Key dimension (number of features per vector)	$d_k$	len(key)
Scaling factor (keeps scores stable)	$\sqrt{d_k}$	np.sqrt(len(key))
Attention weights (percentages summing to 1)	$\text{softmax}(\cdot)$	softmax(scores)
Final attended output	$\text{Attention}(Q,K,V)$	attended_output

The scaling by $\sqrt{d_k}$ prevents the dot-product scores from growing too large when the key dimension is high — without it, softmax would saturate and gradients would vanish (the same problem Lina saw in Part 5, resurfacing in a new form).

Attention vs. RNN/LSTM: Which Approach for Long-Range Dependencies?

Approach	What it does	Best for	Tradeoff
RNN / LSTM	Processes tokens one at a time, carrying a hidden state (and cell state for LSTM) across steps. Signal must survive every intermediate step to reach the end.	Short-to-medium sequences; tasks where temporal order is inherently sequential; low-memory settings where you can’t afford $O(n^2)$ compute.	Signal degrades over long distances even with gates. Can’t “skip” irrelevant steps — the model must process every token between the signal and the point where it’s needed.
Attention	Computes relevance scores between every pair of tokens simultaneously, then blends values based on those scores. The model “jumps straight” to relevant context.	Long sequences where specific distant context matters; parallelizable training (no sequential dependency between steps).	$O(n^2)$ memory and compute — every token attends to every other token. No inherent word-order awareness without positional encoding.

Rule of thumb: If the key signal is 500 steps back, an LSTM has to protect it through 500 intermediate updates — that’s 500 chances for it to degrade. Attention simply computes a relevance score between the current token and that distant token directly, in one step. The cost is quadratic scaling and no built-in sense of order. Lina would reach for attention when she needs to find specific signal across a long session without relying on it surviving a chain of gates.

Next in this series, Lina will see how to run this attention mechanism multiple times in parallel (called “multi-head attention”) and how to stack these layers to build a full Transformer. For now, you’ve got the core idea: attention is a search system that lets neural networks focus on what matters.

What you learned:

• Attention focuses on relevant parts of the input
• Query, Key, and Value are three learned transformations of the same data
• Similarity scores measure how much a Key matches a Query
• Softmax converts scores into percentages (attention weights)
• The final output is a weighted sum of Values, guided by attention weights
• The same word gets different representations depending on context

Lina has already spotted a gap, though. Basic attention treats the sentence like a bag of words—it doesn’t care about word order at all. “The manager approved the billing charge” and “The billing charge approved the manager” would look nearly identical to plain attention. In Part 8, Lina will fix that with positional encoding and split this single attention head into many. Multi-head attention lets the network attend to different aspects of the data at once. See you there.

Check Your Understanding

Questions below move from simple recall up to open-ended design, roughly following Bloom’s Taxonomy.

Remember What do Query, Key, and Value each represent in the attention mechanism?

Understand In your own words, explain why the word “charge” needs a different vector representation depending on whether the sentence is about billing or about a phone battery.

Apply Using the article’s similarity-score method (the dot product of Query and Key), calculate the similarity between query = [0.5, 1.0] and key = [0.8, 0.3].

Analyze In the article’s full worked example, the three attention weights (about 33%, 33%, and 34%) ended up nearly equal, even though the raw similarity scores weren’t identical. Walk through what a real transformer’s Query and Key vectors would need to look like differently for the model to confidently decide “charge” means “billing charge,” instead of landing on a near-even split.

Evaluate Softmax always makes the attention weights sum to 100%. Critique this design choice: what happens if none of the words in a sentence are actually relevant to the word being processed—does forcing the weights to sum to 100% handle that gracefully, or does it create a problem?

Create Design a 3-word example sentence (different from “the manager approved the billing charge”) where a word is genuinely ambiguous between two very different meanings, the way “charge” is in this article. Name both meanings and which nearby words would disambiguate each one.

---

Related articles

• LSTMs and GRUs: Giving Networks a Memory)
• Multi-Head Attention and Positional Encoding: Giving Transformers a Sense of Order)

References & Further reading

• Bahdanau, D., Cho, K., & Bengio, Y. (2014). “Neural Machine Translation by Jointly Learning to Align and Translate.” Proceedings of ICLR 2015. — The paper that introduced additive attention, the first attention mechanism applied to neural machine translation, enabling the decoder to “look back” at relevant source words instead of relying on a fixed-length context vector.
• Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). “Attention Is All You Need.” Advances in Neural Information Processing Systems (NeurIPS 2017). — The paper that introduced the Transformer architecture, replacing recurrence entirely with scaled dot-product attention and multi-head attention.

---