Why LLMs Forget the Middle: Understanding Context Windows and Lost in the Middle

Last time, Rae built her first complete RAG pipeline for her support bot. She chunked the 500-page product manual, embedded each chunk with a sentence-transformer model, and retrieved the right passages with FAISS when a customer asks a question. Her bot started answering real customer questions from the manual. But she soon noticed something odd. Sometimes the bot misses a fact she can see right there in the retrieved chunks—it’s just buried in the middle of a long passage. The retrieval worked, the chunk was relevant, but the LLM didn’t use it.

1. The Problem: Your LLM Reads Like a Tired Student

Rae noticed something odd with her support bot. When a customer asked about a feature mentioned near the start or end of the retrieved context, the bot answered correctly. But when the same answer was buried in the middle of a long passage—even one she could see right there in the retrieved chunks—the bot would hallucinate or say “I don’t know.” It’s like spending hours reading a technical manual, only to realize you remember the introduction and conclusion perfectly, but the middle is a blurry fog.

Large Language Models (LLMs) do the same thing. We often talk about “context windows” as if they’re buckets that hold information with perfect clarity until they overflow. The reality is messier. Even when a model can technically see a piece of information, it often ignores it if that information sits in the middle of a long prompt.

This isn’t a bug. It’s a fundamental behavior of the attention mechanism. Researchers call this the “Lost in the Middle” phenomenon. So what happens when we hide a simple fact inside a long document?

import openai
import os

# Imagine we have a long document about a fictional company.
# We will place a specific fact (The CEO's favorite color) at different positions.
def test_retrieval_position(fact_position="middle"):
    fact = "The CEO's favorite color is Electric Purple."
    filler = "The company produces high-quality widgets for the global market. " * 100
    
    if fact_position == "start":
        prompt = f"{fact}\n{filler}"
    elif fact_position == "end":
        prompt = f"{filler}\n{fact}"
    else:
        prompt = f"{filler[:len(filler)//2]}\n{fact}\n{filler[len(filler)//2:]}"
    
    prompt += "\nQuestion: What is the CEO's favorite color? Answer in one word."
    
    # In a real test, you'd call an API here.
    # For this example, we simulate the common failure mode.
    print(f"Testing fact at: {fact_position.upper()}")
    # If position is middle, models often hallucinate or say 'I don't know'
    return "Electric Purple" if fact_position != "middle" else "Unknown/Blue"

Now run it with the fact at each of the three positions:

print(test_retrieval_position("start"))
print(test_retrieval_position("middle"))
print(test_retrieval_position("end"))

This block simulates the “Lost in the Middle” problem by placing a target fact (“The CEO’s favorite color is Electric Purple.”) at three different positions in a long document.

• fact = "The CEO's favorite color is Electric Purple." — the target fact to hide inside a long document.
• filler = "..." * 100 — the filler variable creates a 100×-repeated sentence to simulate a long context.
• if fact_position == "start": prompt = f"{fact}\n{filler}" — the fact goes before the filler.
• elif fact_position == "end": prompt = f"{filler}\n{fact}" — the fact goes after the filler.
• else: prompt = f"{filler[:len(filler)//2]}\n{fact}\n{filler[len(filler)//2:]}" — the fact is inserted at the halfway point, using filler[:len(filler)//2] (first half) and filler[len(filler)//2:] (second half) to split the filler around it.
• return "Electric Purple" if fact_position != "middle" else "Unknown/Blue" — the function returns "Electric Purple" for start and end positions but "Unknown/Blue" for the middle position—simulating the real-world failure mode where models accurately recall facts at the edges but miss them in the center.

In a real test, you’d replace the simulated return with an actual LLM API call and measure the difference.

The model usually nails the answer when the fact sits at the start or end. Put it in the middle, though, and accuracy drops—sometimes by over 50%. This is the “U-shaped” performance curve that haunts long-context applications.

2. What a Context Window Actually Is

The context window is a hard ceiling on the model’s short-term memory. It’s measured in tokens—the basic units of text the model processes.

One token runs about 0.75 words. So a 4,000-token window sounds roomy, but that’s only ~3,000 words—roughly one long blog post. Push past the limit and the model can’t see the extra text. It either throws an error or, more often, drops the earliest parts of the conversation to make room for what’s new.

⚠️ Requires: pip install tiktoken

import tiktoken

def count_tokens(text, model="gpt-4"):
    encoding = tiktoken.encoding_for_model(model)
    tokens = encoding.encode(text)
    return len(tokens)

text = "Data science is the study of data to extract meaningful insights for business."
num_tokens = count_tokens(text)
print(f"Token count: {num_tokens}")
# For this sentence, it's 15 tokens. 
# If your limit is 8192, you've used 0.18% of your budget.

This block uses tiktoken to count tokens in a sample sentence.

• tiktoken.encoding_for_model("gpt-4") — loads the tokenizer specific to GPT-4’s vocabulary (the same cl100k_base encoding used in the RAG pipeline article).
• encoding.encode(text) — converts the string into a list of token IDs—each integer represents one token.
• len(tokens) — gives the total count.
• The sample sentence — produces 15 tokens, and the comment notes that against an 8,192-token limit, that’s only 0.18% of the budget—illustrating how small a single sentence is relative to even a modest context window.

The broader point: when Rae retrieves 5–10 chunks of product-manual text and stacks them into a prompt, she might be using 2,000–5,000 tokens—well within the limit, but large enough for the “Lost in the Middle” effect to kick in.

Here’s the thing, though. Even when you stay within the limit, the model doesn’t weight every token equally. It has a soft attention budget, and it spends most of that budget on the edges.

3. Attention: Why the Model Focuses on the Edges

Why does this happen? It’s structural. Transformers use Position Embeddings to track where a word sits in a sentence.

1. The Start Advantage: The first few tokens are the “anchor.” Every subsequent token looks back at them, so they set the tone and the topic.
2. The End Advantage: The last few tokens are the most recent. When the model is about to generate an answer, the tokens right next to the “Question:” prompt are the freshest in its mathematical memory.
3. The Middle Slump: Middle tokens sit far from both the start and the output. In a sea of 30,000 tokens, a sentence at token 15,000 is just “noise” the model has to sift through.

4. The Empirical Evidence: Where Models Actually Look

The Lost in the Middle paper by Liu et al. (2023) tested models like GPT-3.5 and Claude. The results were stark. As the input context grows, a model’s ability to retrieve a specific fact from the middle plummets.

So a model with a 32,000-token window might be less accurate at finding information than one with an 8,000-token window. More space just means more “middle” for the information to get lost in.

5. Why This Matters: Real-World Consequences

This isn’t just a theoretical quirk. It breaks real systems:

• RAG (Retrieval-Augmented Generation): Put the most relevant of 10 retrieved documents in the middle of the prompt, and the LLM may miss it entirely.
• Legal Analysis: A critical clause buried on page 40 of a 100-page contract? The model might summarize the whole document as if that clause doesn’t exist.
• Long Conversations: The model can forget specific instructions from ten minutes ago once the conversation has moved on.

6. Workarounds: How to Work Around Lost in the Middle

Since we can’t easily change how Transformers work, we have to change how we talk to them. A few strategies that help:

• Strategy 1: Reordering. Put the most important data at the start or end of the prompt.
• Strategy 2: Chunking. Rather than one giant prompt, split the task into three smaller ones.
• Strategy 3: Reranking. Use a smaller, faster model to rank documents by relevance, then place the top 3 at the start of your prompt.

def reorder_context(chunks, query_relevance_scores):
    # Sort chunks so the most relevant are at the 'edges'
    sorted_chunks = [x for _, x in sorted(zip(query_relevance_scores, chunks), reverse=True)]
    
    # Put best chunk first, second best last, third best second, etc.
    final_context = []
    for i, chunk in enumerate(sorted_chunks):
        if i % 2 == 0:
            final_context.insert(0, chunk)
        else:
            final_context.append(chunk)
    return "\n".join(final_context)

chunks = ["Chunk A (Low)", "Chunk B (High)", "Chunk C (Med)"]
scores = [0.1, 0.9, 0.5]
print(reorder_context(chunks, scores))
# Result puts 'Chunk B' at the start where attention is highest.

This block reorders retrieved chunks to place the most relevant ones at the edges of the context—where attention is highest.

• sorted(zip(query_relevance_scores, chunks), reverse=True) — pairs each score with its chunk and sorts by score descending.
• [x for _, x in ...] — the list comprehension extracts just the sorted chunks, discarding the scores (the _ is a throwaway variable).
• if i % 2 == 0: final_context.insert(0, chunk) — even-indexed chunks (0th, 2nd, 4th…) are inserted at position 0, pushing them to the front.
• else: final_context.append(chunk) — odd-indexed chunks (1st, 3rd, 5th…) are appended to the end.
• This “zipper” pattern — puts the highest-relevance chunk first, the second-highest last, the third-highest second, and so on—mirroring the U-shaped attention curve.
• With the sample data — “Chunk B” (score 0.9) ends up at the start where the model pays the most attention, “Chunk C” (score 0.5) goes to the end, and “Chunk A” (score 0.1) sits in the middle where it matters least.

7. Why Newer Models Are Better (But Not Perfect)

GPT-4o and Claude 3.5 Sonnet have much larger context windows (128k to 200k+). They use techniques like “LongRoPE” or better training data to help the model stay focused. These are significantly better than models from two years ago. Still, the U-shaped curve persists. You can’t assume a 128k window means 100% recall across the whole 128k.

Lost in the Middle: Mitigate It or Switch to a Bigger Window?

Rae has two broad options when her support bot misses facts in the middle of retrieved context.

Option A: Mitigate with reordering, fewer chunks, and reranking. Keep your current model and context window. Instead, reorder retrieved chunks so the most relevant sit at the edges (start and end) of the prompt—the reorder_context function from Section 6 is a lightweight version. Retrieve fewer but higher-quality chunks (top-3 instead of top-20) to shrink the “middle.” Add a reranking step: retrieve 10 chunks with a fast embedding model, then use a slower cross-encoder to pick the best 3 and place them at the edges. This costs engineering effort but no extra per-query API spend. Best when your retrieved context is already manageable (under ~10k tokens) and you just need to nudge the model toward the right passages.

Option B: Switch to a bigger context window. Use a model like GPT-4o or Claude 3.5 Sonnet with 128k–200k token windows. Paste more—or all—of your retrieved context in one shot. Less engineering effort, but the U-shaped curve still applies: even at 128k tokens, facts in the middle of the window get less attention than facts at the edges. Processing 128k tokens per query is also slower and more expensive per call. Best when your documents genuinely need the full window and you’re willing to trade cost for simplicity.

When Option A wins: You’re on a startup budget, your retrieved context is under ~10k tokens, and you care about latency and cost per query. The reordering and reranking overhead is small compared to the cost of feeding 128k tokens to a frontier model for every customer question—which is exactly Rae’s situation with her product manual.

When Option B wins: Your documents are so long that RAG can’t retrieve well (e.g., the answer requires synthesizing information from 20+ distant sections), and you’re willing to pay for the bigger window. But remember: a bigger window lowers the ceiling for lost-in-the-middle errors; it doesn’t eliminate the U-shaped curve.

Rae’s current choice: Her product manual’s top-5 retrieved chunks are well under 10k tokens total. Reordering and reranking is the cheaper fix for now. But the existence of 128k+ windows keeps nagging at her—if the model can technically read the whole manual, why bother with retrieval at all?

8. The Hard Part: Why This Is Genuinely Difficult to Fix

The hard part: Attention is expensive.

Standard attention has “quadratic” cost. Double the text length and the computational work quadruples. To handle long windows, engineers use “sparse attention” — only looking at some tokens. The model runs faster, but it’s also more likely to “skip” the middle tokens. We trade perfect memory for the ability to read long documents at all.

Standard self-attention computes a relationship between every pair of tokens in the input. For a sequence of $n$ tokens, the attention matrix is $n \times n$—meaning the computation scales quadratically:

$\text{Attention cost} \propto n^2$

If you double the sequence length from $n$ to $2n$, the cost goes from $n^2$ to $(2n)^2 = 4n^2$—four times the work for twice the text.

Plain English	Statistical symbol	Python equivalent
Number of tokens in the input	$n$	len(tokens)
Attention computation cost (quadratic)	$O(n^2)$	n ** 2
Doubling the sequence length quadruples the cost	$(2n)^2 = 4n^2$	(2 * n) ** 2
Sparse attention only computes $k$ neighbors per token	$O(n \cdot k)$, where $k \ll n$	n * k (where k << n)

This is why engineers use “sparse attention” to make long windows feasible: they trade $O(n^2)$ for $O(n \cdot k)$ where $k$ is a fixed number of neighbors per token. But this is exactly what makes it easier for the model to “skip” middle tokens—by not computing attention between distant pairs, some information in the middle never gets amplified. We are trading perfect memory for the ability to read long documents at all.

9. What to Do: Practical Guidance for Your Use Case

A quick decision tree for your next project:

1. Under 2,000 tokens? Don’t sweat it — the model will likely see the whole prompt.
2. 2,000 to 10,000 tokens? Put your key instructions at the very bottom, right before the model starts generating.
3. Over 10,000 tokens? Use a Reranker so the most relevant info isn’t buried in the middle, or split things with a “Map-Reduce” approach (summarize parts separately first).

10. Looking Ahead: What’s Changing

Researchers are developing “Linear Attention” and “State Space Models” (like Mamba) that drop this quadratic cost. Someday we may have models that treat the middle of a document with the same respect as the beginning. For now, though, position matters.

11. Recap and Next Steps

• Lost in the Middle means LLMs struggle with information in the center of long prompts.
• Attention favors the start (context) and the end (recency).
• Workaround: Keep your most important data at the edges of your prompt.

But the bigger-window alternative keeps nagging at Rae. GPT-4o and Claude 3.5 Sonnet handle 128k–200k tokens. So why bother with chunking, embedding, and retrieval at all? Why not paste the entire product manual into the prompt and let the model read it? In the next article, we’ll pit RAG against long-context models head-to-head—and see which one actually wins for Rae’s support bot in 2026.

Check Your Understanding

Questions below move from simple recall up to open-ended design, roughly following Bloom’s Taxonomy.

Remember What is the “Lost in the Middle” phenomenon, and what does the U-shaped performance curve describe?

Understand Explain in your own words why attention creates a “start advantage” and an “end advantage,” and why tokens in the middle of a long prompt are treated as noise.

Apply Using the decision tree from Section 9, you have a 15,000-token prompt where the single most important instruction sits at token 7,000. Which of the article’s workarounds should you apply, and how would you implement it?

Analyze Section 4 claims a model with a 32,000-token window can be less accurate at retrieving a fact than an 8,000-token model. Walk through the reasoning: why does a larger context window create more “middle” rather than simply more room for information?

Evaluate The article’s reorder_context function places the highest-relevance chunk at the start and the second-highest at the end. Critique that strategy: describe a real RAG scenario where this “put the best at the edges” ordering could still fail to retrieve the needed fact.

Create Design a “map-reduce” summarization strategy for a 100-page legal contract where the critical clause sits on page 40. Describe how you’d chunk, summarize, and recombine the document so the LLM reliably surfaces that clause instead of burying it.

---

Related articles

• Building Your First RAG Pipeline: Chunking, Embedding, and Retrieval)
• RAG vs Long-Context Models: Which Wins in 2026? A Practitioner’s Guide

References & Further reading

• Liu, N.F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2023). Lost in the Middle: How Language Models Use Long Contexts. arXiv:2307.03172
• tiktoken — OpenAI’s Tokenizer (GitHub)

---