Building Your First RAG Pipeline: Chunking, Embedding, and Retrieval

Which chunking strategy should Rae use for her product manual?

Naive / Fixed-Length Chunking — The simplest approach: split every N characters or tokens. Fast, predictable, no dependencies beyond Python’s built-in slicing. Use this for prototyping or when your documents are uniformly structured (e.g., a list of one-line FAQs). The trade-off: it slices right through words wherever the count happens to land — in the example above, “descaled” comes out as “d” and “escaled” at a 30-character chunk size. When a customer searches for “descaled,” the embedding can’t match the fragment, and the retrieval silently fails.

Overlap Chunking — Fixed-length splits, but the last M characters of Chunk A repeat at the start of Chunk B. This ensures that if a key fact straddles a boundary, it appears fully in at least one chunk. The trade-off: your index grows by roughly M/N percent (you’re storing the same text twice), and you can retrieve the same passage in two consecutive chunks. Worth the added complexity when your documents have medium-length sentences that frequently cross chunk boundaries — which is exactly Rae’s situation with her product manual’s paragraphs.

Semantic Chunking — Split at natural boundaries: sentence ends, double newlines, or heading markers. This preserves the integrity of each passage so every chunk is a complete thought. The trade-off: chunk sizes vary wildly (3 tokens vs. 300 tokens), which complicates embedding quality and context budgeting. Worth it when your documents have clear structural markers (Markdown headings, numbered steps, section breaks) — like Rae’s product manual, which has numbered sections and sub-headers.

A diagnostic, not a verdict: There’s no single best chunking strategy — it depends on your corpus. Before picking one, check: does your content have consistent structural markers (headings, numbered steps)? Semantic chunking is worth the setup. Is it mostly uniform, medium-length prose where sentences routinely straddle arbitrary boundaries, and you need something you can reason about without per-document logic? Overlap chunking at 256–512 tokens with 50–100 tokens of overlap is a reasonable thing to try first. Are you prototyping fast on short, uniform snippets where a split mid-phrase barely matters? Naive fixed-length is fine to start. For Rae’s numbered manual with sub-headers, the structural-markers condition points toward semantic chunking — but she should verify against her own retrieval failures rather than assume the diagnosis is right before she’s tested it.

When to avoid semantic chunking: If your documents have no clear structural markers (flat text, run-on paragraphs), semantic chunking degrades to sentence-splitting, which can produce hundreds of tiny, context-free fragments. In that case, overlap chunking with a larger window is safer.

4. Embeddings: Converting Words to Numbers the LLM Can Search

Computers don’t understand words — they understand numbers. An embedding is a vector, a list of floating-point numbers that drops a piece of text into a multi-dimensional map of meanings.

Two sentences with the same meaning — say, “How’s the weather?” and “Is it raining outside?” — land close together on that map. We measure that closeness with Cosine Similarity.

from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity

model = SentenceTransformer('all-MiniLM-L6-v2')

chunks = [
    "Use citric acid to descale the machine.",
    "The portafilter should be cleaned daily.",
    "Mix the descaling solution with warm water."
]

# Convert text to vectors
embeddings = model.encode(chunks)

# Compare the first chunk to the others
sim_1_2 = cosine_similarity([embeddings[0]], [embeddings[1]])[0][0]
sim_1_3 = cosine_similarity([embeddings[0]], [embeddings[2]])[0][0]

print(f"Similarity (Descale vs Clean): {sim_1_2:.4f}")
print(f"Similarity (Descale vs Solution): {sim_1_3:.4f}")
# The score for 'Descale vs Solution' (0.68) is meaningfully higher than
# 'Descale vs Clean' (0.37) — both are machine-maintenance sentences, so
# neither score is near zero, but the model still ranks the more directly
# related pair higher. The math 'felt' the meaning, even if the gap is
# smaller than you'd guess from the words alone.

5. Retrieval: Finding the Right Chunks

When a user asks a question, we don’t search for keywords like a 1990s search engine. We embed the question and look for the “Top-K” nearest neighbors.

If $K=3$, we grab the three chunks with the highest similarity scores. The embedding model handles this well — even if the user uses different words (e.g., “reset” vs. “restart”), it knows they are related.

6. Putting It Together: Your First Working RAG Pipeline

Let’s build a mini-system with FAISS, a library that searches through thousands of vectors fast.

⚠️ Requires: pip install faiss-cpu

import faiss
import numpy as np
from sentence_transformers import SentenceTransformer

# 1. Setup
model = SentenceTransformer('all-MiniLM-L6-v2')
documents = [
    "Step 1: Fill the water tank to the max line.",
    "Step 2: Add the descaling powder to the tank.",
    "Step 3: Run the brew cycle without coffee.",
    "Step 4: Rinse the tank and run two more cycles."
]

# 2. Embed and Index
embeddings = model.encode(documents)
dimension = embeddings.shape[1]
index = faiss.IndexFlatL2(dimension)
index.add(embeddings.astype('float32'))

# 3. Query
query = "How do I use the powder?"
query_vec = model.encode([query])

# Find top 2 chunks
D, I = index.search(query_vec.astype('float32'), k=2)

print("Retrieved Chunks:")
for idx in I[0]:
    print(f"- {documents[idx]}")

# The system correctly retrieved Step 2 and Step 1. 
# Now you would send these to an LLM to generate a final answer.

A quick note on what “nearest” means here: IndexFlatL2 ranks by Euclidean distance, not cosine similarity — the two only produce the same ranking when the vectors are unit length. That’s not a gap you need to close by hand for this pipeline, though: all-MiniLM-L6-v2’s published pipeline ends with a normalization layer, so model.encode() already returns unit-length vectors (you can check with np.linalg.norm(embeddings, axis=1) — it comes back as 1.0 for every row). If you swap in an encoder that doesn’t normalize internally, you’d need to call faiss.normalize_L2() yourself before indexing and querying, or the L2 ranking and the cosine ranking can disagree.

7. Debugging Your Pipeline: Why Retrieval Fails

Sometimes RAG fails. You ask a question, the answer comes back “I don’t know,” and the info is sitting right there in the doc.

Common failure modes:

• The ‘Lost in the Middle’ problem: Retrieve 20 chunks and the LLM often ignores the ones in the middle of the list.
• Bad Chunking: Your chunk size was 30 characters, so a word like “descaled” got cut in half.
• Weak Embeddings: Your model doesn’t understand industry-specific jargon (medical terms, legal terms, etc.).

The Fix: Print your retrieved chunks during development. If those chunks are irrelevant, no amount of LLM “smartness” will save the final answer.

8. What’s Next: Reranking and Hybrid Search

Once your basic pipeline runs, you can add a few upgrades:

1. Reranking: Retrieve 10 chunks, then use a slower, smarter model to pick the best 3.
2. Metadata: Only search through documents from “Year 2024.”
3. Hybrid Search: Combine keyword search (BM25) with vector search to get coverage that neither method alone provides.

Recap:

• LLMs have limited memory; RAG provides a “library” they can look at.
• Chunking breaks text down; embeddings turn it into math.
• Retrieval finds the most relevant math vectors.
• Always check your retrieved chunks before blaming the LLM!

Rae ships her first RAG pipeline, and her support bot starts answering real customer questions from the product manual. But she soon notices something odd: sometimes the bot misses a fact she can see right there in the retrieved chunks—it’s just buried in the middle of a long passage. The retrieval worked. The chunk was relevant. But the LLM didn’t use it. In the next article, we’ll explore why LLMs forget the middle of their context—and what Rae can do about it.

Check Your Understanding

Questions below move from simple recall up to open-ended design, roughly following Bloom’s Taxonomy.

Remember What are the three main steps of a RAG pipeline described in the article?

Understand In your own words, explain what Cosine Similarity measures, and why “How’s the weather?” and “Is it raining outside?” would end up with vectors close together even though they share almost no words.

Apply Using the article’s naive chunking example (splitting the descaling sentence into 30-character chunks), identify where the fixed 30-character split cuts the word “descaled” in half, and explain why Overlap chunking would fix that specific case.

Analyze The article’s FAISS example retrieves Step 2 and Step 1 for the query “How do I use the powder?” Walk through why an embedding-based search can match “powder” to “descaling powder” in Step 2 even though the query never uses the word “descaling.”

Evaluate The article’s “Lost in the Middle” failure mode says LLMs often ignore chunks in the middle of a long retrieved list. Critique the common fix of “just retrieve more chunks to be safe” (e.g., top-20 instead of top-3): what does this evaluation harness for RAG actually need to catch, if simply retrieving more doesn’t guarantee the LLM uses it?

Create Design a chunking strategy for a new document type the article doesn’t cover: a legal contract with numbered clauses and cross-references (e.g., “as defined in Section 4.2”). Would you use fixed-length, overlap, or semantic chunking, and what specific failure would you expect if you chunked this document the wrong way?

Related articles

• Vector Databases Compared: When You Actually Need One)
• Why LLMs Forget the Middle: Understanding Context Windows)

References & Further reading

• Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, V., Küttler, D., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., & Kiela, D. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv:2005.11401
• LangChain — Text Splitters Documentation
• LlamaIndex — Chunking & Node Parsing