Evaluating LLM Output Beyond "It Looks Right"

Last time, Rae’s support bot hallucinated a fabricated refund policy for a customer named Marco—the infamous “burrito refund” incident. She patched it with a RAG fix, a lower temperature, and a stronger system prompt. A week later, another fabricated answer slipped through. Then another. She couldn’t eyeball every response the bot generated to check if it “looked right.” She needed a systematic way to measure whether her fixes were working—and to catch hallucinations before her customers did.

1. The Problem: Why ‘Looks Good’ Isn’t Enough

Rae did what most builders do early on: typed a few test questions into her support bot, read the answers, nodded when they looked right. Polite, well-structured, professional-sounding. She showed the bot to her co-founder, who was impressed. She hit ‘deploy.’

Two days later, a frantic message from her customer success lead. The bot had told a customer their warranty covers “damage caused by solar flares” (it doesn’t) and promised a full replacement for a discontinued model that was never sold (it shouldn’t).

What happened? Rae fell into the eyeballing trap. LLMs are masters of mimicry. They sound confident while being completely wrong. This is the hardest part of working with generative AI — a model can fail silently. It doesn’t crash with a Red Screen of Death. It just lies to your users in a convincing tone.

‘Looks good’ doesn’t scale. You can check 10 outputs by hand. You can’t check 10,000. If your evaluation strategy is just reading a few responses, you’re flying blind.

2. Three Dimensions of LLM Quality: Correctness, Relevance, and Safety

Before we start measuring, we need to know what we’re measuring. Not all good answers are good for the same reason. Quality usually falls into three buckets:

• Correctness (Factuality): Does the answer match the ground truth? If someone asks for the subscription price, is the number right? This is our most objective metric.
• Relevance: Does the answer address the user’s actual intent? A model might deliver a perfectly factual lecture on coffee history when the user just asked for the nearest cafe. That fails relevance.
• Safety: Does the model avoid harmful content, bias, or private data leaks?

So, correctness comes down to truth, relevance to utility, and safety to risk. Your app dictates which one matters most. A creative writing tool needs high relevance but can tolerate low correctness; a medical bot demands near-perfect correctness and safety. For Rae’s support bot, all three carry weight. A wrong warranty claim (a correctness failure) is just as bad as ignoring what the customer asked (a relevance failure) or leaking another customer’s order details (a safety failure).

3. Automated Metrics: Speed vs. Nuance

We want evaluation to be fast and cheap. Automated metrics like BLEU, ROUGE, and Cosine Similarity handle that.

BLEU and ROUGE were built for translation and summarization. They count how many words in the LLM’s output match a ‘reference’ (correct) answer. Cosine Similarity takes a different approach. It uses embeddings (math vectors) to check if the meaning of two sentences is close, even when the words differ.

So what happens when we run these on real examples?

import numpy as np
from sklearn.metrics.pairwise import cosine_similarity
from nltk.translate.bleu_score import sentence_bleu

# Our 'Gold Standard' reference answer
reference = "The capital of France is Paris."

# Three different LLM attempts
outputs = {
    "Correct": "Paris is the capital city of France.",
    "Hallucination": "The capital of France is Lyon.",
    "Off-Topic": "France is a country in Europe known for its food."
}

print(f"Reference: {reference}\n")

Now score each candidate answer against the reference:

for label, text in outputs.items():
    # Simple word overlap (BLEU) - expects list of tokens
    ref_tokens = reference.lower().split()
    text_tokens = text.lower().split()
    score_bleu = sentence_bleu([ref_tokens], text_tokens)
    
    print(f"[{label}] Output: {text}")
    print(f" -> BLEU Score: {score_bleu:.4f}")
    print("-" * 30)

This block demonstrates how automated metrics like BLEU can mislead you about factual correctness.

• reference — the gold-standard answer: “The capital of France is Paris.”
• outputs — a dictionary with three candidate answers: a “Correct” one that restates the fact, a “Hallucination” that swaps “Lyon” for “Paris” (a factual error), and an “Off-Topic” response about French cuisine.
• reference.lower().split() — tokenizes the reference by lowercasing and splitting on whitespace, producing a list of tokens like ["the", "capital", "of", "france", "is", "paris."].
• text.lower().split() — does the same for each candidate.
• sentence_bleu([ref_tokens], text_tokens) — computes the BLEU score: the first argument is a list of reference token lists (wrapped in brackets because BLEU supports multiple references), and the second is the candidate’s tokens. BLEU measures n-gram overlap — more shared words means a higher score.

The key insight: the “Hallucination” answer shares many surface words with the reference (“The capital of France is…”) and may score nearly as well as the “Correct” answer, even though “Lyon” is factually wrong. Word overlap doesn’t equal truth — which is exactly the trap Rae’s eyeballing fell into.

What this actually means: The ‘Hallucination’ answer often scores higher on BLEU than the ‘Correct’ one if it shares more words (like ‘The capital of France is…’). That’s the catch. Automated metrics get fooled by word overlap. They’re great for catching massive regressions at scale, but they miss the nuance of truth.

4. Human Evaluation: The Gold Standard (and Its Costs)

Want to know if a joke lands or a technical explanation makes sense? You need a human. We’re the gold standard because we understand context.

But humans are inconsistent. Two people rating a response on a 1-5 scale might give it a 3 and a 5. So we bring in a Rubric—strict scoring rules—and measure Inter-rater Agreement with Cohen’s Kappa.

from sklearn.metrics import cohen_kappa_score

# Two humans rate 5 chatbot responses for 'Tone' (1=Bad, 2=Neutral, 3=Good)
rater_a = [3, 2, 3, 1, 2]
rater_b = [3, 1, 3, 1, 3]

kappa = cohen_kappa_score(rater_a, rater_b)
print(f"Inter-rater Agreement (Kappa): {kappa:.2f}")

# Interpretation:
# 0.0 - 0.20: Slight agreement
# 0.41 - 0.60: Moderate agreement
# 0.81 - 1.00: Almost perfect agreement

This block measures how consistently two human raters agree on their evaluations.

• rater_a / rater_b — lists of ratings (1=Bad, 2=Neutral, 3=Good) for 5 chatbot responses.
• cohen_kappa_score(rater_a, rater_b) — computes Cohen’s Kappa, which measures agreement between two raters while correcting for agreement that would happen by pure chance. A Kappa of 1.0 means perfect agreement, 0.0 means agreement no better than random, and negative values mean worse than random.
• The interpretation comments — show standard thresholds: below 0.20 is “slight” agreement, 0.41–0.60 is “moderate,” and 0.81+ is “almost perfect.”

If the Kappa comes back low (below 0.4), the rubric is too vague — the raters aren’t disagreeing because they’re wrong, but because the scoring criteria aren’t precise enough. For Rae, this means if she asks her co-founder and customer success lead to grade bot responses and they keep disagreeing, she needs a tighter rubric, not better raters.

If your Kappa sits below 0.4, your instructions are too vague. The raters aren’t wrong—they just lack a clear picture of what “good” looks like.

5. Hybrid Evaluation: Combining Automated and Human Signals

You don’t have the budget to check everything by hand. The fix is Uncertainty Sampling. Score everything with automated metrics, then route only the ‘middle’ or ‘low’ scores to humans.

A 0.99 similarity score is probably fine. A 0.1 is definitely broken. But 0.6 — that’s where humans should step in. You save money without sacrificing quality.

Evaluation Methods: Which Should Rae Reach For?

Rae mapped out three approaches to evaluating her support bot’s outputs. Each has a distinct speed-cost-nuance tradeoff:

Automated metrics (BLEU, ROUGE, cosine similarity):

• Speed: Fast — you can score 10,000 outputs in seconds.
• Cost: Cheap — pure computation, no API calls or human time.
• Nuance: Low. These metrics measure word overlap or embedding distance, not truth. A hallucination that shares surface words with the reference (“The capital of France is Lyon”) can score as well as or better than a correct answer phrased differently. Good for catching massive regressions at scale, but blind to factual errors that “sound right.”
• When to use: As a first-pass filter across your full test set. Flag anything with a low score for closer review.

Human evaluation (with rubrics + Cohen’s Kappa):

• Speed: Slow — a human reads and scores each response individually.
• Cost: Expensive — human time is your most scarce resource.
• Nuance: Highest. Humans understand context, tone, and factual accuracy in ways no metric can replicate. The gold standard for “is this actually a good answer?”
• Tradeoff: Inconsistent without a strict rubric. Two raters can disagree on a 1-5 scale just because “good” is vague. A precise rubric + Cohen’s Kappa above 0.6 keeps ratings reliable.
• When to use: On a sampled subset (10-20% of outputs), especially for edge cases and borderline scores where automated metrics are unreliable.

LLM-as-a-Judge (use a powerful model to grade a weaker one):

• Speed: Fast — one API call per evaluation, parallelizable across thousands of outputs.
• Cost: Moderate — you pay for the judge model’s tokens, but it’s far cheaper than human time at scale.
• Nuance: Medium-high. A strong model can assess factual accuracy, relevance, and tone better than BLEU or cosine similarity. But it has its own biases: it may prefer verbose answers, agree with the model it’s grading (sycophancy), or hallucinate its own judgments.
• When to use: When you need more nuance than automated metrics but can’t afford human review on every output. The next article covers how to build this into a repeatable harness.

Rae’s strategy: Use automated metrics as a fast first pass across the entire test set, send the uncertain middle (scores around 0.4–0.7) to human review, and experiment with LLM-as-a-Judge for the rest.

6. Task-Specific Evaluation: Retrieval, Summarization, and Classification

One size doesn’t fit all.

• Retrieval (RAG): If your bot searches documents, use Precision@k — were those top results actually useful?
• Summarization: ROUGE gives you coverage. But you’ll also want a faithfulness check, so the model isn’t inventing new facts.
• Classification: Stick with standard F1-Scores.

def calculate_mrr(relevant_indices):
    """Mean Reciprocal Rank: How high up was the first right answer?"""
    for i, is_relevant in enumerate(relevant_indices):
        if is_relevant:
            return 1 / (i + 1)
    return 0

# Example: The correct doc was at position 3 in the search results
results = [False, False, True, False]
mrr = calculate_mrr(results)
print(f"MRR for this query: {mrr:.3f}") # 0.333 means the first right answer was 3rd.

This block implements Mean Reciprocal Rank (MRR), a metric for evaluating retrieval quality — exactly what Rae needs for her RAG pipeline.

• calculate_mrr(relevant_indices) — takes a list of booleans where each element indicates whether the search result at that position is relevant.
• for i, is_relevant in enumerate(relevant_indices) — iterates through the results with their zero-based indices.
• return 1 / (i + 1) — when it finds the first True (relevant) result, returns the reciprocal of its 1-based position — so position 1 gives 1.0, position 2 gives 0.5, position 3 gives 0.333. If no result is relevant, it returns 0.
• results = [False, False, True, False] — the example means the correct document was at position 3 (index 2), so MRR = 1/3 ≈ 0.333.

For Rae’s support bot, this tells her how high up in the search results the right product manual section appears — if the correct chunk is buried at position 10, her bot is unlikely to surface the right answer.

7. Catching Silent Failures: Drift, Bias, and Edge Cases

Models drift. Maybe you updated a prompt, or the underlying API shifted on you. Your model starts giving shorter answers, or skews more biased.

Watch your outputs over time. If average response length drops by 50%, or ‘Safety’ scores dip for users in a specific region, you’ve got a problem.

8. Putting It Together: A Practical Evaluation Workflow

Here’s what a real-world pipeline looks like:

1. Define a Test Set: Gather 100 questions where you already know the answers.
2. Run Automated Scores: Score every new model version against those 100 questions.
3. Spot Check: Have a human look at 10% of the outputs.
4. Compare: If the new model averages lower than the old one, don’t deploy.

# A mock-up of a production evaluation report
results = [
    {"id": 1, "sim_score": 0.92, "human_verified": True, "status": "Pass"},
    {"id": 2, "sim_score": 0.45, "human_verified": False, "status": "Flag for Review"},
    {"id": 3, "sim_score": 0.88, "human_verified": True, "status": "Pass"}
]

pass_count = sum(1 for r in results if r['status'] == "Pass")
print(f"Evaluation Complete: {pass_count}/{len(results)} passed.")

This block simulates a production evaluation report — the kind of output Rae would generate after running her support bot against a test set.

• results — a list of dictionaries, each representing one evaluated response with an id, a sim_score (cosine similarity to a reference answer), a human_verified boolean, and a status field (“Pass” or “Flag for Review”).
• pass_count = sum(1 for r in results if r['status'] == "Pass") — uses a generator expression: it iterates through all results, yielding 1 for each one whose status is “Pass” and 0 otherwise, then sums them up — a concise Python idiom for counting items matching a condition.
• The final print — displays 2/3 passed, summarizing the evaluation run.

Response 2, with a sim_score of 0.45, was flagged for human review — it’s in the uncertain middle zone where automated metrics can’t be trusted alone.

9. Common Pitfalls and How to Avoid Them

• Cherry-picking: Testing only the easy questions — the ones you already know the model gets right.
• Metric Gaming: Tweaking your prompt to push up a BLEU score, even when the actual answer gets worse for the user.
• Over-trusting Automation: Treating a high similarity score as proof the answer is factually correct.

10. What’s Next: Building Your Evaluation System

Don’t try to build a perfect system on day one.

1. Start by saving every input and output to a CSV or database.
2. Run a simple similarity check against a few ‘gold’ answers.
3. Once a week, read 20 random responses.

As you grow, you can add more complex tools. In the next part of this series, we’ll look at LLM-as-a-Judge—where a larger model like GPT-4 grades a smaller one.

Until then, stop eyeballing. Start measuring.

Rae now has a framework for what to measure—correctness, relevance, and safety—and a sense of which tools to reach for at each stage. But a framework isn’t a system. She needs to build something repeatable: a harness that runs these checks automatically every time she tweaks a prompt, swaps a model, or updates her retrieval pipeline. In the next article, she’ll build exactly that.

Check Your Understanding

Questions below move from simple recall up to open-ended design, roughly following Bloom’s Taxonomy.

Remember What are the three dimensions of LLM quality the article describes, and what does each one measure?

Understand In your own words, explain why the “Hallucination” answer in the BLEU example can score higher than the “Correct” answer, and what this reveals about a fundamental limitation of word-overlap metrics.

Apply Using the article’s calculate_mrr function, what would the MRR be if the correct document appeared at position 5 (index 4) instead of position 3?

Analyze The article’s Uncertainty Sampling approach routes only “middle” similarity scores (like 0.6) to human reviewers, trusting very high (0.99) and very low (0.1) scores automatically. Walk through a scenario where this strategy would let a genuinely bad answer slip through undetected — what kind of failure would produce a high similarity score despite being wrong?

Evaluate The article warns about “Metric Gaming”—changing a prompt just to raise a BLEU score without improving the actual answer. Critique the article’s own recommended workflow (Section 8), which gates deployment purely on “if the new model has a lower average score than the old one.” What about this gate itself creates pressure toward metric gaming, and what would you add to close that gap?

Create Design an evaluation rubric (following the article’s Cohen’s Kappa approach) for grading chatbot responses on “Empathy” in a customer support context. Define 3 rating levels with concrete criteria for each, specific enough that two different human raters would likely agree — addressing the article’s point that vague rubrics produce low Kappa scores.

---

Related articles

• Why LLMs Confidently Make Things Up: Understanding and Catching Hallucination)
• Building a Simple LLM Evaluation Harness in Python

References & Further reading

• Papineni, K., Roukos, S., Ward, T., & Zhu, W. (2002). BLEU: a Method for Automatic Evaluation of Machine Translation. ACL 2002. doi:10.3115/1073083.1073135
• Esuli, A., Marcheggiani, D., & Sebastiani, F. (2023). RAGAS: Automated Evaluation of Retrieval Augmented Generation. github.com/explodinggradients/ragas
• DeepEval. The Open-Source LLM Evaluation Framework. docs.confident-ai.com

---