Building a Simple LLM Evaluation Harness in Python

Why You Can’t Just Ask If an LLM Is Good

Last time, Rae mapped out the three dimensions of LLM quality—correctness, relevance, and safety—and saw how automated metrics like BLEU can be fooled by word overlap while human evaluators need tight rubrics to stay consistent. She had a framework for what to measure and a rough workflow: test set, then automated scores, then spot check, then compare. But a framework isn’t a system. She couldn’t run it automatically, couldn’t run it on every prompt change, and couldn’t compare two models side by side. She needed to build something repeatable.

So we’re back to the same support-bot scenario that’s been haunting her since the “burrito refund” incident two articles ago. You just spent three hours crafting the perfect prompt for your new customer support bot. You test it with three questions: “What’s your return policy?”, “How do I track my order?”, and “Are you a robot?” The bot nails all three. You feel like a genius.

Then you launch it. Within an hour, a customer asks, “Can I return a half-eaten burrito?” The bot offers them a full refund and a discount code for a competitor. What happened?

Intuition is a terrible way to judge Large Language Models. These models are probabilistic—they don’t always give the same answer twice. Change the temperature setting or even a single word in your prompt and the output can shift dramatically.

‘Looks good to me’ doesn’t scale. You might spot-check five responses, but not 500. Without a repeatable system, you’re cherry-picking the examples that impressed you and ignoring the ones where the model hallucinated. A ‘harness’ is just a reproducible way to score many outputs against a standard—one that turns ‘I think this model is better’ into ‘This model is 14% more accurate on edge cases.’

The Core Idea: Test Cases, Predictions, and Scores

Think of an evaluation harness as a standardized test for your AI. Every harness has three main parts:

1. The Test Case: The input (the prompt) and the ‘ground truth’ — what you expect to happen.
2. The Prediction: The actual response the LLM generates.
3. The Scorer: The logic that compares the prediction to the ground truth and gives it a grade.

Here’s what that looks like in code. We’ll start with a simple skeleton that calculates accuracy.

class SimpleHarness:
    def __init__(self, model_fn):
        # model_fn is a function that takes a string and returns a string
        self.model_fn = model_fn

    def run(self, test_cases):
        results = []
        for case in test_cases:
            # 1. Get the prediction
            prediction = self.model_fn(case['prompt'])
            
            # 2. Score it (Simple exact match for now)
            score = 1 if prediction.strip() == case['expected'].strip() else 0
            
            results.append({"prompt": case['prompt'], "score": score})
            
        # 3. Aggregate
        avg_score = sum(r['score'] for r in results) / len(results)
        return avg_score, results

# Let's simulate a 'model' that is 50% accurate
def dummy_model(prompt):
    import random
    return "Paris" if random.random() > 0.5 else "London"

test_data = [
    {"prompt": "What is the capital of France?", "expected": "Paris"},
    {"prompt": "Where is the Eiffel Tower?", "expected": "Paris"}
]

harness = SimpleHarness(dummy_model)
final_score, details = harness.run(test_data)
print(f"Average Accuracy: {final_score * 100}%")

This block defines the minimal skeleton of an evaluation harness — the pattern Rae will reuse throughout this article.

• SimpleHarness.__init__ — takes a model_fn, which is any function that accepts a string prompt and returns a string response. This abstraction lets Rae swap in a dummy model, a local model, or a real API call without changing the harness. self.model_fn = model_fn stores it as an instance attribute.
• run method — iterates over test_cases (a list of dicts), calling self.model_fn(case['prompt']) to get a prediction for each case.
• score = 1 if prediction.strip() == case['expected'].strip() else 0 — a ternary expression that does exact-match scoring. .strip() removes leading/trailing whitespace before comparing, so " Paris " and "Paris" are treated as equal.
• results.append({"prompt": case['prompt'], "score": score}) — collects each result.
• avg_score = sum(r['score'] for r in results) / len(results) — uses a generator expression inside sum(), a concise Python idiom for computing the mean of a list of numbers.
• dummy_model — uses random.random() > 0.5 to simulate a 50%-accurate model, returning “Paris” or “London” randomly.

When Rae runs this, she’ll see the average accuracy bounce around 50% on each run — which is exactly why this skeleton uses exact match (simple but brittle) as a starting point before upgrading to smarter scorers.

If the score comes back 0.5, the model got exactly half the questions right. That’s the foundation for everything we’ll build next.

Setting Up Your First Test Cases

Before you write a single line of AI code, you need data. A loose list of strings in your head won’t cut it. You need a structured format like JSON so your harness can read it consistently.

Each test case should have a category. Why? Your model might be great at ‘General Knowledge’ but terrible at ‘Math.’ One big average hides that detail.

import json

# This is how you should structure your data
raw_data = [
    {
        "id": 1,
        "category": "geography",
        "prompt": "What is the capital of Japan?",
        "expected": "Tokyo"
    },
    {
        "id": 2,
        "category": "math",
        "prompt": "What is 2 + 2?",
        "expected": "4"
    }
]

def load_and_validate(data):
    print(f"Loaded {len(data)} test cases.")
    categories = set(d['category'] for d in data)
    print(f"Categories found: {', '.join(categories)}")
    return data

test_cases = load_and_validate(raw_data)

This block shows how to structure test-case data for Rae’s harness.

• raw_data — a list of dictionaries, each with an id (unique identifier), a category (for grouping results by topic), a prompt (the input the model receives), and an expected (the ground-truth answer).
• load_and_validate — takes this data and prints a summary.
• len(data) — gives the count of test cases.
• set(d['category'] for d in data) — uses a generator expression inside a set() constructor to extract all unique category names, so if Rae has 50 test cases spanning “returns”, “shipping”, and “warranty”, she’ll see exactly which categories exist at a glance.
• ', '.join(categories) — joins the set elements into a comma-separated string for display.
• import json — included because in production, Rae would load this data from a .json file rather than hardcoding it, but the structure is identical.

Start small. You don’t need 10,000 cases. Twenty well-written cases covering edge cases — the weird stuff — beat 100 easy questions.

Calling the LLM and Collecting Outputs

Now we need to talk to a real model. Things get messier here. APIs fail. Connections drop. You might hit a ‘rate limit’ — the provider telling you to slow down.

When evaluating, we usually set temperature=0. That makes the model as deterministic as possible. A high temperature means your evaluation results shift every time you run the script, which defeats the purpose of a benchmark.

import time

# Mocking an API call with retry logic
def call_llm_with_retry(prompt, retries=3):
    for i in range(retries):
        try:
            # Imagine this is: openai.ChatCompletion.create(...)
            # We'll simulate a success
            if prompt == "": raise ValueError("Empty prompt!")
            return f"Response to: {prompt}" 
        except Exception as e:
            if i == retries - 1: raise e
            print(f"Error occurred, retrying in {2**i} seconds...")
            time.sleep(2**i)

response = call_llm_with_retry("Hello world")
print(f"Model said: {response}")

This block implements retry logic with exponential backoff — essential when Rae’s harness makes hundreds of API calls and any one of them could transiently fail.

• call_llm_with_retry(prompt, retries=3) — takes a prompt string and a maximum number of retry attempts (defaulting to 3).
• for i in range(retries) — the loop iterates up to 3 times.
• if prompt == "": raise ValueError("Empty prompt!") — simulates a validation error inside the try block. In practice, this would be the actual API call (e.g., openai.ChatCompletion.create(...)).
• return f"Response to: {prompt}" — if the call succeeds, this immediately exits the function.
• except Exception as e — catches the exception if one occurs.
• if i == retries - 1: raise e — checks if we’ve exhausted all retries (0-indexed, so on the 3rd attempt i is 2, which equals 3 - 1). If so, the exception is re-raised so the caller knows it truly failed.
• print(...) / time.sleep(2**i) — otherwise, logs the retry and implements exponential backoff: 2**0 = 1 second on the first retry, 2**1 = 2 seconds on the second. This doubling gives the API time to recover before trying again, rather than hammering it with immediate retries.

What’s actually going on here? We use ‘exponential backoff.’ First try fails, we wait 1 second. Second fails, we wait 2 seconds. It’s the polite way to handle API errors.

Scoring: From Output to a Number

This is the hardest part of evaluation.

Ask “What is the capital of France?” and the model says “The capital is Paris.” A simple Exact Match scorer calls that wrong, because “Paris” != “The capital is Paris.” We need smarter ways to score.

1. Exact Match: Great for code or multiple choice.
2. Fuzzy Match: Checks if the expected word is somewhere in the answer.
3. Similarity: Uses math (embeddings) to see if the meanings are close.

def exact_match(prediction, expected):
    return 1.0 if prediction.strip().lower() == expected.strip().lower() else 0.0

def contains_scorer(prediction, expected):
    return 1.0 if expected.lower() in prediction.lower() else 0.0

pred = "The answer is Paris."
gold = "Paris"

print(f"Exact Match: {exact_match(pred, gold)}") # 0.0
print(f"Contains Match: {contains_scorer(pred, gold)}") # 1.0

This block defines two scorer functions that represent different levels of leniency.

• exact_match(prediction, expected) — returns 1.0 only if the two strings are identical after .strip().lower() (removing whitespace and normalizing case). So "The answer is Paris." vs "Paris" returns 0.0 because the full strings differ, even though the answer is correct.
• contains_scorer(prediction, expected) — more forgiving: expected.lower() in prediction.lower() checks if the expected answer appears as a substring anywhere inside the prediction. So "paris" in "the answer is paris." evaluates to True, returning 1.0.
• pred and gold — demonstrate the difference: Exact Match scores 0.0 (too strict for natural-language responses), while Contains Match scores 1.0 (catches the right answer even when the model adds conversational padding).

For Rae’s support bot, this distinction matters — if the bot says “Yes, our return policy allows returns within 30 days,” an exact-match scorer looking for “30 days” would fail, but a contains scorer would pass.

The Contains Match score of 1.0 tells us the model was wordy, but it still had the right answer inside its response.

Putting It Together: Your First Harness

Let’s build the full machine. We’ll add a progress bar with tqdm so you’re not staring at a blank screen, wondering if it froze.

from tqdm import tqdm

class EvalHarness:
    def __init__(self, model_name, model_fn, scorer_fn):
        self.model_name = model_name
        self.model_fn = model_fn
        self.scorer_fn = scorer_fn

    def run_eval(self, cases):
        results = []
        print(f"Starting eval for {self.model_name}...")
        
        for case in tqdm(cases):
            prediction = self.model_fn(case['prompt'])
            score = self.scorer_fn(prediction, case['expected'])
            
            results.append({
                "id": case['id'],
                "prediction": prediction,
                "score": score
            })
            
        avg = sum(r['score'] for r in results) / len(results)
        return {"model": self.model_name, "score": avg, "details": results}

# Let's run it
def my_model(p): return "Tokyo" # A model that only knows Tokyo

my_cases = [
    {"id": 1, "prompt": "Capital of Japan?", "expected": "Tokyo"},
    {"id": 2, "prompt": "Capital of France?", "expected": "Paris"}
]

harness = EvalHarness("Tokyo-Bot-v1", my_model, contains_scorer)
report = harness.run_eval(my_cases)

print(f"\nFinal Report for {report['model']}:")
print(f"Score: {report['score'] * 100}%")

This block assembles the full evaluation harness — the complete loop Rae will use to test her support bot.

• EvalHarness.__init__ — takes three arguments: model_name (a label for display/comparison), model_fn (the function that calls the model), and scorer_fn (the scoring logic). All three are stored as instance attributes.
• run_eval method — iterates over cases with tqdm(cases). tqdm wraps the iterable in a progress bar so Rae can see how many of her test cases have been processed.
• self.model_fn(case['prompt']) / self.scorer_fn(prediction, case['expected']) — for each case, gets the model’s prediction and scores it.
• results.append({...}) — stores the id, prediction, and score for each case.
• avg = sum(r['score'] for r in results) / len(results) — computes the average score across all cases.
• Return value — a dictionary with the model name, overall score, and per-case details, so Rae can drill into which cases failed, not just the aggregate.
• my_model — def my_model(p): return "Tokyo" is a stub that always returns “Tokyo” regardless of input, simulating a model that only knows one answer.

When run against my_cases (one Japan question, one France question), the harness scores 50%: it gets Japan right but France wrong, and the contains_scorer (defined earlier) is used as the scoring function.

The output shows a score of 50%. That tells us exactly where the model is failing — it doesn’t know France.

Comparing Two Models: The Real Test

Here’s the real test. Does GPT-4 actually outperform GPT-3.5 on your specific task, or is a small local model enough? Run the same harness across both.

# Imagine these are two different model functions
model_a = lambda p: "Paris" if "France" in p else "Wrong"
model_b = lambda p: "Paris" if "France" in p else "Tokyo"

results_a = EvalHarness("Model A", model_a, contains_scorer).run_eval(my_cases)
results_b = EvalHarness("Model B", model_b, contains_scorer).run_eval(my_cases)

print(f"Model A Score: {results_a['score']}")
print(f"Model B Score: {results_b['score']}")

This block demonstrates the killer feature of having a harness: apples-to-apples model comparison.

• model_a / model_b — defined as lambda functions, concise one-line functions. lambda p: "Paris" if "France" in p else "Wrong" checks if the prompt contains “France” and returns “Paris” if so, otherwise returns “Wrong” (for Model A) or “Tokyo” (for Model B).
• Behavior difference — both models handle the France question identically, but they differ on the Japan question: Model A returns “Wrong” (failing contains_scorer), while Model B returns “Tokyo” (passing).
• EvalHarness("Model A", model_a, contains_scorer).run_eval(my_cases) — creates a harness instance and immediately calls run_eval in one expression, a common Python pattern called method chaining.
• results_a / results_b — store the results, each a dict with a 'score' key. When printed, Model A scores 0.5 (got France right, Japan wrong) while Model B scores 1.0 (got both right).

For Rae, this is how she’d compare, say, GPT-3.5 vs. GPT-4 on her actual support-bot test set — same questions, same scorer, different model functions.

If Model B hits 100% and Model A hits 50%, you’ve got data-driven proof of which one to pick. No more guessing.

Scaling and Debugging: What Goes Wrong

Calling an LLM 1,000 times costs real money. Before a big eval, do a dry run on 2 or 3 cases first.

Here’s a quick way to add cost tracking:

def estimate_cost(num_cases, price_per_1k):
    cost = (num_cases / 1000) * price_per_1k
    print(f"Estimated cost for {num_cases} cases: ${cost:.4f}")
    return cost

estimate_cost(500, 0.01) # 1 cent per 1k tokens

This block adds a simple cost estimator — a guardrail before Rae runs a large evaluation batch.

• estimate_cost(num_cases, price_per_1k) — takes the number of test cases and the price per 1,000 tokens.
• cost = (num_cases / 1000) * price_per_1k — scales the per-1k-token price to the actual number of calls: dividing by 1000 converts the per-1k rate to a per-call rate, then multiplying by num_cases gives the total.
• f"${cost:.4f}" — formats the cost to 4 decimal places.
• estimate_cost(500, 0.01) — computes the cost for 500 cases at $0.01 per 1k tokens: (500 / 1000) * 0.01 = 0.5 * 0.01 = 0.005, so $0.0050.
• # 1 cent per 1k tokens — comment clarifying the pricing unit.

In practice, Rae would also factor in the average number of tokens per request (input + output) rather than assuming exactly 1k tokens per call — but this function gives her a quick sanity check before committing to a full run.

If the estimate says $50.00 and your account only has $5.00, you’ll be glad you checked.

Build vs. Buy: Should You Adopt a Framework?

Build Your Own Harness vs. Adopt an Existing Framework

Rae now has a working harness — but she also knows she’s not the first person to need one. Open-source frameworks like RAGAS, DeepEval, and promptfoo all solve this same problem. Should she keep building her own or switch?

Build your own (like this article):

• Pros: Full control over every scoring decision. No dependency to maintain. You understand exactly why a score is what it is because you wrote the logic. Works with any model, any API, any data format — no framework opinions to fight against. Zero install overhead for a small team.
• Cons: You’re reinventing wheels: logging, caching, result serialization, multi-model comparison dashboards, CI/CD integration. If your harness grows, you’ll end up rebuilding features these frameworks already provide. No community-maintained metric implementations (faithfulness, answer relevancy, context precision) that are battle-tested on thousands of real RAG pipelines.
• When to use: When you’re early-stage (like Rae), your evaluation needs are simple (exact match, contains, cosine similarity), and you want to deeply understand the mechanics before abstracting them away. Also when your scoring logic is highly custom and doesn’t fit any framework’s built-in metrics.

RAGAS (Retrieval-Augmented Generation Assessment):

• Pros: Purpose-built for RAG pipelines — exactly Rae’s use case. Provides metrics like faithfulness (did the answer stick to the retrieved context?), answer relevancy, and context precision out of the box. Integrates with LangChain and most vector DBs. Active community, frequent updates.
• Cons: Opinionated about what “good RAG” means. Metrics like faithfulness require an LLM-as-a-judge call (extra API cost per evaluation). Less flexible for non-RAG use cases. Can be overkill if you just need simple accuracy on a 20-case test set.
• When to use: When your primary concern is RAG quality (retrieval + generation) and you want pre-built, tested metrics rather than implementing faithfulness checks from scratch.

DeepEval:

• Pros: Pytest-style API — feels native if your team already uses pytest. Covers a broad range of metrics (faithfulness, hallucination, toxicity, bias). Good documentation and active development.
• Cons: Heavier dependency. Some metrics require LLM-as-a-judge calls. Abstraction layer adds learning curve.
• When to use: When you want a comprehensive testing framework that integrates with your existing Python test suite and covers safety/fairness metrics beyond just accuracy.

promptfoo:

• Pros: CLI-first tool designed specifically for prompt comparison and A/B testing. Outputs matrix-style comparison tables showing how different prompts/models perform across the same test cases. Language-agnostic (works with any API). Excellent for the exact use case of “which prompt is better?”
• Cons: Less focused on RAG-specific metrics. Configuration is YAML/JSON-based rather than Python, which may feel less flexible if you want programmatic scoring logic.
• When to use: When your primary goal is comparing prompts and models side by side (which is exactly what Rae is about to do in the next article).

Rae’s decision: For now, her custom harness is the right tool — she has 20 test cases, two scorers, and full understanding of the logic. Once she needs faithfulness checks on retrieved context (which she will, once her RAG pipeline grows), RAGAS becomes worth the adoption cost. And when she wants to systematically compare dozens of prompt variations, promptfoo’s comparison matrices will save her hours. But today, the harness she just built is enough to start making data-driven decisions.

Next Steps: Beyond the Basics

Once you have this working, you can get fancy:

• LLM-as-a-Judge: Use a very smart model (like GPT-4) to score the output of a smaller model.
• Custom Metrics: If you’re building a coding assistant, try running the code to see if it actually works.
• Continuous Eval: Run this harness every time you change your prompt to make sure you didn’t accidentally break something.

What we covered:

• Intuition is biased; you need a system.
• A harness is just a loop: Prompt -> Model -> Scorer.
• Temperature=0 is your friend for testing.
• Always track your costs and start with a small test set.

So go build your first harness — and stop guessing whether your AI is any good.

With the harness running, Rae can finally measure whether her prompt changes actually help. She’s about to need that ability. Her support bot’s answers are technically correct but clunky, verbose, and sometimes unhelpful. She starts systematically testing prompt variations — tweaking phrasing, structure, and instructions — and watching the scores move with each change. In the next article, we’ll look at the prompt engineering patterns that actually move the needle.

Check Your Understanding

Questions below move from simple recall up to open-ended design, roughly following Bloom’s Taxonomy.

Remember What are the three main parts of an evaluation harness described in the article?

Understand In your own words, explain why the article recommends temperature=0 when running evaluations, and what would happen to your benchmark results if you left temperature high instead.

Apply Using the article’s contains_scorer logic (expected.lower() in prediction.lower()), would the prediction "I believe the answer might be Tokyo, though I'm not fully sure" score 1.0 or 0.0 against the expected answer "Tokyo"? (Verified by running the article’s own function: contains_scorer returns 1.0 — "tokyo" appears as a case-insensitive substring of the prediction, even though the model is hedging with uncertainty language.)

Analyze The article shows that Exact Match scores the prediction "The answer is Paris." as 0.0 against the expected "Paris", while Contains Match scores it 1.0. Walk through a scenario where this same leniency in Contains Match would produce a false positive—a case where the prediction contains the expected text but is still actually wrong.

Evaluate The article’s cost-estimation function only projects cost based on num_cases and price_per_1k, and recommends a “dry run” on 2-3 cases beforehand. Critique this cost check: what does it not account for that could still cause an actual bill to wildly exceed the estimate, even after a successful dry run?

Create Design a scorer function (not necessarily full code, just the logic) for a coding-assistant harness where “the model wrote a Python function that solves the task” needs to be graded automatically. What would you check instead of exact-match or contains-match, and what’s one risk of your approach giving a false “pass”?

---

Related articles

• Evaluating LLM Output Beyond “It Looks Right”
• Prompt Engineering Patterns That Actually Improve Your LLM

References & Further reading

• Zheng, L., Chiang, W. L., Sheng, Y., et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023. arxiv.org/abs/2306.05685
• RAGAS — Automated Evaluation of Retrieval Augmented Generation. github.com/explodinggradients/ragas
• promptfoo — Test & compare LLM prompts, models, and RAG. promptfoo.dev
• DeepEval — The Open-Source LLM Evaluation Framework. docs.confident-ai.com

---