Why LLMs Confidently Make Things Up: Understanding and Catching Hallucination

• prompt — a question about the “Treaty of New York in 1752 between France and the Aztec Empire,” an event that never happened (the Aztec Empire fell to the Spanish in the 1500s, and there was no such treaty in 1752).
• print(f"Prompt: {prompt}\n") — displays the fabricated question.
• if os.getenv("OPENAI_API_KEY") — checks whether a real key is configured. If so, the code makes an actual call via client.chat.completions.create(...) and uses whatever the model returns.
• response (no-key branch) — a representative example of the kind of confident, fabricated answer a model produced when we asked it. It’s illustrative, not a guaranteed transcript: a live call to the same prompt won’t reproduce this exact wording, since LLM sampling is stochastic.
• print(f"AI Response: {response}") — shows the model’s fabricated answer.
• Key insight: the model doesn’t fact-check the premise of the question. It sees “Treaty,” “New York,” and “France” and generates text that sounds like a history book—prioritizing the vibe of a factual answer over actual facts. This is exactly what happened with Rae’s burrito refund incident: the customer’s question contained assumptions about a refund policy, and the bot played along instead of pushing back.

• candidates — a dictionary mapping possible next words to their probabilities, standing in for the model’s learned distribution over the vocabulary.
• "the", "a", "yesterday", "Mars" — four candidate continuations, with "the" at 45% and "Mars" at 20%.
• random.choices(list(candidates.keys()), weights=candidates.values()) — performs weighted random sampling: it picks one word from the candidate list, with each word’s probability proportional to its weight.
• [0] — extracts the single chosen word from the list that random.choices returns.
• Return value — the chosen word, simulating one step of next-token prediction.
• Key insight: if the model picks "Mars", it isn’t because the explorer actually went there—it’s because "Mars" had a high enough probability in the model’s training data to be a valid grammatical continuation. This is the core mechanism behind both creative writing and hallucination: the model is rolling weighted dice, not consulting a fact database.

The core mechanism of an LLM is next-token prediction: given the tokens seen so far (the context), the model computes a probability distribution over the entire vocabulary, then samples (or greedily picks) the next token.

$P(w_i \mid w_1, w_2, \ldots, w_n) = \frac{e^{z_i}}{\sum_{j=1}^{V} e^{z_j}}$

where $z_i$ is the raw logit (unnormalized score) the model assigns to token $w_i$, and $V$ is the vocabulary size. This is the softmax function — it converts raw scores into a probability distribution that sums to 1.

The log-prob of a chosen token is the natural log of that probability:

$\text{logprob}(w_i) = \log P(w_i \mid w_1, \ldots, w_n) = z_i - \log\!\left(\sum_{j=1}^{V} e^{z_j}\right)$

A log-prob near 0 (e.g., -0.01) means the model assigned near-100% probability to that token — very confident. A log-prob of -4.50 means the probability was roughly $e^{-4.5} \approx 0.011$, or about 1.1% — the model was essentially guessing.

• response_data — a list of dictionaries, each holding a token and its logprob (the natural log of the probability the model assigned to that token when it was generated).
• "The", "capital", "of", "France", "is" — the first five tokens, with log-probs near 0 (e.g., -0.01, -0.02, -0.05), meaning the model was nearly 100% confident in each. These are common words with high probability in almost any context.
• "Limoges" — the final token, with a log-prob of -4.50, which corresponds to a probability of roughly $e^{-4.50} \approx 0.011$ (about 1.1%). The model was essentially guessing.
• for item in response_data — iterates through each token in response_data.
• if item['logprob'] < -1.0 — flags any token whose log-prob falls below -1.0 (roughly 37% confidence — $e^{-1.0} \approx 0.37$).
• Warning print — when the condition triggers, it prints a warning identifying the uncertain token.
• Key insight: this is the basis of a “BS detector” — if a factual claim is built from low-probability tokens, the model is likely fabricating. The catch: most production APIs (like OpenAI’s) don’t expose per-token log-probs by default, so this technique requires an API that supports it or running an open-source model locally.

• knowledge_base — a dictionary holding the ground truth about the Treaty of New York: it was a 1790 agreement between the US and the Creek people (a real historical fact).
• rag_answer(query) — takes a query string and performs two steps.
• context = knowledge_base.get("Treaty of New York", "No data found.") — looks up the relevant context, simulating the retrieval step of RAG (in production, this would be a vector database search returning the most semantically similar chunk). If no data is found, it falls back to the string "No data found."
• prompt = f"Using only this info: {context}, answer this: {query}" — embeds the retrieved context directly into the prompt, constraining the model to answer only from the provided context.
• Return value — the correct answer, grounded in the retrieved fact.
• Key insight: by giving the model the truth upfront, we move the task from “remembering” (where it fails) to “summarizing” (where it excels). For Rae’s support bot, this is exactly what she needs: instead of letting the bot invent a refund policy, she retrieves the actual policy text and instructs the model to use only that.

Hallucination Mitigation Strategies: When Each Helps and When It Falls Short

Rae tried several approaches after the burrito refund incident. Here’s what works, what partially works, and what doesn’t fully solve the problem:

RAG (Grounding via retrieval):

• When it helps: When the answer exists in the retrieved context and the question is about a specific fact (e.g., “What’s the return policy for the burrito warmer?”). The model reads the answer and summarizes it — moving from “remembering” to “reading comprehension.”
• When it doesn’t fully solve: When the model ignores the retrieved context and hallucinates anyway (a “context adherence” failure — the model sees the context but generates something else). Also when the retrieval step fetches the wrong chunk — garbage in, garbage out. RAG reduces hallucination but doesn’t eliminate it.

Lower temperature:

• When it helps: Temperature controls how “creative” the token sampling is. Setting temperature to 0 (greedy decoding) makes the model always pick the single highest-probability token, reducing random creative leaps. Good for factual QA where you want the most probable answer, not the most inventive one.
• When it doesn’t fully solve: Low temperature reduces random hallucinations but not systematic ones. If the model’s probability map is wrong (e.g., it learned a common misconception from training data), temperature 0 will confidently output the wrong answer every time — consistently and confidently wrong.

Confidence / log-prob thresholds:

• When it helps: When you can access the model’s per-token log-probs. Tokens with very low log-probs signal the model is guessing — you can flag or reject responses where key factual tokens fall below a threshold (e.g., log-prob < -1.0).
• When it doesn’t fully solve: Log-probs measure the model’s confidence, not its correctness. A model can be very confident and very wrong — high probability does not equal true. Also, most production APIs (OpenAI, Anthropic) don’t expose per-token log-probs by default, making this impractical for many deployments without running an open-source model locally.

Refusal prompting (system prompts that say “If you don’t know, say you don’t know”):

• When it helps: For questions clearly outside the model’s knowledge or the retrieved context. A strong system prompt like “Answer only using the provided context. If the answer is not in the context, say ‘I don’t know’” can reduce fabrication by giving the model permission to abstain.
• When it doesn’t fully solve: Models still hallucinate within the guardrails — they may find a plausible-sounding answer in the context that isn’t actually there, or subtly misread it. Refusal prompting reduces the surface area but doesn’t close the gap entirely.

Rae’s takeaway: No single technique eliminates hallucination. She needs RAG for grounding, a low temperature for consistency, refusal prompts for guardrails — and critically, a way to measure whether these fixes are actually working. Which is exactly what she’ll build next.

Summary Checklist

• LLMs are not databases: They predict the next likely word, not the next true fact.
• Fluency is not accuracy: A well-crafted sentence can still be wrong.
• Check the log-probs: If you can access the model’s confidence scores, use them to flag shaky answers.
• Use RAG: Don’t let the AI guess. Give it the data it needs.

Rae patches the burrito refund incident with a RAG fix, a lower temperature, and a stronger system prompt. A week later, another fabricated answer slips through. Then another. She can’t eyeball every response the bot generates to see if it “looks right.” She needs a systematic way to measure whether her fixes are working—and to catch hallucinations before her customers do. Next article, she’ll learn how to evaluate LLM output beyond just checking whether it “looks right.”

Check Your Understanding

Questions below move from simple recall up to open-ended design, roughly following Bloom’s Taxonomy.

Remember What is next-token prediction, and why does the article say it means an LLM has a “probability map” rather than a “truth database”?

Understand Explain in your own words why the article calls an LLM a “world-class mimic, not a librarian.” What is the difference between the two when a question asks about the Treaty of New York?

Apply Using the logprob thresholds from Section 4, a model outputs the sentence “The capital of France is Limoges” with the token Limoges scored at -4.50. Write out the loop condition from the code that would flag Limoges, and state what the warning would tell you about the model’s confidence.

Analyze The article says the model does “the exact same math” when writing a poem about a blue cat and when writing a legal brief about a law that doesn’t exist. Walk through why one output is labeled “creative” and the other “hallucination,” given that the underlying mechanism is identical.

Evaluate Section 5 proposes Self-Consistency as a hallucination detector: ask the same question multiple times, and treat disagreement as a sign of fabrication. Critique that approach—describe a realistic scenario where three confident, consistent answers could all still be wrong, and explain why agreement alone isn’t proof of truth.

Create Design one additional detection or prevention technique beyond log-probs, self-consistency, and RAG that a production system could use to catch confident hallucinations. Describe how it would work, what failure mode it targets, and one limitation it would still have.

Related articles

• RAG vs. Long-Context Models: Which Wins in 2026?
• Evaluating LLM Output Beyond “It Looks Right”

References & Further reading

• Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, S.J., Madotto, A., & Fung, P. (2023). Survey of Hallucination in Natural Language Generation. arXiv:2202.03629
• Bender, E.M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? FAccT ‘21. doi:10.1145/3442188.3445922
• OpenAI. (2024). Logprobs API documentation. platform.openai.com/docs/api-reference/chat