Can a Tiny Model Judge a Giant One? Inside LAGER and INSPECTOR

The Problem with LLM-as-a-Judge

What if the best judge for your AI system isn’t a giant model, but a tiny one?

The standard approach—LLM-as-a-Judge—is straightforward. You take a capable model like GPT-4 or Opus 5, give it a question and a couple of candidate answers, and ask it to pick the better one.

But the problems are well-documented.

Cost and latency. Every evaluation generates tokens one at a time. A single pairwise comparison might use hundreds of tokens just to produce one judgment. Scale that to thousands of evaluations across dozens of model checkpoints, and the costs add up fast.

Systematic biases. Several studies have documented these in detail. A 2025 study tested 15 different LLM judges across 150,000 evaluation instances and found significant position sensitivity—answers preferred based on their slot, not their quality. A 2026 paper introduced “Prior Prejudice,” showing that LLM judges conflate their own training beliefs with response quality. And “The Silent Judge” found that when biases affect the verdict, the model’s chain-of-thought explanation almost never mentions them.

So the system is expensive and biased, and the explanations aren’t trustworthy either. That’s the motivation for alternatives. Today I want to focus on one specific line of research getting attention: using internal model representations for evaluation. These approaches use a smaller model that can barely write a coherent sentence—one you wouldn’t trust to summarize a paragraph—but whose internal representations carry a strong signal about whether an answer is good or bad.

Two main papers to look at here, and they take different approaches.

LAGER: Aggregating Logits Across Layers

The first paper is “Enhancing LLM-as-a-Judge Alignment with Human via Internal Representations,” or LAGER, presented at NeurIPS 2025.

LAGER starts from a simple observation. When an LLM evaluates a response and outputs a score token—like “4” or “5”—that decision passes through the model’s standard language modeling head. The score comes from the final layer’s representation, processed through a linear transformation and softmax. But what if the best signal isn’t in the final layer? They tested this by extracting logits—raw, pre-softmax values—for score-related tokens from every layer. Then they learned a weighted combination: 33 parameters total for a Llama-3.1-8B model. Applied softmax, got a probability distribution over scores, and computed the expected value.

Say the model assigns 45% probability to a score of 4 and 55% to a score of 5. LAGER returns 4.55—a continuous score that captures uncertainty. Aggregating across layers consistently outperformed using only the final layer. On the Flask, HelpSteer, and BIGGen benchmarks, LAGER improved alignment with human judgments by up to 7.5% over standard methods. It matched the performance of reasoning-based judges that use chain-of-thought—without generating any extra reasoning tokens.

When the researchers added chain-of-thought reasoning to LAGER—asking the model to explain its evaluation before scoring—it performed worse. That’s worth noting: it implies reasoning distracted from the direct evaluative signal already present in the hidden states. So LAGER doesn’t just avoid the cost of reasoning tokens; in this setup, reasoning is actively harmful to the quality of judgment.

This suggests the middle-to-upper layers encode task-relevant representations that align more closely with human judgment than the final output layer does. The final layer is optimized for next-token prediction, and it seems to lose some evaluative signal along the way. The hidden states carry something like a “pre-reasoning” signal—more reliable than either reasoning-based prompting or standard final-layer scoring.

One catch: LAGER needs access to intermediate layer logits. So open-weight models only. With API-only models like GPT-4 or Claude, you can’t extract these. You’re stuck with what the authors call “E-Score”—just the final layer’s logits with expected value computation. That still helps, but the gains shrink. The layer weights are model-specific, too. Train them on Llama-3.1-8B and they won’t carry over to Qwen-2.5-14B. Each new model needs its own lightweight calibration. LAGER is a low-cost intervention—33 learned parameters—with meaningful improvements, but it needs open weights and per-model calibration. I’d lean toward it if you’re already running an open-weight judge and want a cheap way to optimize.

INSPECTOR: Can a Tiny Model Judge at All?

The second paper, “Rethinking LLM-as-a-Judge: Representation-as-a-Judge with Small Language Models via Semantic Capacity Asymmetry,” was accepted at ICLR 2026. It asks: do we even need a large model for evaluation? The core hypothesis, Semantic Capacity Asymmetry, claims that evaluation requires significantly less semantic capacity than generation. Recognizing a good answer is easier than producing one, so a small model—one that can’t generate well—might still have enough representational capacity for evaluation.

They tested this with small language models—Qwen3-0.6B (600 million parameters), Qwen3-1.7B, and Llama-3.2-1B—by training lightweight probing classifiers on their hidden states. The process:

1. Feed the small model a question and a candidate answer.
2. Extract hidden state representations from internal layers while it processes the text.
3. Instead of generating text, pass those hidden states through a trained classifier that predicts evaluation scores.

For training data, they used Llama-3-8B to produce responses. A medium model generates more variable quality than a frontier model would, which is actually desirable—you want examples across the full quality spectrum. They then used DeepSeek-V3 to score those responses along five dimensions, and those scores became the labels for training the probe. Worth noting: you’re not eliminating LLM judges entirely. You’re amortizing their cost across many future evaluations. The large judge does the heavy lifting once, during training. After that, the small model with its probe handles evaluation at a fraction of the cost.

The five dimensions are:

• Semantic consistency — does the answer stick to the problem facts
• Logicality — are the reasoning steps valid
• Informativeness — are essential steps included
• Fluency — is it grammatical and clear
• Factuality — are claims correct and supported

Ask a 0.6 billion parameter model to “rate this answer” through standard prompting, and it often performs barely above chance—sometimes 19% F1 on a five-way task. Probe its hidden states with a classifier instead, and the same model hits 48%. The knowledge is in there. The model just can’t get it out through text generation. On reasoning benchmarks—GSM8K, MATH, and GPQA—probed small models substantially outperformed their prompted counterparts. For binary quality filtering—just distinguishing high from low quality—the probed models closely approximated much larger ones like DeepSeek-V3, reaching 80 to 90 percent F1.

INSPECTOR works much better as a coarse filter than a precise scorer, though. On binary classification—high versus low—it hits 80 to 90 percent F1. Push to the full 1-to-5 scale and that drops to 50 or 60 percent. The authors explicitly position it as a filtering mechanism for data curation, not a replacement for careful human judgment when you need granularity. The best configuration turns out to be simple: average across all token positions, then a logistic regression classifier. More complex classifiers and pooling strategies don’t consistently help. So the deployment story is straightforward—no neural probes, no attention mechanisms.

The Caveats

The coverage around this tends to oversimplify. The caveats deserve attention.

Labeled data requirement. Training the probe requires a dataset of (question, answer, score) triples. The paper uses surprisingly little—fewer than 100 examples per quality level, after balancing. Call it roughly 500 total for full 5-class evaluation, or less if you’re just doing binary high-low filtering. We don’t know yet whether 100 examples is enough for a new domain, or whether that figure is specific to math and science reasoning.

The probe is a separate component. You now have a model plus a classifier to maintain. The classifier may need retraining if your task distribution shifts. This isn’t “free evaluation.” It’s evaluation with a different cost structure.

Task scope. The paper tests reasoning tasks: math problem solving, science QA. Transfer to creative writing, code generation, long-form content, or safety evaluation is untested. Different paradigms entirely.

No bias analysis. The paper doesn’t test whether probed evaluations exhibit the same biases as standard LLM judges—position bias, verbosity bias, and so on. Those biases might be reduced. They might show up in a different form. Unknown.

The probe really is task-specific. The paper does test one form of generalization: training on one math dataset and testing on another. For fine-grained 1-to-5 scores, performance collapses to 10-25% F1. Binary transfer is better—35-62%—but still lands below in-distribution performance. Don’t expect to train on algebra and evaluate geometry without retraining.

Unlike LAGER’s consistent “middle-to-upper” pattern, INSPECTOR finds that optimal layers vary by evaluation aspect. Fluency sometimes peaks in early layers; factuality in late ones. Layer selection becomes more of an empirical search than a principled rule.

The “semantic capacity asymmetry” hypothesis is elegant, but the evidence spans only a few model sizes and architectures. A plausible explanation, but not a settled one.

So INSPECTOR shows that small model probing is viable and can approximate larger judges for binary quality filtering. It needs labeled data and a trained probe, though. The approach hasn’t been validated across diverse task types or bias dimensions either, and its generalization boundaries are narrower than they first appear.

Where This Actually Stands

If you’re already using an open-weight LLM as a judge—say Llama-3.1-8B or Qwen-2.5-7B—LAGER is a low-effort upgrade. Train 33 layer weights, aggregate the logits, and get a continuous score. The setup is minimal and the gains are real. Reasoning doesn’t help—it actually hurts. Hidden states carry the signal more reliably than any chain-of-thought you could prompt. For smaller models in the 1–3B parameter range, the kind that could run on-device, INSPECTOR shows binary filtering is viable. But you need to train and maintain probing classifiers. You still need a capable LLM judge to generate the training labels. Both approaches require access to internal representations. Neither works with API-only models.

Both approaches

Trying It Yourself

So, what do you actually need if you want to try this yourself?

For LAGER, the code is at github.com/sustech-nlp/LAGER. You need a supported open-weight model (Llama or Qwen variants), a small set of examples to calibrate the layer weights, and the ability to extract intermediate logits — which requires model modifications. Think of it as a plug-in for existing evaluation pipelines. Low friction if you’re already in the open-weight ecosystem.

For INSPECTOR, the code is at github.com/zhuochunli/Representation-as-a-judge. You need a small language model (they tested Qwen3 and Llama-3.2 variants), a labeled evaluation dataset for your task — or the willingness to generate one using the two-stage pipeline — training infrastructure for the probing classifier, and validation against held-out data to check probe quality. More setup, more maintenance, but it makes evaluation feasible on smaller, cheaper hardware.

A few questions worth asking before you pick an approach:

Do you need interpretable evaluation? Standard LLM-as-a-Judge gives you chain-of-thought reasoning. It’s not always accurate — we covered that — but it gives you something to inspect. Probing and logit aggregation give you a score and nothing else. If you need to understand why something scored poorly, these methods won’t help.

Is your evaluation task well-defined? If you’re evaluating factual correctness in reasoning tasks, these approaches are validated. If you’re evaluating creative quality, humor, persuasiveness, or safety, these haven’t been tested for those domains.

Do you have labeled data? For INSPECTOR specifically, you need human judgments — or a capable LLM judge to generate them — for training. If you don’t have them, you’re limited to LAGER-style approaches that work with existing models but require open weights.

Can you accept task specificity? The probe or layer weights are optimized for specific evaluation dimensions. If your evaluation criteria change, you need to retrain or recalibrate.

Where This Research Is Headed

Probing and logit aggregation aren’t the only alternatives to standard LLM-as-a-Judge. A few others are worth mentioning for comparison.

Fine-tuned evaluator models. Models like PROMETHEUS-2 and Auto-J are LLMs specifically fine-tuned for evaluation. They’re built to be better judges out of the box. The trade-off: you need to fine-tune them yourself or use someone else’s fine-tuned model, which may not match your evaluation criteria.

Ensemble judges. A 2026 study showed that running multiple LLM judges simultaneously improves alignment with human scores in free-form QA evaluation. More judges means more cost, but potentially more reliable scores.

Rubric-based evaluation. Simple, interpretable, no model needed. For many production use cases—content moderation, quality filtering with clear criteria—this is still the right choice. Don’t use a model if a deterministic rule works.

The probing and logit aggregation approaches fit into this landscape as a specific niche: low-cost evaluation when you already have access to model internals. They’re not a replacement for everything, but they’re a useful tool for the right situations.

So where is this research heading, and what’s still unknown?

Bias analysis. Neither paper systematically tests whether probed evaluations exhibit position bias, verbosity bias, or other known biases. That’s a critical gap before anyone should fully trust these methods.

Task generalization. We need results on creative tasks, code evaluation, long-form content, multimodal inputs. The current evidence is limited to reasoning QA.

Minimum data requirements in new domains. For INSPECTOR, the paper uses ~100 examples per score class for math and science reasoning. Whether that transfers to legal documents, medical diagnoses, or customer support conversations is unknown.

Cross-task transfer. The question of whether you can train a probe on one task and apply it to a related task has a partial answer: poorly for multiclass, modestly for binary. The boundary of what counts as “related enough” is still unclear.

Integration with edge AI. LiteRT-LM and on-device inference are growing rapidly. The combination of small-model probing plus on-device execution is promising, but no one has demonstrated it working end-to-end yet.

Mechanistic understanding. We have correlational evidence that middle layers are better for evaluation, but no causal explanation. For INSPECTOR, why layer preferences vary so much by evaluation aspect remains an open question. Interpretability research could help here.

The code is public, the papers are on arXiv. Read, experiment, and decide for yourself. If you try it, I’d love to hear what you find—especially if something doesn’t work the way the papers claim. That’s how science progresses.

Check Your Understanding

Questions below move from simple recall up to open-ended design, roughly following Bloom’s Taxonomy.

Remember What do LAGER and INSPECTOR each extract from a model in order to evaluate a response—logits from which layers, versus what kind of internal representation?

Understand In your own words, explain the “Semantic Capacity Asymmetry” hypothesis that motivates INSPECTOR—why might recognizing a good answer require less capability than generating one?

Apply Using LAGER’s expected-value scoring method, if a model assigns 30% probability to a score of 3 and 70% probability to a score of 4, what continuous score would LAGER return?

Analyze The article notes that adding chain-of-thought reasoning to LAGER made it perform worse, while a standard LLM-as-a-Judge relies entirely on generating reasoning before scoring. Walk through why “the hidden states carry a pre-reasoning signal” is a plausible explanation for this, and what it implies about where in the model the evaluative judgment actually happens versus where it gets articulated.

Evaluate Both LAGER and INSPECTOR require access to model internals and therefore can’t be used with API-only models like GPT-4 or Claude. Critique the practical impact of this constraint: for a team currently using GPT-4-as-a-judge, what would actually have to change in their model stack to adopt either approach, and is that a reasonable ask?

Create Design a validation experiment to test one of the paper’s open caveats: whether INSPECTOR’s probed evaluations carry the same position bias (preferring the first vs. second answer) as standard LLM-as-a-Judge. What data would you need, and what result would tell you the bias transferred versus didn’t?