The Big Picture
Small language models can predict—before answering—how an external judge would score their reply; giving them a simple performance 'report card' or fine-tuning them makes those predictions reliable enough to route difficult queries to bigger models and save cost.
ON THIS PAGE
Key Findings
Large, reasoning-focused models can estimate how a judge will score their answers right out of the box, but many small models are miscalibrated and guess at random. Supplying small models with a concise performance summary (a 'report card' built from past judge ratings) dramatically improves their self-prediction, and supervised fine-tuning that uses past judge scores can match or exceed report-card performance without the extra input tokens. Prediction quality varies by task: smaller models improve most on tasks where judge scores are concentrated, and models appear better at recognizing harder queries, as reflected in past judge ratings.
Test your agentsValidate against real scenarios
By the Numbers
1Experimental sweep covered 11 different language models across 5 datasets (MedQA, LongFact, AIME 2024, SciCode, MMLU-Pro).
2Reproducing the full set of experiments takes about 48 hours on eight NVIDIA H100 GPUs (as reported in the paper).
3Prior work used as motivation shows a single large-model judge can correlate with human preferences at roughly a 0.8 Spearman coefficient, enabling practical report-card construction.
Why It Matters
ML engineers and product teams building multi-model systems can use these prediction methods to route queries: let low-cost models answer when confident and send the rest to bigger models to save compute and reduce errors. Research teams and ops teams running on-device or low-cost models can use report cards or low-cost fine-tuning to improve reliability without constantly invoking expensive evaluators.
Key Figures
Fig 1: Figure 1 : Overview of the report card framework. Queries are sent to multiple LLMs whose responses are jointly evaluated by an LLM/agentic judge. The judge’s responses are converted into report cards that capture a model’s performance across different query types. At inference time, the model predicts its own capability to answer a new query, enabling cost-efficient routing to a minimally capable model without the characteristic overconfidence of direct self-assessment.
Fig 2: Figure 3 : The distribution of the LLM judge scores for each of the models on the MedQA dataset. Note the wide spread of performance here across the different models.
Fig 3: Figure 7 : The zero-shot and contextual prediction accuracy of all the models on the AIME 2024 dataset (a green arrow indicates improvement over zero-shot). Note the dramatic improvement for smallest models here.
Fig 4: Figure 13 : Distribution of LLM judge scores on AIME 2024 dataset under independent evaluation.
Ready to evaluate your AI agents?
Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.
Learn MoreYes, But...
Experiments are single-turn only; multi-turn conversations may change when a model should ask for help. The approach depends on the choice of an external judge, which can carry biases and limitations of its own. Report cards add input overhead (many tokens) while fine-tuning requires maintaining additional model weights or sacrificing a model instance—trade-offs that must be measured in deployment.
Deep Dive
They propose two simple workflows so a model can predict how an external model-based judge would rate its answer before generating it. Predict-Answer/Act (PA) asks the model to predict the judge score for its would-be reply; Reason-Predict-Reason-Answer/Act (RPRA) adds an initial reasoning step before prediction and another before answering. The team tests these ideas across five diverse datasets and eleven models, using a large reasoning model as the judge. Out of the box, large reasoning models can often predict judge scores reasonably well, but many small models are poorly calibrated and either over- or under-confident.
Two practical fixes work well. First, generate a simple report card: evaluate each small model across a variety of tasks with the judge and feed the modal performance summary to the model at prediction time; that contextual cue markedly improves prediction accuracy for small models. Second, use the hindsight trick to relabel past queries with judge scores and fine-tune small models to predict those scores directly; fine-tuned small models match or exceed report-card performance while avoiding the token overhead of long report-card inputs. The approaches vary in effect by dataset and task difficulty, and the paper frames this as a feasibility study with clear next steps: test routing systems in production, extend to multi-turn dialogs, and explore stronger alignment-specific rubrics.
Test your agentsValidate against real scenarios
Credibility Assessment:
Authors affiliated with Meta (top AI lab) and multiple experienced researchers — strong credibility despite arXiv posting.