Key Takeaway
An automated Judge model can reliably reproduce human rankings of multimodal AI outputs (images, audio, text, video) while giving structured error labels and natural-language feedback, making large-scale evaluation and continuous improvement practical.
ON THIS PAGE
What They Found
The Judge produces 0–5 scores plus error types and plain-language explanations, and its rankings match human annotators across four modalities. Its absolute scores are slightly lower than human averages—about 0.1–0.3 points—because it penalizes mismatches between answers and their justifications. The Judge preserves relative ordering of systems (so it’s reliable for comparing models) and can generate supervision signals useful for model fine-tuning or automated monitoring.
Explore evaluation patternsSee how to apply these findings
Data Highlights
1Judge outputs are on a 0–5 Likert scale and include structured error types and natural-language explanations.
2Judge scores are systematically 0.1–0.3 points lower than the average of three human annotators, reflecting stricter penalties for inconsistent reasoning.
3Evaluation covered 3 large models (Gemini-2.5, Phi-4, Qwen-2.5) across 4 modalities (text, audio, image, video); the Judge preserved the relative ranking of models in every modality.
Why It Matters
Engineers building or monitoring multimodal AI systems can use the Judge to scale evaluation without needing as many human raters. Evaluation teams and researchers can treat Judge outputs (score + error type + explanation) as ready-made signals for fine-tuning models, automated monitoring, or pre-production testing.
Key Figures
Fig 1: Figure 1: System architecture of the MLLMs evaluation framework. The framework takes multimodal inputs and task instructions to query multiple tested MLLMs. Each model produces an answer along with its justification. A judge model, following evaluation instructions, assesses the responses and outputs scores (0–5) and error analyses.
Fig 2: Figure 2
Fig 3: Figure 3: The input chart example for image case study.
Ready to evaluate your AI agents?
Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.
Learn MoreLimitations
The Judge is slightly conservative in absolute scoring and may need calibration if you want human-like numeric scores. The benchmark used small, fixed samples from public datasets, so broader generalization needs more testing on larger or domain-specific data. The current work evaluates the Judge in a forward-pass setting only—closing the loop to fine-tune models with Judge feedback was proposed but not validated here.
Deep Dive
A scalable evaluation pipeline was built to judge multimodal model outputs: for each example (text, image, audio, or video), tested models generate an answer and a justification; the Judge compares those outputs to ground truth and the underlying multimodal evidence and returns a 0–5 score, an error type, and a natural-language explanation. The benchmark pulls fixed-size samples from public datasets to avoid accidental overlap with training data and evaluates three large multimodal models across four modalities. Results show the Judge closely tracks human annotators in relative terms—models that humans rank higher are also ranked higher by the Judge—while giving slightly lower absolute scores (about 0.1–0.3 points lower on the 0–5 scale) because it strictly penalizes reasoning inconsistencies and hallucinations. Because each judged instance includes a scalar score, an error category, and an explanation, those outputs can be reused as supervision signals to fine-tune models, build reward models for reinforcement learning from model feedback, or support continuous agent evaluation and monitoring. Future work aims to close the loop by using Judge feedback to directly improve models and to extend evaluation from single answers to full multi-turn agent interactions.
Avoid common pitfallsLearn what failures to watch for
Credibility Assessment:
No clear author reputation or established venue — only arXiv.