Key Takeaway
Forcing multimodal language models to produce structured, concept-based explanations reduces their image classification accuracy, and correct predictions are best signalled by explanations that point to discriminative visual features.
ON THIS PAGE
The Evidence
Requiring richer explanations (from simple labels to formal logic-style rules) steadily lowers classification accuracy. Models often produce syntactically plausible explanations that are not grounded in the image — they mimic the form of a logical answer without the visual evidence. The single judge metric most tied to correct answers is Local Discriminativeness: explanations that cite features that separate the chosen class from the alternatives predict correctness. See the Mutual Verification Pattern for a related approach.
Data Highlights
1Accuracy falls as explanation complexity rises: 93.8% with no explanation requested down to 90.1% when models must produce formal logic axioms (E1→E5).
2Providing 5 labeled support images instead of 1 raises accuracy by +7.0 percentage points (p = 2.0×10^-13) and increases Local Discriminativeness by +0.26 (p = 1.5×10^-6).
3Overall average accuracy across models and conditions is 92.6%, but model behavior varies: one model dropped from 95.1% to 83.0% across conditions while another stayed above 95% in all settings.
Why It Matters
Engineers building AI agents and systems that must both decide and explain should care because asking for formal explanations can harm core task performance. Technical leaders and researchers evaluating model trust and transparency should treat classification accuracy and explanation quality as separate metrics and prioritise explanations that point to class-distinguishing visual evidence. For architectural guidance, consider the Orchestrator-Worker Pattern.
Explore evaluation patternsSee how to apply these findings
Key Figures
Fig 1: Figure 1: Overview of the In-Context Learning Classify-Explain and Evaluate pipeline. The Multimodal LLM acts as both classifier and explainer, while a separate LLM-as-a-judge evaluates nine concept-based XAI metrics (1–5 ↑ \uparrow ): see Table 1 . The example shown corresponds to E3 (feature-based explanation), where the model is prompted to identify the minimal sufficient set of critical, concrete, and observable visual features needed to support the classification. The judge, using a different MLLM and prompt, scores the generated explanation across the same nine evaluation dimensions.
Fig 2: Figure 2: LLM-as-a-judge scores across nine explanation quality metrics by condition (E2–E5), averaged over all models and datasets. DL Axioms (E5) shows a markedly smaller polygon, particularly on Textual Groundedness, Specificity, Instruction Following, and Logical Coherence. Metric abbreviations as in Table 1 .
Fig 3: Figure 3: Spearman ρ \rho between LLM-as-a-judge XAI metrics and classification accuracy across explanation conditions E2–E5. Significance markers indicate Bonferroni-corrected p p -values over 9 × 4 = 36 9\times 4=36 tests: * p Bonf < 0.05 p_{\mathrm{Bonf}}<0.05 , ** p Bonf < 0.01 p_{\mathrm{Bonf}}<0.01 , *** p Bonf < 0.001 p_{\mathrm{Bonf}}<0.001 ; n.s. not significant. Metric abbreviations as in Table 1 .
Fig 4: Figure 4: Mean classification accuracy as a function of Local Discriminativeness (LD) score, pooled across all models, datasets, and explanation conditions (E2–E5).
Ready to evaluate your AI agents?
Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.
Learn MoreLimitations
Results come from four datasets and four multimodal models; behaviour may differ on other domains or larger model families. The judge model scored explanations without seeing the support examples, so metrics like Local Discriminativeness can underestimate explanations that are only meaningful relative to the support set. Pretrained knowledge leakage cannot be fully ruled out under in-context evaluation, so some explanations may reflect memorized knowledge rather than in-context reasoning. For monitoring and guardrails, see Emergence-Aware Monitoring Pattern.
The Details
The study evaluated whether off-the-shelf multimodal language models can both classify images from a few labeled examples and produce concept-based, machine-verifiable explanations. Experiments used a controlled few-shot setup (1- and 5-shot; 2–4 classes) across four image datasets (CIFAR-10, texture, flowers, pet breeds). Models were asked to produce explanations at five levels of formality, from free-text rationales to Description Logic-style axioms. An independent reasoning model acted as a judge and scored nine explanation-quality metrics (for example: textual groundedness, hallucination, local discriminativeness). Findings show a consistent trade-off: asking for richer, more formal explanations reduces classification accuracy (93.8% → 90.1%), while adding more support images boosts both accuracy and the judge's discriminative explanation score. The judge's Local Discriminativeness metric — whether the explanation cites features that separate the chosen class from the alternatives — correlates best with correct predictions. Models often produce syntactically correct formal outputs that lack grounded visual evidence, suggesting current instruction-tuning emphasises surface form over true concept grounding. Practical implication: evaluate explanations separately from accuracy, prioritise explanations that point to discriminative visual features, and consider training or tuning models specifically for grounded, verifiable explanations rather than expecting them to emerge from standard instruction tuning. For design guidance, consider Human-in-the-Loop Pattern and Consensus-Based Decision Pattern to structure verification and decision-making.
Need expert guidance?We can help implement this
Credibility Assessment:
No affiliations or author reputation provided, arXiv preprint with no citations — fits unknown/low-credibility category.