At a Glance
Combining feedback for multiple evaluation criteria turns helpful, specific critiques into generic advice, so joint prompt tuning usually stalls or harms judge quality—tuning each criterion separately works better right now.
ON THIS PAGE
What They Found
Optimizing a single prompt to judge multiple quality dimensions at once often fails: in most configurations the optimized prompt never beat the original generic prompt. Two distinct failure modes explain why: (1) when the feedback generator must summarize multiple criteria in one call, its suggestions lose task focus (optimization-time gradient dilution), and (2) when independently optimized task instructions are merged, longer or louder rubrics dominate and drag down others (inference-time instruction interference). Diagnostics show the optimizer reliably follows whatever feedback it receives, so the core problem is poor, diluted gradient signals rather than optimizer noncompliance. A2A Protocol Pattern
Not sure where to start?Get personalized recommendations
By the Numbers
16 of 10 mode×validation setups never improved beyond the initial generic prompt.
2Single-task optimization with MAE validation produced the only clear per-task gain: +0.031 Spearman at step 2.
3Gradient specificity fell 59% when gradients were produced jointly (9.0 → 3.7 on a 1–10 scale).
What This Means
Engineers building automated evaluators or score-based agents should care because joint prompt tuning can silently degrade per-criterion accuracy; keep multi-criterion diagnostics in your pipeline. Technical leaders and researchers planning to scale judge prompts across many metrics need to weigh the trade-off between cheaper single-prompt evaluation and the quality loss documented here. Guardrails Pattern
Key Figures
Fig 1: Figure 1: Overview of the optimization pipeline. Each step consists of four stages: (1) the task model predicts scores, (2) the loss LLM critiques predictions against ground truth, (3) the gradient LLM converts critiques into instruction edits, and (4) the optimizer LLM rewrites the prompt. The decomposition mode determines whether each stage operates per-task (Separate) or over all tasks jointly (Combined). Only the per-task instruction text is updated; the prompt skeleton and output format remain frozen throughout optimization.
Fig 2: Figure 2: Per-task Spearman ρ \rho over optimization steps on SummEval , averaged over N = 3 N\!=\!3 runs (shaded bands show min–max). Columns: five decomposition modes (Single through CCC). Top row: MAE validation. Bottom row: val=none . Gray line: task-averaged ρ \rho ; stars mark best step. Black diamonds (right axis, multi-task modes only): accumulated hypervolume indicator. Without validation, multi-task modes degrade; with it, they plateau. Only Single improves.
Fig 3: Figure 3: Gradient specificity (1–10 scale, higher is more task-focused) by decomposition mode. Per-task modes (Single, SSS, SSC) produce highly specific gradients (mean 9.0). All-task modes (SCC, CCC) drop to 3.7, a 59% reduction. Error bars: ± \pm 1 std across seeds and steps.
Fig 4: Figure 4: Gradient specificity for SSC vs. SCC after swapping the gradient LLM to DeepSeek-V4-Pro while keeping the other stages on Qwen. Bar values: SSC 8.82 ± 0.10 8.82\pm 0.10 , SCC 4.22 ± 0.26 4.22\pm 0.26 . Overall drop: 52%.
Ready to evaluate your AI agents?
Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.
Learn MoreConsiderations
Results come from experiments on one benchmark (SummEval) with four criteria and a small number of runs (N=3), so effect sizes may vary in other settings. Gradient specificity and adherence were scored by an LLM evaluator, which could introduce evaluator bias. Fixing only one failure mode (e.g., per-task inference to avoid interference) reduces the problem but can increase compute cost; combined architectural fixes are likely required for robust multi-criteria tuning. Multi-Agent Research Synthesis
Methodology & More
They tested a four-stage prompt-rewriting pipeline that: (1) scores examples with a task model, (2) has a loss model critique those scores, (3) has a gradient model turn critiques into edit suggestions, and (4) an optimizer rewrites per-task instruction text. The key design choice was whether each stage handled tasks separately or all together; five decomposition modes span combinations from fully separate to fully combined. Experiments used SummEval (four judge criteria) and measured Spearman rank correlation versus human scores, with two validation modes (strict MAE gating and no gating). Across 3 seeds and 12 optimization steps, most multi-task setups either stagnated or degraded; only single-task runs with MAE validation produced a meaningful improvement (+0.031 Spearman). Tree of Thoughts Pattern Blackboard Pattern
Avoid common pitfallsLearn what failures to watch for
Credibility Assessment:
Affiliations include Amazon and an NSF-Simons institute (recognized institutions), giving stronger credibility despite low author h-indexes and arXiv venue.