Key Takeaway
A compression-based, order-sensitive comparison of raw model outputs predicts when pairs of models will make the same wrong answers on other tasks — beyond surface similarity and overall capability.
ON THIS PAGE
What They Found
Measuring how similar two models are at the level of the sequence structure in their raw outputs (after removing simple frequency effects) identifies pairs that are more or less likely to fail together on separate benchmarks. That order-specific compression residual is distinct from ordinary semantic similarity semantic similarity and explains correlated failure across ten disjoint benchmark families. The negative association is consistent across all ten benchmarks, suggesting the measure captures process-level differences relevant to redundancy.
Explore evaluation patternsSee how to apply these findings
Data Highlights
138 models evaluated on 100 open-ended prompts; correlated failure tested on 10 disjoint benchmark families
2Cross-benchmark mean association between order-specific compression residual and correlated failure: −0.216 (95% interval: [−0.309, −0.122])
3Semantic distances ranged narrowly [0.159, 0.388] while raw compression distances spanned [0.484, 0.859], showing different population geometry
Why It Matters
Engineers building multi-model or multi-agent systems should use this as a lightweight black-box audit to find models whose failures are likely independent, improving redundancy and resilience. Technical leaders and safety teams can use the signal to prioritize pairing models that reduce common-mode failure when assembling ensembles or agent teams.
Key Figures
Fig 2: Figure 2: Semantic, raw-compression, and order-specific views of the same responses. The lower matrices show semantic distance (left) and raw paired PPMd NCD (right). The upper-right matrix shows the order-specific compression distance obtained by residualising NCD against the within-prompt byte-permutation control and averaging across prompts. 20 of 38 models are displayed to simplify visualisation, reported matrix associations use the 703 off-diagonal pairs. Annotated boxes contain response excerpts and illustrate positive, near-zero, and negative residuals. Positive values indicate pairs further apart in sequential organisation than marginal byte composition predicts, and negative values indicate pairs closer than predicted.
Fig 7: Figure 7: Pairwise effect of residualisation. All 703 model pairs are plotted against semantic distance. Point colour records raw NCD and point size also records semantic distance; both encodings are held fixed across panels. The panels show raw NCD, its permutation control, and the retained order-specific residual. Rank associations with semantic distance are printed within each panel.
Fig 9: Figure 9: Within-model semantic and compression variability. (a) Each point is one model, positioned by its mean semantic distance and mean raw NCD over 100 prompts. Horizontal and vertical bars are 95% prompt-bootstrap intervals. The line is the OLS fit across the 38 model means, the band refits that relationship in each prompt-bootstrap sample, and colour records the separately fitted mean intra-model order-specific residual from Equation 22 . (b) Model-specific NCD deviation δ m \delta_{m} from the fitted semantic trend. Intervals resample prompts jointly across models and refit the trend in every replicate. Positive values indicate more compression variability than semantic distance predicts; negative values indicate less. The figure therefore displays both the shared response-variability component and the model-specific departures from it.
Ready to evaluate your AI agents?
Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.
Learn MoreConsiderations
The result is observational and pairwise within the evaluated model set; it does not prove that picking distant models will always improve end-to-end ensemble performance. Measurements were taken at the model input–output layer using one compressor (PPMd) and a permutation control; different compressors or harness-level traces (tool use, memories, communication) could change results. The method flags effective redundancy risk but further tests are needed before using it as a deployment gate for production systems. Context Drift
Deep Dive
Estimate how different two model configurations are by compressing their raw text outputs and comparing compressed lengths. To focus on sequence structure rather than simple output frequency, the method subtracts a permutation-based control that preserves byte frequencies but destroys order; the remaining order-specific residual is taken as a proxy for inferred differences in the generative process that produced the outputs. Diversity is measured from 38 models responding to 100 open-ended prompts, and the residual distances are tested as predictors of whether model pairs produce the same wrong answers on ten separate closed-form benchmark families, while controlling for semantic similarity and capability. To further contextualize, consider applying A2A Protocol Pattern to coordinate between agents, and examine potential Chain of Thought Pattern implications in complex reasoning tasks.
Avoid common pitfallsLearn what failures to watch for
Credibility Assessment:
ArXiv preprint with two authors but no affiliations or known reputation; limited signals of established credibility.