Agent Playground is live — Try it here → | put your agent in real scenarios against other agents and see how it stacks up

Key Takeaway

Which AI model wrote a cover letter can change hiring outcomes even when candidate quality is identical; cheaper, locally run models are consistently disadvantaged and some screening models amplify that gap.

The Evidence

Evaluator models form a stable ranking of cover-letter writers: applicants who use certain hosted AI models get systematically better scores than those using other models, independent of real qualifications. Locally hosted models (used by people who can’t pay for APIs) are persistently under-ranked. Some evaluator models show strong self-preference, and writer-model mismatch can push weaker applicants ahead of stronger ones at measurable rates. Human-in-the-Loop Pattern

Key Data

1Model mismatch displaced high-fit candidates by up to 7.3 percentage points (about 1.8 out of 25 high-fit candidates displaced in the worst pairing).
2Top writer models led head-to-head win rates around 54–56% (e.g., Gemini 3.5 Flash 56%, GPT-5-mini 56%, DeepSeek V3.2 54%).
3The study ran 414,000 automated evaluations (11 writer models × 9 evaluator models × 10 jobs × 50 candidates across three conditions) to isolate model-of-origin effects.

Why It Matters

Engineers and product leaders building or choosing automated screening tools should care because evaluator choice directly affects fairness and hiring quality. HR and compliance teams should audit which screening models they run and prefer the more neutral evaluators identified here to reduce unequal outcomes for applicants using lower-cost tools. Agent Registry Pattern
Test your agentsValidate against real scenarios
Learn More

Key Figures

Figure 2: Relative Preference Gap Δ pref ​ ( e , w ) \Delta_{\mathrm{pref}}(e,w) : mean residual s ~ e , j , c , w \tilde{s}_{e,j,c,w} averaged over all (job, candidate) pairs, for the cover-letter-only (left) and combined CV + Cover letter (right) conditions. Columns sorted by decreasing gap; bold values indicate p < 0.05 p<0.05 (one-sample t t -test of the 50 per-candidate gaps against 0, per cell); true for 80 80 – 84 % 84\% of cells, so bold is mostly a significance flag: it marks gaps distinguishable from zero, regardless of magnitude. Bold black-bordered cells (a separate marker) mark the self-preference diagonal. Locally hosted models are consistently disfavoured, regardless of licensing; self-preference is strongest in the combined setting.
Fig 2: Figure 2: Relative Preference Gap Δ pref ​ ( e , w ) \Delta_{\mathrm{pref}}(e,w) : mean residual s ~ e , j , c , w \tilde{s}_{e,j,c,w} averaged over all (job, candidate) pairs, for the cover-letter-only (left) and combined CV + Cover letter (right) conditions. Columns sorted by decreasing gap; bold values indicate p < 0.05 p<0.05 (one-sample t t -test of the 50 per-candidate gaps against 0, per cell); true for 80 80 – 84 % 84\% of cells, so bold is mostly a significance flag: it marks gaps distinguishable from zero, regardless of magnitude. Bold black-bordered cells (a separate marker) mark the self-preference diagonal. Locally hosted models are consistently disfavoured, regardless of licensing; self-preference is strongest in the combined setting.
Figure 3: Rank gap g ¯ e , w \bar{g}_{e,w} as a heatmap (evaluators × \times writers) for the cover-letter-only (left) and combined CV + Cover letter (right) conditions; red indicates a writer is over-ranked by that evaluator, blue indicates under-ranked. Rows are ordered by increasing cumulative ∑ w | g ¯ e , w | \sum_{w}|\bar{g}_{e,w}| (least to most biased), shown as the adjacent bar chart; columns follow the same order for every writer that is also an evaluator, so the bold-bordered self-preference diagonal (as in Figure 2 ) renders as an actual diagonal, with the two writer-only, locally hosted models appended at the end. Gemini 3.5 Flash is the most biased evaluator in the cover-letter-only condition but the most neutral once the CV is also available; DeepSeek V3.2 becomes the most biased evaluator in the combined condition. Locally hosted models are persistently under-ranked across all evaluators.
Fig 3: Figure 3: Rank gap g ¯ e , w \bar{g}_{e,w} as a heatmap (evaluators × \times writers) for the cover-letter-only (left) and combined CV + Cover letter (right) conditions; red indicates a writer is over-ranked by that evaluator, blue indicates under-ranked. Rows are ordered by increasing cumulative ∑ w | g ¯ e , w | \sum_{w}|\bar{g}_{e,w}| (least to most biased), shown as the adjacent bar chart; columns follow the same order for every writer that is also an evaluator, so the bold-bordered self-preference diagonal (as in Figure 2 ) renders as an actual diagonal, with the two writer-only, locally hosted models appended at the end. Gemini 3.5 Flash is the most biased evaluator in the cover-letter-only condition but the most neutral once the CV is also available; DeepSeek V3.2 becomes the most biased evaluator in the combined condition. Locally hosted models are persistently under-ranked across all evaluators.
Figure 4: Both panels refer to the combined CV + Cover letter condition. Left: head-to-head win rates WR ⁡ ( w 1 , w 2 ) \mathrm{WR}(w_{1},w_{2}) ; triples where the evaluator matches either writer are excluded to remove self-evaluation confounds. Gemini 3.5 Flash leads (56%), closely followed by GPT-5-mini and DeepSeek V3.2 (56% and 54%); locally hosted models consistently lose. Right: net displacement rate Δ ​ R ​ ( w 2 , w 1 ) \Delta R(w_{2},w_{1}) (percentage points above the same-writer baseline) for the scenario where all high-fit candidates (defined as the top 25 by each evaluator’s own CV-only ranking, recomputed per evaluator, job, and run) use writer w 1 w_{1} (rows), and all moderate-fit candidates (bottom 25 by the same ranking) use writer w 2 w_{2} (columns). Diagonal cells (grey “–”) are the control condition, w 1 = w 2 w_{1}=w_{2} , by definition 0 and omitted. Off-diagonal cells show the mean net rate and, beneath it, ± \pm its standard deviation across the nine evaluators. In the most adverse pairing (Llama-3.1-8B vs. DeepSeek V3.2), approximately two (1.8) out of twenty-five high-fit candidates are displaced solely due to model mismatch.
Fig 4: Figure 4: Both panels refer to the combined CV + Cover letter condition. Left: head-to-head win rates WR ⁡ ( w 1 , w 2 ) \mathrm{WR}(w_{1},w_{2}) ; triples where the evaluator matches either writer are excluded to remove self-evaluation confounds. Gemini 3.5 Flash leads (56%), closely followed by GPT-5-mini and DeepSeek V3.2 (56% and 54%); locally hosted models consistently lose. Right: net displacement rate Δ ​ R ​ ( w 2 , w 1 ) \Delta R(w_{2},w_{1}) (percentage points above the same-writer baseline) for the scenario where all high-fit candidates (defined as the top 25 by each evaluator’s own CV-only ranking, recomputed per evaluator, job, and run) use writer w 1 w_{1} (rows), and all moderate-fit candidates (bottom 25 by the same ranking) use writer w 2 w_{2} (columns). Diagonal cells (grey “–”) are the control condition, w 1 = w 2 w_{1}=w_{2} , by definition 0 and omitted. Off-diagonal cells show the mean net rate and, beneath it, ± \pm its standard deviation across the nine evaluators. In the most adverse pairing (Llama-3.1-8B vs. DeepSeek V3.2), approximately two (1.8) out of twenty-five high-fit candidates are displaced solely due to model mismatch.
Figure 5: Left: cross-evaluator Ridge coefficient for each feature (cover-letter-only condition; Cosine Similarity to CV excluded, see text), one dot per evaluator plus a bold bar for the cross-evaluator mean, ranked top to bottom by that mean; rows shaded when all nine evaluators agree in sign (green = unanimous reward, red = unanimous penalty). Right: each writer’s own mean use of that feature (mean ± \pm 1 std), standardised to the same units. Row labels are coloured by feature category. As throughout, Cosine Similarity to Job Ad/CV measures the generated letter’s embedding against the job ad/CV. It is not the job ad and CV measured against each other (Figure A1 ). Cosine Similarity to Job Ad is both the most strongly and the most unanimously rewarded feature by a wide margin; Paragraph Count, Anger, and Neutral are the clearest unanimous penalties.
Fig 5: Figure 5: Left: cross-evaluator Ridge coefficient for each feature (cover-letter-only condition; Cosine Similarity to CV excluded, see text), one dot per evaluator plus a bold bar for the cross-evaluator mean, ranked top to bottom by that mean; rows shaded when all nine evaluators agree in sign (green = unanimous reward, red = unanimous penalty). Right: each writer’s own mean use of that feature (mean ± \pm 1 std), standardised to the same units. Row labels are coloured by feature category. As throughout, Cosine Similarity to Job Ad/CV measures the generated letter’s embedding against the job ad/CV. It is not the job ad and CV measured against each other (Figure A1 ). Cosine Similarity to Job Ad is both the most strongly and the most unanimously rewarded feature by a wide margin; Paragraph Count, Anger, and Neutral are the clearest unanimous penalties.

Ready to evaluate your AI agents?

Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.

Learn More

Yes, But...

The experiment uses API-hosted models as evaluators while locally hosted models only act as writers, so results reflect real-world asymmetry but not all possible hosting configurations. Candidate quality was fixed using embedding similarity to the job ad, a practical proxy that may miss some nuances of fit. Effects reported are within the subset of applicants who used AI; candidates who avoid AI altogether may face even larger disadvantages, so these numbers are likely a lower bound. Vector Database

The Details

A controlled audit tested whether the identity of the writing model — not the applicant’s skills — changes automated screening outcomes. Eleven writer models (nine accessed via provider APIs and two run locally) generated tailored cover letters for 50 candidates across 10 job postings. Nine of the API-hosted models also served as evaluators and scored every candidate under three conditions: CV only, cover letter only, and CV plus cover letter. Using a fixed, external measure of candidate quality (embedding similarity to the job ad) isolated the effect of which model produced the letter. Findings show a persistent preference hierarchy: some writer models consistently get better scores, while locally run models lose out. Evaluators vary in how much bias they introduce, and many reward stylistic features such as closeness to the job ad’s language and readability. In head-to-head tests and a rank-displacement analysis, writer-model mismatch alone could bump weaker candidates ahead of stronger ones — up to ~7.3 percentage points in the worst case. Practical implications: employers are blind to applicants’ tool choices and should adopt neutral evaluation models and prospective audits to avoid unintentionally favoring better-resourced applicants; engineers can use the paper’s auditing framework to test new model combinations before deployment. A2A Protocol Pattern auditing framework
Test your agentsValidate against real scenarios
Learn More
Credibility Assessment:

ArXiv preprint, no affiliations, modest author h-indices (up to 6) and zero citations — recognizable authors but not strongly established.