Key Takeaway
Running an agent on the same task with and without a skill, then using four targeted diagnostics to rewrite the skill, boosts pass rates and execution reliability—closing almost half the gap to human-written guidance.
ON THIS PAGE
Key Findings
Evaluating skills by contrasting agent behavior with and without the skill reveals specific failure modes (wrong activation, ignored instructions, missing procedural steps, or harmful guidance). Turning those diagnostics into a short Agent Registry Pattern and letting a language model rewrite the skill raised pass rates and made agents far more reliable in executing multi-step workflows. In tests, most gains came from preventing brittle executions (better coverage of solution paths) rather than improving correctness of already-completed solutions. The same loop can be run continuously to build a library of reusable skills for production agents.
Not sure where to start?Get personalized recommendations
Data Highlights
1Pass rate on SkillsBench improved from 32.5% to 41.6% — a 28% relative increase, closing 47–67% of the gap to human-written skills.
2Solution-path coverage (execution reliability) rose from about 49% to 72.7%, while correctness among completed tasks stayed roughly the same (~57%).
3In a continual learning setting on spreadsheet tasks, iteratively refined skills grew a 22-item library that increased pass rate from 16.0% to 52.0% on held-out tasks.
What This Means
Engineers building AI agents: get a practical loop to turn draft skills into reusable, reliable procedures without heavy human oversight. Product and platform leaders: reduce support load and failure spikes by improving agent execution reliability rather than only tweaking outputs. Researchers tracking agent tooling: this gives a measurable, unsupervised path to improve skill libraries and study failure modes.
Key Figures
![Figure 1 : [Top] SkillAxe overview. The agent runs each task with and without the current skill (Phase 1). Four unsupervised metrics diagnose quality impact, trigger precision, instruction compliance, and solution-path coverage (Phase 2). An LLM refiner uses the resulting improvement brief to produce an updated skill (Phase 3), iterating until convergence (Phase 4). [Bottom] Qualitative example: court-form-filling (reward: 0.50 → \rightarrow 1.00). Each row shows a before/after change traceable to a specific metric.](https://arxiv.org/html/2606.10546v1/2606.10546v1/figures/framework.png)
Fig 1: Figure 1 : [Top] SkillAxe overview. The agent runs each task with and without the current skill (Phase 1). Four unsupervised metrics diagnose quality impact, trigger precision, instruction compliance, and solution-path coverage (Phase 2). An LLM refiner uses the resulting improvement brief to produce an updated skill (Phase 3), iterating until convergence (Phase 4). [Bottom] Qualitative example: court-form-filling (reward: 0.50 → \rightarrow 1.00). Each row shows a before/after change traceable to a specific metric.
Fig 2: Figure 2 : UMAP projection of trigger embeddings for three SkillsBench skills. Positive trigger phrases form activation regions (solid hulls); exclusions form neighboring rejection regions (dotted hulls). Task queries near the decision boundary expose ambiguous or over-broad triggers.
Fig 3: Figure 3 : Overall pass rates on SkillsBench (77 tasks, k = 2 k{=}2 trials, Method D). Under both evaluation protocols, SkillAxe substantially outperforms unimproved LLM skills and closes 47–67% of the gap to human-authored skills.
Fig 5: Figure 5 : UMAP projection of the trigger embedding space for six SpreadsheetBench library skills. Same visualization as Figure 2 but within a single domain (spreadsheets). Even semantically related skills ( merge sheets vs. sorting & filtering ) form distinct clusters with separated activation ( + ) and exclusion ( – ) zones.
Ready to evaluate your AI agents?
Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.
Learn MoreYes, But...
Rule-level checks can't always spot when a whole skill teaches a bad strategy that looks internally consistent. Interactions and conflicts between multiple active skills are not modeled, so multi-skill deployments may still need manual review. The system relies on language models as judges and rewriters, which can introduce bias and calibration issues and currently uses a single-update refinement loop rather than many iterative rounds. bias and calibration issues
Full Analysis
SkillAxe evaluates a reusable instruction (a “skill”) by running the same task twice: once without the skill and once with it. It uses a language model to judge the paired outputs and computes four diagnostic scores — overall quality impact, trigger precision (does the skill activate for the right tasks), instruction compliance (does the agent follow the guidance), and solution-path coverage (does the skill support all plausible ways to solve the task). Those diagnostics are compiled into a concise improvement brief improvement brief that a language model then rewrites into a new version of the skill, which can be re-tested and iterated.
On a 77-task benchmark spanning coding, data work, and domain apps, this loop raised pass rates by 28% relative to raw language-model authored skills and closed 47–67% of the gap to human-authored skills. Most of the benefit came from higher execution reliability: agents completed more tasks without breaking the workflow, while the correctness of already-completed tasks stayed about the same. In a production-style continual learning experiment with spreadsheet tasks, SkillAxe built a compact 22-skill library that bumped held-out task success from 16% to 52%, showing the approach scales as a practical engine for accumulating useful procedural knowledge. Limitations include undetected structural misalignment in some skills, unmodeled skill-to-skill conflicts, and dependence on LLM-based evaluation and rewriting, suggesting human oversight or additional execution feedback may still be needed for the last mile of quality.
Explore evaluation patternsSee how to apply these findings
Credibility Assessment:
Contains a top-tier researcher (Sumit Gulwani, h-index 65) which strongly signals credibility despite being an arXiv preprint.