Back to Ecosystem Pulse
EvaluationExperimental
self-bench
by mupt-ai
Run private, repo-driven benchmarks of coding agents using merged PRs
TypeScript
Updated Oct 7, 2026
Share:
Summary
Benchmarks coding agents and models against a project's real history by converting merged pull requests into Harbor-formatted evals. It replays PRs and testcases from your repository to produce repeatable, private evaluations Planning Pattern that reflect real developer workflows. Outputs are designed for local, repo-scoped scoring so teams can measure model performance on their actual codebase.
The Value Proposition
As agents are used to write and modify code, evaluating them on synthetic tasks misses real-world failure modes and developer expectations. Self-bench lets teams run continuous, private evaluations Human-in-the-Loop Pattern against their own merged PRs so agent track records reflect actual repo behavior, not curated benchmarks. This tightens the feedback loop for agent reliability, regression detection, and pre-production testing of code-writing agents.
Best For
Engineering teams who want continuous, private evaluation of code-writing agents on their real repository history Agent Registry Pattern
Real-World Examples
- Validate code-writing agents against past merged PRs to detect regressions
- Continuously evaluate new models on your repository before deployment
- Create private Harbor-formatted evals that reflect real developer workflows
Works With
harbor
Topics
agent-evaluationai-agentsbenchmarkcoding-agentsevalsharborllmllm-evaluationswe-bench
Similar Tools
agent-playground
Keywords
agent-to-agent evaluationA2A evaluationagent track recordrepo-driven benchmarks