At a Glance
Index-backed retrieval (simple semantic search) answered more repository questions correctly and at lower cost than delegation-based multi-agent exploration for read-only, indexable codebases.
ON THIS PAGE
What They Found
Across 15 Python repositories and four large language models, a retrieval agent that uses a pre-built index outperformed a hierarchical agent that spawns sub-agents to explore the repo. index-backed retrieval reached a 65.2% pass rate versus 46.2% for the deep agent, and it produced fewer outright failures while using fewer input tokens. The deep agent introduced a new, silent class of coordination failures when passing context between planner and sub-agent, which often made it less reliable despite its ability to explore live files.
Not sure where to start?Get personalized recommendations
Data Highlights
1Semantic search pass rate 65.2% vs Deep Agentic search pass rate 46.2% (pooled across four models).
2Deep Agent produced a higher fail rate: 34.4% vs Semantic 21.9% (pooled).
3Index upkeep is cheap in the study: embedding all 15 repositories cost under $1 total; delegation was 7.0% of Deep calls but launched sub-agent traces responsible for 68.8% of Deep tool calls.
What This Means
Engineers building code assistants should prefer an index-backed retrieval agent as a default for read-only questions over repositories that can be indexed, because it’s both more accurate and cheaper. Technical leads evaluating agent stacks should reserve complex delegation-based designs for cases that truly require live exploration or long multi-task sessions where keeping a clean context matters.
Key Figures
Fig 1: Figure 1: The two paradigms compared in this study. (a) Semantic search, realised by a ReAct agent, which retrieves pre-filtered chunks from a vector index built in advance. (b) Deep Agentic search, realised by a Deep Agent, which builds no index and delegates exploration to a sub-agent that works in its own context window and returns only a condensed result. The designs differ in what reaches the main context of the agent that answers.
Fig 2: Figure 2: Agent architectures. A ReAct agent is a loop between a language model and a set of tools. A Deep Agent contains that same loop and extends the tool set with planning, a virtual filesystem, and the ability to delegate work to sub-agents.
Fig 3: Figure 3: The action spaces of the two agents. Both act on the same target repository, and they differ in what they are allowed to do to it. The ReAct agent is held to the three tools of the benchmark protocol, namely a file reader, a repository structure lookup, and retrieval over the vector index. The Deep Agent has the default tool set of its harness, which replaces retrieval with terminal style exploration and adds planning and delegation to a sub-agent.
Fig 4: Figure 4: Sensitivity of the accuracy comparison to the Pass threshold. Each line is the Pass rate of Semantic minus the Pass rate of Deep for one model, as the cutoff is swept from 50 to 90 on the 0 to 100 judge scale. The dashed line marks the benchmark cutoff of 70. The magnitude of the gap depends on the cutoff, but every line stays above zero, so the ordering does not.
Ready to evaluate your AI agents?
Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.
Learn MoreYes, But...
Findings apply to read-only question answering on repositories that can be indexed in advance and may not hold for long, multi-turn sessions or tasks that write or execute code. Each paradigm was tested with a specific open-source agent harness, so different implementations or prompt/config choices could change magnitudes. The study used fixed repository snapshots, so the practical trade-offs around index staleness and incremental updating require operational judgment. Versioning
Methodology & More
An experiment compared two common ways agents find information in a codebase: a simple semantic retrieval over a pre-built index, and a hierarchical “deep” agent that explores the live filesystem by delegating to sub-agents. The test used 15 Python repositories, 720 questions from a benchmark, four large language models, and identical judging criteria. Every run recorded tool calls, tokens, and full execution traces so accuracy, cost, and failure modes could be measured together. Results were consistent: index-backed retrieval scored higher on accuracy for every model and question type while consuming fewer tokens and costing less per correct answer. The deep agent’s extra machinery didn’t reduce failures; instead it introduced coordination failures at the planner-to-sub-agent hand-off that often produced confidently wrong answers. The practical takeaway is to default to a cheap index-backed retrieval agent for read-only, indexable repos and to use delegated exploration only when you cannot index the repository, need live reads, or must preserve a clean orchestrator context across many tasks. A Orchestrator-Worker Pattern that uses an index by default and falls back to exploration where needed is a natural next design to try.
Avoid common pitfallsLearn what failures to watch for
Credibility Assessment:
No specified affiliations and low author h-indexes; arXiv preprint with no citations — limited institutional/reputation signals.