The Big Picture
A structured multi-agent, test-driven harness converts expert-written rule interpretations and labeled models into executable, inspectable compliance checkers, boosting average element-level accuracy from 46.8% to 84.98%.
ON THIS PAGE
Key Findings
A deterministic multi-agent loop (planner, generator, evaluator) that follows test-driven development produces far more reliable compliance checkers than single-shot or unstructured feedback approaches. The full ARCHER harness raised mean accuracy from 0.4681 to 0.8498 across ten real requirements and four model deployment tiers. Deterministic orchestration outperformed delegating control to a language model, and decomposing feedback across specialized agents rescued weaker self-hosted models, bringing them close to frontier API performance. Cost versus accuracy favors self-hostable models at many budgets when paired with the ARCHER harness. test-driven development ARCHER harness
Test your agentsValidate against real scenarios
Key Data
1ARCHER (Harness 5) raises mean union accuracy from 0.4681 to 0.8498 — a +38.2 percentage-point (82% relative) improvement.
2Frontier-tier model with structured feedback (GPT-5.5 under test-driven refinement) reached 0.9468 union accuracy.
3Self-hosted DeepSeek-v4-flash jumps from 0.6486 (Harness 3) to 0.9542 under ARCHER, while an open on-premise model collapsed from 0.4754 to 0.1719 under unstructured feedback.
What This Means
BIM engineers and developers who need repeatable, inspectable rule checks can use this harness to turn expert knowledge into runnable checkers without hand-coding every rule. Technical leads and compliance teams evaluating cloud vs on-premise options will find the cost–accuracy and auditability trade-offs useful for choosing a deployment tier. inspectable rule checks Human-in-the-Loop
Key Figures
Fig 1: Figure 1: End-to-end architecture of ARCHER. Left: Domain experts digest regulatory Codes of Practice and BIM models to establish the Rule Interpretation ( r r ) and labelled training and test pairs ( D t r a i n D_{train} , D t e s t D_{test} ). Right: ARCHER’s deterministic multi-agent harness drives Planner, Generator, and Evaluator agents using sandboxed tools and workspace files to iteratively synthesize, evaluate, and refine Python code ( main.py ), allowing up to 30 iterations and max 2 replans, outputting an Executable BIM Checker ( p p ). The final BIM checker Python script is evaluated on the held-out test pairs ( D t e s t D_{test} ).
Fig 2: Figure 2: Test union accuracy versus harness sophistication, pooled across the 10 scenarios; exact values are annotated at each point and the dashed line shows the overall trend (mean over the four backbones). ARCHER (Harness 5) is the best configuration for every backbone model, lifting the self-hosted DeepSeek-v4-flash to within 2.2 points of the frontier tier.
Fig 3: Figure 3: Mean token consumption across scenarios by harness configuration and backbone model (log scale). Single-agent harnesses (0–3) stay within a narrow band; multi-agent harnesses (4–5) grow by an order of magnitude, and the growth is steepest for the weakest backbone.
Fig 4: Figure 4: Cost–accuracy trade-off across all 24 (harness, model) configurations. Each marker is one run of a backbone model under one harness; the small number beside each marker is the harness ID (0–5, cf. Table 1 ), so, e.g., a green triangle labeled “5” is DeepSeek-v4-flash under ARCHER (Harness 5). The dashed line traces the Pareto frontier: configurations for which no alternative is both cheaper and more accurate. The frontier is populated almost entirely by the self-hostable models at low-to-mid budgets and by DeepSeek-v4-flash and GPT-5.5 under ARCHER at the high end.
Ready to evaluate your AI agents?
Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.
Learn MoreYes, But...
Domain experts must author the plain-language rule interpretations and label training/test models up front; that human setup effort is not included in the cost analysis. Each rule is synthesized and evaluated in isolation, so cross-rule dependencies or contradictions are not handled yet. The generated checkers are executable Python with geometry code, so non-programmers audit via the expert-facing artifacts while engineers inspect the produced source. Human-in-the-Loop
Deep Dive
ARCHER accepts two expert-authored artifacts: a plain-language Rule Interpretation that disambiguates a regulation and a set of BIM models labeled with per-element verdicts (pass, fail, alert, exempt, or manual check). A deterministic harness then runs a planner, generator, and evaluator in a loop using test-driven development: generate a checker, run it against labeled examples, score the outputs, and iteratively refine until a checkpoint is selected. The system caps iterations (up to 30) and replans, emits an executable Python checker, and evaluates it on held-out test models. Experiments covered ten real requirements and four backbone model tiers representing cloud frontier APIs, budget APIs, self-hosted open models, and strict on-premise setups, with costs normalized to a common per-token proxy. planner, generator, and evaluator mutual verification
Not sure where to start?Get personalized recommendations
Credibility Assessment:
ArXiv preprint with no listed affiliations and low h-index for known author (h=3); limited signals of established reputation.