The Big Picture
Guiding an AI agent with explicit engineering skills, typed tools, and shared, inspectable artifacts raises end-to-end executable success from 56.8% to 88.6% and makes failures localizable.
ON THIS PAGE
The Evidence
An artifact-centered agent that records every intermediate model, validation, and tool run produces far more reliable, executable engineering results than a generic language-only agent. On a 150-scenario benchmark, the guided workflow matched models and executed backends much more often, while the trace preserved the steps needed to diagnose errors human-in-the-loop evaluation. Remaining failure modes cluster around bad input values and reconstructing models from images or drawings, and one structure family (continuous beams) showed a notable regression under the bundled workflow.
Not sure where to start?Get personalized recommendations
Data Highlights
1Bundled guided workflow success: mean Success Rate 88.6% versus 56.8% for generic-only — a 31.8 percentage-point gain.
2Automatic-mode model-matching: 90.8% (skill selection 100%, structure-type detection 98.0%); generic-mode model consistency only 70.5% despite 92.0% artifact existence.
3Invalid-value cases passed 70.9% of recorded outcomes compared with a 91.0% interactive-case mean; multimodal structure recognition hit 93.6% while model-matching lagged at 84.7%.
What This Means
Engineers building AI assistants for architecture, engineering, and construction should care because traceable artifacts let you run, verify, and debug automated analyses instead of trusting prose, enabling better Explainability and smoother collaboration across teams. Technical leaders evaluating agent reliability can use the benchmark to compare workflow-level behavior, find where handoffs fail, and set release gates that prevent regressions in specific structure types.
Key Figures
Fig 1: Figure 1 : From a structural-engineering request to reviewable evidence. Unlike a text-only response, StructureClaw preserves the skills, tool executions, and artifacts that support the reported result.
Fig 2: Figure 2 : Overall framework of StructureClaw. Multimodal engineering context enters a StructureClaw agent kernel, where capability context, engineering tools, and artifact state support a ReAct-style workflow. The local engineering sandbox executes model construction, validation, analysis, code checking, and report generation, producing structural artifacts and enabling validation-guided refinement.
Fig 3: Figure 3 : Composition of StructureClaw-Bench. The heatmap shows structure-type coverage across task families, the bar chart decomposes interactive robustness cases, and the pie charts summarize multimodal input format and Chinese/English balance.
Fig 4: Figure 4 : Descriptive evaluation profiles. The radial display zooms the 50–100 range: observed scores at or below 50 share the inner ring, and larger scores are linearly expanded toward the outer ring. Appendix C defines the axes and their abbreviations. Gaps denote unevaluated axes; radar area is not an evaluation metric.
Ready to evaluate your AI agents?
Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.
Learn MoreYes, But...
The benchmark is diagnostic, not a certification: passing a scenario does not mean a design is construction-ready and professional review remains required. Each scenario was run once with a 15-minute timeout, so run-to-run variability and statistical rankings are not established. The automatic workflow bundles several changes (skills, priors, validators), so improvements are attributed to the overall system rather than a single component Evaluation-Driven Development.
Methodology & More
StructureClaw organizes structural-engineering work as a chain of inspectable artifacts (interpreted requirements, a computable model, validation records, solver outputs, and a final report) rather than as a single text reply. The system splits responsibilities into domain skills, typed tools, backend providers, and a shared structural-model format so each step produces auditable state that later steps consume. StructureClaw-Bench contains 150 controlled scenarios (half English, half Chinese) across three families: standard workflows, interactive edge cases, and multimodal reconstruction from images or DXF files. Success is measured by assertion-level checks that require the full evidence chain to be consistent and executable in a single run. This design aligns with the A2A Protocol Pattern and reflects a broader Compound AI System approach.
Avoid common pitfallsLearn what failures to watch for
Credibility Assessment:
Contains at least one fairly established author (Wenjie Liao, h-index 23) though venue is arXiv and most authors/affiliations are from a non-top-tier university (Tianjin University). Overall reasonably credible.