At a Glance
When evaluated on real, multi-stage hospital stays where models must actively ask for information, the best large model only achieves about 0.31 decision F1 — and models find diagnoses much easier than actual patient management.
ON THIS PAGE
Core Insights
ClinEnv converts real electronic health records into multi-stage, interactive clinical scenarios where an AI must query four specialist agents (patient, nurse, lab, history) before making decisions. The strongest model reaches roughly 0.31 decision F1 overall, with diagnosis recovery far higher than management actions (about 0.51 vs 0.17). How much a model asks for and how efficiently it uses tests and labs are separate, measurable skills that predict different failure modes than outcome-only tests.
Test your agentsValidate against real scenarios
Key Data
1Best decision-level F1 across models: 0.306 (about 31% correct on staged decisions).
2Diagnosis vs management gap: diagnosis F1 ≈ 0.511 versus management F1 ≈ 0.17.
3Dataset scale: 3,509 admissions → 9,297 decision stages and 26,043 ground-truth decisions; 49.8% of cases are long-horizon (3+ stages).
Implications
Engineers building AI systems that will interact with clinical workflows should use ClinEnv to test whether a model can actively gather information and make sequential decisions, not just answer single-step questions. Technical leaders and evaluation teams can use the benchmark to compare models on both outcome accuracy and process behavior (how many queries they make, how much lab waste they cause) before any pre-production deployment.
Key Figures
Fig 1: Figure 1: Overview of ClinEnv . Patients’ admissions are preprocessed into event timelines, converted into multi-stage cases via a five-step pipeline, and evaluated in an interactive environment where the model queries specialized agents before committing decisions, scored on both process and outcome quality.
Fig 2: Figure 2: Per-stage performance on ClinEnv . Decision F1 (a) declines monotonically with stage index. Information coverage (b) rises; coverage efficiency (c) falls. Shaded bands are standard errors.
Fig 3: Figure 3: Coverage-waste on ClinEnv . Information coverage is inversely related to laboratory waste: models that gather more relevant information also waste less.
Fig 4: Figure 4: Frequent clinical entities in ClinEnv . Top diagnosis, medication, and procedure entities among structured ground-truth decisions.
Ready to evaluate your AI agents?
Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.
Learn MoreYes, But...
Scores measure agreement with the actual recorded care in one U.S. academic center, so a reasonable alternative clinical choice could be marked as incorrect. Cases use U.S. coding, pricing, and English notes, which limits straightforward generalization to other countries or systems. Some construction steps use large models, but structured ground-truth labels (drug names, diagnosis codes) are derived deterministically from the electronic health records to avoid circular evaluation. Ground Truth
Deep Dive
ClinEnv turns real hospital admissions from a widely used research database into multi-stage, interactive cases. For each stage an AI must actively query four specialized agents — a simulated patient, nurse, laboratory, and history — before committing to diagnoses, medication actions (start, stop, switch, adjust), or procedures. The benchmark matches submitted decisions to the recorded clinical course using deterministic rules: diagnosis and procedure matching follows a hierarchical code comparison, while medications use an action gate plus a drug-class hierarchy for partial credit. The environment also logs process metrics such as information coverage, number of queries, and laboratory cost or waste so that how the model gathers evidence is scored alongside what it ultimately decides. Tree of Thoughts Pattern across seven large models, performance is modest: the best model scores about 0.306 decision F1 overall, recovers diagnoses much better than management actions (≈0.51 vs ≈0.17), and struggles more on longer cases with three or more stages. Models differ widely in information strategy — some achieve similar accuracy while making many more queries and wasting more labs, others are more parsimonious. These results show that single-step, outcome-only benchmarks can overstate clinical readiness: sequential reasoning and appropriate information gathering, and reliable action choices. ClinEnv makes those dimensions measurable and comparable, helping teams find where an agent actually fails before fielding it in care settings. information coverage
Test your agentsValidate against real scenarios
Credibility Assessment:
ArXiv preprint with no specified affiliations and authors with low h-index (8–9). Limited provenance and no citations suggest emerging/limited information.