Agent Playground is live — Try it here → | put your agent in real scenarios against other agents and see how it stacks up

At a Glance

Language models struggle to turn surprising observations into fully supported physical laws: only one of twelve models discovered altered physics through free exploration, and even its findings were often incomplete.

Core Insights

An automated framework called LongChang put language models into simulated 'glass rooms' with one hidden change to physics and let them explore freely. Only one model (GPT-6-Astra) discovered altered laws in free exploration runs; the other eleven models made no successful discoveries. Even when a model found something, its reports often had strong experimental evidence but failed to state the full law or its exact conditions. Giving a model a hint about law discovery raised its success rate, showing guidance helps convert experiments into correct hypotheses. capability-discovery-pattern
Test your agentsValidate against real scenarios
Learn More

Data Highlights

1GPT-6-Astra achieved a 16.7% discovery rate under free exploration; the other 11 evaluated models scored 0% on discovery.
2GPT-6-Astra produced evidence-supported reports 97.6% of the time, but only 19.0% of its reports were judged correct about the full altered law.
3Adding a law-discovery hint increased GPT-6-Astra’s discovery rate from 16.7% to 28.6% and reduced mean exploration length from 39.3 to 30.9 turns.

What This Means

Engineers building agents that run experiments or investigate unexpected events should care, because models can observe anomalies but fail to turn them into complete, test-backed rules. Technical leads evaluating agent reliability should use separate metrics for evidence support versus correct hypothesis formation. Researchers studying multi-agent evaluation can use LongChang to test how reviewer and designer agents affect discovery. Mutual Verification Pattern Tree of Thoughts Pattern

Key Figures

Figure 1: LongChang’s glass room. Explorers manipulate objects and build tools. Other LLMs evolve scenes using exploration records without human intervention.
Fig 1: Figure 1: LongChang’s glass room. Explorers manipulate objects and build tools. Other LLMs evolve scenes using exploration records without human intervention.
Figure 3: An example of an altered physical law. When the two balls are close, an upward force makes them rise together. Labels 1–5 identify successive snapshots.
Fig 3: Figure 3: An example of an altered physical law. When the two balls are close, an upward force makes them rise together. Labels 1–5 identify successive snapshots.
Figure 5: Case study of research question formation in Scene 15. (a) GPT-6-Astra investigates red’s delayed restart and discovers the altered physical law. (b) Gemini-3.7-Flash observes the restart but still follows its original plan.
Fig 4: Figure 5: Case study of research question formation in Scene 15. (a) GPT-6-Astra investigates red’s delayed restart and discovers the altered physical law. (b) Gemini-3.7-Flash observes the restart but still follows its original plan.
Figure 6: Case study of testing competing explanations.
Fig 5: Figure 6: Case study of testing competing explanations.

Ready to evaluate your AI agents?

Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.

Learn More

Considerations

Results come from simulated environments with intentionally altered laws, so real-world scientific discovery may be harder or different. The evaluation hinges on predefined criteria for evidence support and law correctness, which could bias outcomes toward certain report styles. Only a limited set of scenes and current models were tested, so performance may change with longer runs, different prompts, or future models. Tree of Thoughts Pattern

Deep Dive

LongChang builds small simulated 'glass room' worlds where a single physical law differs from normal physics. Separate language-model agents act as explorers (manipulating objects and building tools), scene designers, and reviewers; explorers receive no task or hint and must notice oddities, form research questions, run experiments, and report laws with experimental support. The framework automatically checks simulated behavior against the hidden laws and uses exploration records to evolve scenes into new challenges. Twelve language models were evaluated. Only GPT-6-Astra found altered laws in free exploration (16.7% of runs); other models recorded observations or built devices but failed to turn anomalies into fully correct, experimentally supported laws. GPT-6-Astra’s reports often had strong experimental evidence yet missed conditions or generality (97.6% evidence support vs 19.0% law correctness). Giving the model a hint that laws might be altered improved identification (28.6%) and shortened explorations, showing that targeted guidance and evaluation that separates evidence from correctness are important. The results highlight gaps in current agent abilities: noticing anomalies is only the first step — designing decisive follow-ups and stating precise, supported rules remain hard. LongChang offers a repeatable way to test agent behavior and agent-to-agent evaluation strategies as systems try to decide what to investigate and how to verify it. Model Context Protocol (MCP) Pattern
Not sure where to start?Get personalized recommendations
Learn More
Credibility Assessment:

One author listed with low h-index (2) and no institutional affiliations; arXiv preprint with no citations — limited but slightly more signal than completely unknown single-author cases.