At a Glance
Evaluating language models by how their probability forecasts and updates match human behavior in live prediction markets reveals capabilities (accuracy, calibration) and failure modes (drift, overreaction) that static benchmarks miss.
ON THIS PAGE
Key Findings
Measuring models inside live prediction markets captures how they form and revise beliefs over time, not just single-shot answers. Consensus-Based Decision Pattern High-capacity models tend to predict outcomes more accurately but update more wildly and cost more to run, while smaller models are steadier and cheaper but less nuanced. Immutable, versioned prompts and a Capability Attestation Pattern pattern prevent training-data leakage and let teams compare models across nearly one million forecast updates and identical events.
Need expert guidance?We can help implement this
Data Highlights
1Collected nearly 1,000,000 probability updates across deployed evaluations.
2Deployed and compared 8 frontier-scale models simultaneously on identical events.
3Used versioned immutable prompts (cryptographic-hash backed) and example cycles (e.g., 466 update cycles) to ensure repeatability and prevent prompt contamination.
What This Means
Engineers building forecasting or decision-support agents who need models that not only answer correctly but update sensibly under new information. Technical product leaders and ops teams evaluating model trade-offs for production—accuracy versus stability versus cost—can use this to pick models and monitoring strategies. Researchers tracking agent behavior and calibration can use live-forward tests to find realistic failure modes.
Key Figures
Fig 2: Figure 2 : Adjusted cumulative P&L versus historical baseline (left) and realized cumulative trading P&L (right).
Fig 3: Figure 3 : Distribution of strategy selection frequency by model.
Fig 4: Figure 4 : Average input and output token consumption by model.
Fig 5: Figure 5 : Depth of model adjustments across edge, expected return, and probability dimensions (log scale).
Ready to evaluate your AI agents?
Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.
Learn MoreConsiderations
Prediction markets reflect crowd beliefs and financial incentives, so model performance there may not generalize to tasks without similar social or economic context. Instruction locking prevents prompt leakage but does not eliminate all sources of evaluation bias or distributional difference from real deployments. Running continuous live evaluations is resource and cost intensive and requires careful selection of events and liquidity levels to yield meaningful comparisons.
The Details
TruthTensor evaluates language models by embedding them as forecasting agents in live prediction markets and tracking how their probability forecasts and updates evolve over time. Prompts are locked and versioned so every model answers identical, forward-looking questions whose outcomes are unknown at prediction time—eliminating the risk that models rely on memorized results. Events are grouped by risk, domain, horizon, and market liquidity so comparisons reveal where models excel or fail (for example, short-term political bets versus long-term technological outcomes).
The system measures not just final accuracy but calibration (how well stated probabilities match outcomes), temporal coherence (how consistently beliefs change), and drift/volatility in updates. Across roughly one million updates and eight models, higher-capacity models showed deeper, often more accurate reasoning but were also more volatile and more expensive to run; smaller models yielded smoother, cheaper forecasts but with less nuance. The platform is practical for model selection and monitoring: it surfaces trade-offs you won’t see in static benchmarks and makes it possible to build model track records and alerting based on real-time belief trajectories. Future extensions could add persona-driven digital twins to simulate particular human decision patterns. Tree of Thoughts Pattern LLM-as-Judge
Need expert guidance?We can help implement this
Credibility Assessment:
Contains an author with moderate h-index (Haruna Isah ~14) — somewhat recognized but not top-tier.