At a Glance
Combine variance-aware payoff corrections with anytime-valid stopping rules to safely stop interactive agent tests much earlier — about 74× fewer hands in realistic poker agent runs — while keeping claims auditable.
ON THIS PAGE
What They Found
Merging a variance-reducing correction for each game with a time-uniform time-uniform confidence sequence makes continuous monitoring valid and sharply reduces the number of games needed. Allowing the value model to update only from past games preserves unbiased corrections, so the variance model can improve online without invalidating results. Two verification modes are provided: an exact bounded certificate when an independent payoff bound exists, and an efficient asymptotic screen that typically stops far sooner. Publishing the corrected payoffs through the stopping time plus metadata lets others recheck any early-stop claim without running new games.
Key Data
1Median variance reduction of corrected payoffs: 54× compared with raw outcomes across 15 poker-agent configurations.
2Median stopping-time reduction: 74× fewer hands needed to reach ±1 big blind precision under the asymptotic screen (α=0.05).
3Naive fixed-sample monitoring produced false positives in 61.35% of simulated leaderboard entries; recomputing the time-uniform intervals on the same selected prefixes overturned essentially all such claims (EB-CS kept 100% and AsympCS 99.9185%).
Implications
Engineers and teams that run costly interactive agent evaluations (for example, multi-agent matches or agent-vs-agent tests) will cut compute and human costs by orders of magnitude. Platform and governance leads benefit too: the protocol gives a reproducible audit trail for any early-stop claim so leaderboards and reports stay trustworthy. reproducible audit trail
Need expert guidance?We can help implement this
Key Figures
Fig 1: Figure 1: Run-level heterogeneity in the 15 HUNL evaluations. Each point is one configuration. Dashed lines mark cross-run medians, and the cross in the EB panel denotes the one ratio censored because the raw stream failed to cross in most reshufflings.
Fig 2: Figure 2: EB-CS width curves on Leduc under different declared bounds. The loose B ′ = 200 B^{\prime}=200 curve nearly equals its deterministic floor 4 B ′ log ( 2 / α ) / t 4B^{\prime}\log(2/\alpha)/t ; the sample-derived B = 22 B=22 curve is descriptive. The analytic B Y = 117 B_{Y}=117 floor anchors are reported in the text.
Fig 3: Figure 3: E2 per-epoch variance under the controlled construction. Predictable past-only learning moves toward the oracle over the observed horizon, while the frozen and raw baselines remain flat.
Fig 4: Figure 4: Injected-effect experiment. A first-crossing fixed CI often claims success before a sequential boundary crosses. Continuing AsympCS under its own stopping rule detects the larger injected effects by 5,000 hands.
Ready to evaluate your AI agents?
Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.
Learn MoreYes, But...
Exact, finite-sample certification requires an independently declared bound on corrected payoffs; when that bound is loose, the exact certificate hits a deterministic precision floor and limits stopping gains. The efficient asymptotic screen depends on variance and regularity conditions and can have nonzero finite-horizon false-positive rates, so it should be calibrated on held-out replications before use. The results assume a fixed evaluated agent and fixed opponent profile; performance under varying opponents or adaptive adversaries needs separate treatment. The finite-sample certification considerations section provides guidance.
Methodology & More
Use a predictable interface that exposes the conditional action probabilities before each action, apply a zero-mean corrective term to each game payoff (so corrected payoffs have much lower variance), and run a time-uniform confidence sequence on the corrected stream. Keep any value-function refitting strictly "past-only" (fit on hands 1..t−1 before using it on hand t) so the correction remains unbiased. Provide two verification streams: an exact bounded-stream certificate that is valid for finite samples when a declared bound on corrected payoffs is set independently, and an asymptotic confidence sequence that is more sample-efficient under standard variance conditions.
On real and controlled poker data (71,439 paired hands across 15 configurations plus full-tree Leduc tests), the corrected stream reduced variance by a median 54× and, under the asymptotic screen, delivered a median 74× reduction in the number of hands needed to reach ±1 big blind precision at α=0.05. A simulated rolling-leaderboard shows naive fixed-sample monitoring reports many false positives (61%), while recomputing the time-uniform intervals on the reported stopping prefixes removes almost all such mistaken claims. Release a corrected-prefix log plus stopping metadata and declared bound so third parties can exactly reconstruct and verify any stopping claim without new games. two verification streams
Need expert guidance?We can help implement this
Credibility Assessment:
All authors have low h-indices, no notable affiliations or top-tier venue; fits the emerging/limited-info category.