The Big Picture
Smaller, instruction-tuned models often follow strict roles and courtroom procedures better than larger models; RPA-Check converts role requirements into 38 yes/no checks to measure role fidelity, procedural stability, and language quality.
ON THIS PAGE
The Evidence
You can measure whether a conversational AI actually behaves like its assigned persona by turning high-level role descriptions into a checklist of concrete, verifiable questions. In a courtroom simulation, smaller 8–9 billion parameter models that were well instruction-tuned kept roles and procedures more reliably than several larger 12–14 billion models, which produced richer arguments but often broke procedural rules. Language quality was high across all models, so the real discriminators were procedural stability and adherence to role constraints. The checklist structure also exposed big per-scenario swings in some models, showing single-case tests can be misleading. The checklist approach can benefit from chain-of-thought checks to enhance transparency and traceability.
Data Highlights
1Seven open-weight models (8–14B parameters) were evaluated using a 38-item boolean checklist derived by the framework.
2Top procedural stability: Gemma-2-9B-it scored dPCS = 0.97 with zero restarts; top behavioral fidelity: Llama-3.1-8B scored dBRF = 0.925 but incurred three restarts.
3Linguistic scores were uniformly high across models (dLFO = 0.90–1.00); one model (Phi-4-14B) showed up to 86 percentage points variance in procedural score between scenarios.
What This Means
Engineers building AI role-players and simulation platforms should use checklist-style evaluation to choose models that keep to their intended behaviors, not just those that sound good. Technical leaders and product owners in training, legal simulation, or any domain with strict rules can use these metrics to prefer smaller, instruction-tuned models when operational continuity and rule compliance matter. See how it aligns with established practices like the Human-in-the-Loop Pattern.
Not sure where to start?Get personalized recommendations
Key Figures
Fig 1: Figure 1 : Architecture of the RPA-Check Framework. The automated evaluation pipeline transforms qualitative behavioral requirements of Role-Playing agents into quantitative metrics through a four-stage process: Dimensions Definition establishes behavioral, procedural, and linguistic criteria; Augmentation expands these dimensions into granular Boolean checklists via an LLM generator; Semantic Filtering ensures checklist objectivity and agent isolation; and Evaluation based on LLM-as-a-Judge approach scores trial transcripts using a Chain-of-Thought (CoT) reasoning protocol. Final performance is quantified via the Quality Score (QS) and Retry Rate (R) metrics to assess role fidelity and operational stability.
Fig 2: (a)
Fig 3: Figure 3 : The player’s flow in LLM Court. After launching the game interface, starting a new game loads the RPAs LLM. Upon selecting a “new case,” the Procedural Case Generation module powered by GPT-5 generates and displays a case description. The player may either save the generated case to begin the game or discard it and request a new one.
Fig 4: Figure 4 : Schematic of the LLM Court simulation flow. The diagram illustrates the multi-phase execution logic managed by the FSM. The pools delineate the operational boundaries between the system-level procedural logic, the AI, i.e., the LLM-driven RPAs, and the interactive human player input (Defense). Process Logic is divided into three distinct chronological stages: the Introduction Phase (fixed sequential setup), the Interrogation Phase (dynamic dialogue), and the Verdict Phase (unscripted argumentation and final conclusion). In the dynamic dialogue, the Analyzer determines the turn-taking order at runtime via specific <NextSpeaker> tags, diverging from standard sequential flow to simulate realistic courtroom interactions.
Ready to evaluate your AI agents?
Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.
Learn MoreYes, But...
The experiments covered only five scenarios per model, so statistical generalization is limited and a larger, auto-generated case corpus is needed. Evaluation relies on a second model as an automated judge; while chain-of-thought checks increase transparency, that introduces potential evaluator bias. Results are tied to the Unity-based courtroom setup and local quantized models, so expect variation when porting to different domains, model families, or cloud-based runtimes. Some architectural patterns that support robust evaluation include the Orchestrator-Worker Pattern.
Methodology & More
RPA-Check is a four-stage, automated pipeline that converts high-level role and procedural requirements into a granular boolean checklist and then scores transcripts with a separate model acting as the judge. The four stages are: define evaluation dimensions (behavioral role fidelity, procedural convergence and stability, linguistic form), expand each dimension into many small yes/no indicators, filter for objectivity and non-redundancy, and finally run an evaluation phase where an evaluator model applies step-by-step checks to each transcript. The approach produces two main metrics: quality scores per dimension and a retry-rate that flags stochastic instability during multi-turn interactions. The framework was validated in LLM Court, a Unity-based forensic simulation where local quantized models (8–14B parameters) played judge, prosecutor, witnesses and interacted with a human defense player under a deterministic state machine. Across seven models and five legal scenarios the checklist (38 items) revealed that instruction tuning and evaluation granularity matter more than raw model size for constrained role-play. Smaller 8–9B models often achieved higher procedural stability and role adherence, while larger 12–14B models produced deeper arguments but suffered from role deviation and higher failure rates in the verdict phase. Linguistic quality was uniformly high, so picking models should prioritize procedural fidelity and low retry rates for deployment in rule-heavy settings. The framework is modular and transferable to other domains (medical training, compliance), though it needs a bigger, diverse scenario corpus and caution about evaluator-model bias before becoming a definitive benchmark. The framework also aligns with established concepts like the Sub-Agent Delegation Pattern and can be framed through the lens of LLM-as-Judge to formalize evaluation roles, while recognizing Insights from the Human-in-the-Loop Pattern help maintain governance during deployment.
Avoid common pitfallsLearn what failures to watch for
Credibility Assessment:
Multiple authors but low h-index values and no listed affiliations; arXiv preprint only—limited established reputation.