Back to Ecosystem Pulse
EvaluationExperimental
dokimos
by dokimos-dev
JUnit-friendly LLM evaluation framework for Java and Kotlin
Java
Updated Aug 15, 2026
Share:
What It Does
Provides an evaluation framework for LLM applications in Java and Kotlin. Implements repeatable metrics, JUnit-compatible extensions, and tooling for retrieval-augmented generation (RAG) and other LLM workflows. Focuses on developer-friendly integrations so JVM teams can run LLM benchmarks and metric-driven checks inside existing test suites. In particular, it supports RAG workflows and LLM workflows.
Why It Matters
As agent-based systems proliferate, having language-native evaluation tooling makes it easier to measure agent behavior and regressions during development. Until now most evaluation stacks target Python; this project fills a gap for JVM shops, enabling reproducible tests, RAG-specific metrics, and integration with CI. That helps teams track agent reliability and establish objective signals before agents are deployed or delegated to others. The emphasis on RAG-specific metrics and CI integration patterns helps teams maintain quality.
Target Use Cases
JVM teams who need to run reproducible LLM and RAG evaluations inside existing Java/Kotlin test suites and CI pipelines. This aligns with practitioners seeking reproducible benchmarks and supports RAG evaluations within established workflows.
Applications
- Run automated LLM and RAG evaluation as part of Java/Kotlin CI pipelines
- Measure regression and reliability metrics for agent-enabled services using JUnit
- Integrate LLM benchmarks into JVM codebases without switching to Python
Works With
langchain4jjunitspring-aikotlinjava
Topics
agent-evaluationagentic-aievaluationevaluation-frameworkevaluation-metricsjavajunitjunit-extensionkoogkotlin+10 more
Similar Tools
agent-playgroundautogen
Keywords
agent-to-agent evaluationllm-evaluationjava