Statistically grounded, provider-agnostic black-box testing for LLM agents: design-of-experiments campaigns, typed verdicts, a failure taxonomy, and ground truth derived from document graphs rather than another model’s opinion.
A second probabilistic model grades the first. Verdicts drift with the judge’s prompt, provider and mood; disagreements are unresolvable because there is no reference answer.
Questions are generated from document-graph topology, so every answer has a graph-verifiable reference. Verdicts are typed, reproducible, and auditable per claim.
Our benchmark study of judge reliability against graph-verified ground truth — When the Judge is Wrong — and the FinStructBench benchmark are published in Research, with reproduction on the roadmap for every headline number we publish.
Factor-group designs cover the input space systematically — repeatable campaigns instead of ad-hoc probing.
Structured pass/fail semantics per claim, with severity classification and hallucination detection.
Failures are classified, not just counted — so fixes target mechanisms, not symptoms.
Auto-generate questions from document structure; score against graph traversals.
Campaign results gate releases, with cost and latency tracked per run.
Synthetic benchmark instances for capital adequacy, credit portfolios, fair lending, model validation and stress testing.
Agent Assurance ships inside the knowlytix package. Install it and everything below works now.
from knowlytix.benchmark import Benchmark, score_answer # DOE testing campaigns: knowlytix.harness.testing # CLI: python -m knowlytix.benchmark --help (FinStructBench)
