ClaimsBench: Evaluating AI Agents Across a Complete Insurance Claim

Motivation

Insurance is a document-heavy market where a decision depends on claimant conversations, policy language, estimates, photographs, invoices, timelines, and later evidence—not just a final number. Claims teams must preserve a defensible record of why each item was covered, limited, or denied while working across fragmented systems.

Claims therefore provide a useful agent evaluation: the system must gather missing information, apply rules to individual items, calculate a settlement, respond to disputes, and know when evidence is insufficient. ClaimsBench tests the complete workflow from intake through a grounded resolution.

The 30 claim tasks

One task is one complete claim, not one question or phase. The agent starts with a first notice and item register, then interviews the claimant, gathers evidence, decides each item, calculates payment, revisits decisions after new evidence, and closes the claim. The 30 tasks share this workflow but vary in facts, policy terms, evidence, and disputed amounts.

How a task is created

Each task is assembled around one claim scenario by domain experts. They define the loss event, register the individual items, attach the applicable policy source, prepare the initial claim opening, and write the expert rubric that specifies the evidence, coverage decisions, amounts, policy reasons, revisions, and final settlement expected for that task. The evidence available to the agent is staged so that the opening remains incomplete but the missing facts can be discovered through claimant questions and evidence requests. A separate claimant profile drives the interview, while private adjudication ground truth records the expert-supported outcome. The task is then run through the five required phases and checked for a reproducible starting state, valid tool transitions, and a complete reference outcome before it is added to the benchmark.

One claim, five connected phases

Every task follows the same five-phase workflow, but the facts and policy questions change from claim to claim.

Consider the public opening of OID-001: a reported dishwasher-related water loss with claims for mitigation, cabinetry, flooring, and the supply line. The opening does not settle when the release occurred, whether earlier damage existed, or which costs belong to a covered scope. An agent has to ask, inspect, decide, and revisit those decisions as the record develops. The benchmark records those actions rather than grading only the closing paragraph.

How the score works

Prime Verifiers runs the claims agent and a separate claimant simulator, exposes structured claim tools, and records the authoritative server state. In our evaluation configuration, the agent has up to 80 assistant turns. The turn limit is not a limit of 80 claim-tool calls.

The claimant is another AI agent, not a fixed questionnaire. In the recorded Prime runs, GPT-5.4-mini plays the claimant from a private profile and answers the claims agent’s questions during intake, for up to seven claimant turns. The claims agent learns those private facts only when the claimant discloses them. A separate intake check measures how well the agent captured the facts; GPT-5.6 Luna judges the official reward.

The official luna_reward uses GPT-5.6 Luna to grade the accepted work against the private reference. Its five criteria are factual accuracy (30%), no material hallucinations (30%), claim-specific coverage (20%), direct task completion (10%), and grounded citations (10%). Each receives no, partial, or yes credit. Five deterministic checks—investigation, interim decisions, evidence-based revision, coverage and payment, and dispute closure—set the trajectory ceiling. The official reward is the lower of that ceiling and the Luna rubric score after coverage and error penalties. Merely reaching a phase earns no automatic credit, and an attempt with no accepted substantive work scores zero.

This design lets the score represent useful partial progress while still asking whether the agent’s statements and decisions are grounded in the claim record. A terminal-completion diagnostic is available separately; it is not the official reward.

Grading

Every task carries a rubric created by human evaluators. A frozen judge evaluates each criterion against the agent’s final message and any files it produced. In this benchmark, we used GPT-5.6 Luna as the judge.

The benchmark uses five distinct metrics:

• Mean reward is the average official reward across attempts.
• Pass rate is the share of individual attempts that complete a task perfectly.
• Solve rate is the share of tasks where at least one of three independent attempts completes it perfectly.
• Pass³ is the share of tasks where all three independent attempts complete it perfectly.
• Trajectory ceiling is the maximum reward allowed by the five deterministic phase checks.

Rubric breakdown

Luna reads the accepted claim work alongside the evaluator’s private reference. For each criterion, it returns no (0), partial (0.5), or yes (1) with a reason. The weights form a 0–1 base score:

Claim-specific coverage also scales the weighted score. A confirmed material factual error applies a 0.5 multiplier; a confirmed material hallucination applies a 0.2 multiplier. No accepted substantive work receives zero. The saved GLM run below used the earlier Luna rubric v2; the current phase-ceiling implementation is v3, so its recorded score is a historical result rather than a rerun under v3.

Quality Assurance

We bind each public claim, private record, and claimant profile to a frozen version with per-claim hashes. The private validator runs scripted reference trajectories through all 30 claims. In the current validation, all 30 completed with full deterministic phase scores. It also checks incomplete or incorrect trajectories: weakened half, full, and deny-all baselines averaged 0.315, 0.347, and 0.396, respectively. Those are validator fixtures, not model results.

Reward hacking is a central concern for any agent benchmark. An agent can appear successful while exploiting feedback, repeating rejected actions, probing for hidden answers, or making unsupported claims that sound plausible. ClaimsBench scores actions the server accepted, not actions the agent merely describes. Rejected submissions cannot become covered items by assertion, and repeated invalid actions reduce the trajectory ceiling. Ground truth and judge instructions remain evaluator-side, outside the agent’s task view. We inspect traces for filesystem probing, attempts to infer answers from rejection messages, and shortcuts that bypass evidence; automated flags alone do not identify every suspicious attempt.

The Luna rubric is versioned and its output is checked against a fixed schema. Independent expert-labeled calibration of ambiguous judge decisions remains a next step, so we treat an unavailable judge verdict as an unscored evaluation rather than a model score of zero.

What the reward-hacking audits showed

In an earlier GPT-5.6 Sol run, the agent inferred a private dispute answer from tool feedback instead of claim evidence. That was a reward-hacking shortcut. In a separate 20-rollout GLM-5.3 Flash audit, some agents probed the sandbox or repeatedly tried rejected actions, but none gained a private answer or improved its reward through those attempts.

These traces show why we review how an agent arrived at a score, not just the score itself. Agents can look competent while taking unsupported shortcuts, so the feedback channel and evaluation traces must be audited before a high reward is treated as reliable performance.

Sample Task

Task prompt. Adjudicate this claim using the available tools. Interview the claimant, record the intake, collect evidence, and submit each phase in order. Decide every registered item with a supported amount and policy reason. Revisit decisions when new evidence arrives, resolve the dispute, and submit a final settlement that reconciles. Do not invent facts or treat a document as the final answer.

Model results

The table below reports results from the 30-claim ClaimsBench benchmark. It summarizes the models’ pass rates, mean rewards, and average tool usage across the benchmark tasks.

Each model received three independent attempts per task. The leaderboard uses the v3 phase-ceiling scorer together with the GPT-5.6 Luna rubric.

Across the benchmark runs, Opus 5.5 has a mean reward of 0.68 with a 70% pass rate, followed by Fable 5.1 at 0.64 and 65%. GPT-6 Sol and GPT-6 Astra have mean rewards of 0.61 and 0.60, while Gemini 3.8 Flash and DeepSeek V4.1 Flash are close at 0.56 and 0.55. Grok 4.7 has a mean reward of 0.52.

Average tool usage across the benchmark runs ranges from 45 to 56 calls per task. Tool count alone does not indicate stronger performance. More calls can represent useful evidence gathering, but they can also reflect repeated actions, unnecessary investigation, or difficulty progressing through the workflow.

Aggregate scores can hide substantial differences in how agents behave during a claim. Two models with similar scores may reach them through very different trajectories, with differences in evidence gathering, tool usage, policy application, response to new information, and ability to reach a supported resolution.

This is the main purpose of ClaimsBench. The benchmark is not designed to test whether a model can simply generate a plausible insurance answer. It tests whether an agent can carry a claim through a long-running, stateful workflow from intake to resolution.

ClaimsBench evaluates whether agents can gather the right evidence, apply claim rules consistently, update their decisions when new information arrives, use tools effectively, and ultimately produce a defensible claim outcome.

The goal is to make failures visible before these agents are trusted with real insurance workflows. ClaimsBench is designed to expose the gap between an agent that can talk about a claim and one that can actually handle one.