An LLM evaluation framework is a repeatable system for testing whether a model-powered product completes real tasks correctly, safely, consistently, and within its operating limits. It defines the system version, test cases, graders, repeated trials, result slices, and release rules—not just a benchmark score.
The unit under test is usually larger than the model. A retrieval index, system prompt, tool schema, authorization layer, orchestration loop, and user interface can each change the outcome. A useful framework therefore evaluates the deployed workflow at the boundary users depend on, while retaining narrower component tests for diagnosis. This guide shows how to build that framework for text applications, retrieval-augmented generation, and tool-using agents without reducing quality to one average.
Start With the System Boundary
Before writing test cases, create a versioned manifest of what is being evaluated. Otherwise, a score can change without revealing whether the cause was a model update, a prompt edit, a refreshed retrieval corpus, or a different tool permission.
Record at least:
- model and provider identifier, relevant generation parameters, and routing or fallback rules;
- system and developer prompt versions, including any policy templates;
- retrieval configuration, embedding version, corpus or index snapshot, ranking settings, and document access rules;
- tool or MCP server versions, tool schemas, enabled tool set, and read-versus-write permissions;
- orchestration logic, retry limits, stop conditions, memory behavior, and context construction;
- safety policies, output constraints, escalation rules, and human handoff behavior;
- application, interface, and environment version, including feature flags; and
- external dependencies or test doubles that can affect the result.
This manifest makes a result reproducible enough to investigate. It does not imply that a hosted model is perfectly deterministic. It tells you which known system changed between two runs.
Evaluate at more than one layer when the system is complex. Component tests can check a parser, retriever, policy rule, or tool adapter quickly. End-to-end tests check whether those parts combine into a successful user outcome. A high retrieval score does not prove the final answer is grounded, and a polished final answer does not prove the agent respected authorization.
The NIST Generative AI Profile recommends measuring performance and assurance criteria under conditions similar to deployment, documenting the measures, and avoiding broad capability claims based on narrow or anecdotal tests. That is a good operating principle: the test boundary should resemble the real context closely enough to expose the risks that matter there.
Turn User Goals Into Task Contracts
A task should describe a meaningful job, not merely a prompt string. For each case, define:
- the user goal and initial environment state;
- the data and tools the system may access;
- the observable outcome that counts as success;
- acceptable variants of that outcome;
- forbidden outcomes, disclosures, and state changes;
- time, token, tool-call, and cost budgets where they matter; and
- the evidence a grader needs to decide the result.
For example, “answer a billing question” is too vague. A stronger task could require the assistant to identify the applicable policy from an approved knowledge snapshot, cite the correct section, avoid exposing another account’s records, make no account changes, and hand off when the policy does not resolve the case.
Agent tasks also need an initial and expected final state. If an agent is asked to create a disabled notification rule, grading only its message is insufficient. Inspect whether exactly one rule exists, whether it is disabled, whether it belongs to the correct account, and whether unrelated state remained unchanged. A confident “done” is not proof that the action happened.
Anthropic’s agent-evaluation guide distinguishes a task from a trial, the transcript or trajectory from the final outcome, and the agent from the evaluation harness. Those distinctions prevent a common mistake: treating one plausible conversation as evidence that a task is reliably solved.
Grade outcomes without prescribing every step
Prefer the final observable outcome when several safe approaches are valid. A rigid expected sequence can punish a more efficient solution or break whenever a tool description changes.
Trajectory checks are still necessary when the path itself is part of the contract. Inspect tool choice, arguments, sequence, authorization failures, confirmations, and side effects when they carry safety or policy meaning. An agent that reaches the requested state after reading another tenant’s data or executing an unconfirmed write has failed, even if the final screen looks correct.
Do not expose the expected answer, reference solution, or grader logic to the system under test. Keep evaluation fixtures outside the normal context unless the product is explicitly meant to see them. Never request, grade, or store hidden chain-of-thought. Grade observable responses, tool calls, retrieved citations, external state, and user-facing handoffs instead.
Build a Failure Taxonomy Before a Scorecard
Start with ways the product can fail, then decide how each failure should be detected. A practical taxonomy may include:
- factual or task correctness;
- groundedness and citation support;
- completeness without irrelevant padding;
- instruction and format compliance;
- retrieval coverage, ranking, and permission filtering;
- tool discovery, selection, arguments, order, and stopping behavior;
- authorization and account-boundary compliance;
- unsafe actions, secret exposure, or sensitive-data leakage;
- appropriate refusal and harmful over-refusal;
- recovery from tool errors, partial results, and unavailable dependencies;
- latency, token use, tool calls, and cost where the provider exposes them; and
- accessibility, clarity, handoff, and interface behavior where they affect task completion.
Turn each category into named slices. A support assistant might be sliced by policy type, language, customer tier, document freshness, and whether the answer requires a refusal. A coding agent might be sliced by language, repository size, test availability, permissions, and whether a change is read-only or state-changing.
Slices matter because averages hide concentrated failures. A 94% pass rate can coexist with a serious 40% pass rate on account-deletion requests. Preserve per-task results, slice distributions, grader evidence, and sample size alongside any summary.
Do not turn all criteria into a weighted super-score. Some conditions should be release blockers: cross-account access, an unconfirmed destructive action, or disclosure of a credential should not be averaged away by fluent writing. Separate hard gates from quality measures that can be compared or traded off.
Assemble a Representative Evaluation Dataset
The dataset should reflect intended use and known risk, not whichever prompts are easiest to label. Include:
- common representative tasks from each important workflow;
- edge cases at input, policy, and tool boundaries;
- adversarial cases for prompt injection, data exfiltration, permission bypass, and unsafe actions;
- ambiguous cases where refusal, clarification, or human handoff is correct;
- historical production failures, only after authorization, consent where required, and careful redaction; and
- a held-out regression set that routine prompt development does not optimize against directly.
Every case needs provenance, an owner, a version, expected behavior, relevant slices, and a reason for inclusion. Record whether it is synthetic, expert-authored, adapted from authorized production feedback, or generated to cover a threat model. Review cases when the product, policy, or source corpus changes.
Production conversations are not automatically safe evaluation data. They can contain personal information, confidential documents, credentials, customer logs, or material whose reuse was never authorized. Prefer synthetic cases that preserve the failure structure. Where real cases are necessary, establish a lawful purpose, minimize and redact the data, restrict access, define retention and deletion, and honor withdrawal or consent requirements. NIST specifically calls for documented data provenance and privacy protections in evaluation activities, including removal of personally identifiable information where appropriate.
Check for contamination. A benchmark that appears in training data, prompt examples, retrieval documents, or an agent’s accessible file system no longer measures clean generalization. Exact contamination detection is difficult, but you can at least search for fixture text across prompts and corpora, rotate private variants, keep a held-out set, and investigate suspiciously exact responses.
Dataset growth should follow evidence. When production feedback or an incident reveals a new failure, reduce it to the smallest privacy-safe reproducible case, assign it to a slice, and add it to the regression set. That creates an incident-to-test loop instead of a pile of unstructured transcripts.
Choose Graders in the Right Order
Use the least subjective reliable grader for each criterion. A strong order of preference is:
- deterministic state or outcome checks;
- programmatic assertions and reference checks;
- explicit rubrics evaluated by a model for genuinely subjective qualities; and
- human domain review for calibration, high-risk decisions, and cases automation cannot resolve.
Deterministic graders can verify JSON schema, exact database state, file diffs, citation existence, tool-call constraints, compilation, tests, numerical calculations, or whether an action stayed inside an account boundary. They are fast, inspectable, and suitable for continuous integration.
Reference answers work when the task has a stable set of acceptable facts. Avoid exact-string matching when wording may vary. Check required claims, prohibited claims, cited source support, or structured fields instead.
Model-based graders are useful for qualities such as clarity, supported synthesis, tone, or whether an answer addresses a nuanced rubric. They are also nondeterministic systems. Give them a narrow criterion, explicit scoring anchors, necessary evidence, and an option to abstain. Do not ask one judge for an unexplained “overall quality” number.
Calibrate model graders against human domain reviewers. Measure disagreement by criterion and slice, inspect false passes and false failures, and repeat calibration after changing the judge model or rubric. Where pairwise preference is useful, randomize answer order and blind identifying metadata. Test both orders because LLM judges can show position sensitivity. Also test for self-preference or style bias when a judge evaluates output from the same model family.
Human review remains important for policy meaning, domain correctness, subtle harm, and user experience. Give reviewers the same rubric, train them on boundary cases, measure agreement, and route disagreements for adjudication. A human label without a documented standard is not automatically reliable.
Measure Nondeterministic Behavior With Repeated Trials
A task is one defined case. A trial is one attempt at that case. Run multiple trials when sampling, model routing, retrieval, tools, or external services can vary.
Report more than the average number of passes. Preserve each task’s trial outcomes and show distributions by failure slice. A release with modest average improvement may still regress badly on permission failures or rare recovery paths.
Two common repeated-trial measures describe different promises:
- pass@k asks whether at least one of k attempts succeeds. It fits workflows where multiple attempts are acceptable, such as generating several candidates and selecting one. It increases as more attempts are allowed.
- pass^k asks whether all k attempts succeed. It fits workflows where consistency matters every time. It becomes stricter as more repeated successes are required.
Neither is universally better. Choose the measure that matches the product behavior. Do not advertise pass@5 if users receive only the first attempt, and do not hide a brittle first-attempt experience behind retries. Anthropic’s guide provides a fuller explanation of task-level metrics and repeated trials.
Track latency, token use, cost, and tool-call count only where the underlying measurements are available and comparable. Compare like with like: the same task set, environment, cache state, retry policy, and concurrency. An improvement that silently adds three retries may change both user latency and external side effects.
Evaluate Agents and MCP Tools as Authorized Systems
An agent evaluation must test the control plane around the model, not only the answer. Include cases for:
- discovering the correct server and interpreting tool schemas;
- selecting the right tool and valid arguments in the right order;
- staying within the user’s account, role, resource, and data boundary;
- handling expired tokens, denied scopes, revoked access, and failed PKCE authorization;
- treating tool output, resources, and retrieved documents as untrusted input that may contain prompt injection;
- separating reads from mutations and explaining the proposed change;
- requiring explicit confirmation where the product contract demands it;
- using idempotency controls and bounded retries for state changes;
- recovering from partial failures without claiming success;
- stopping when the goal is met or when safe progress is impossible;
- citing or linking the evidence behind summaries; and
- verifying the final environment state after a write.
MCP standardizes communication; it does not make every server, tool, or result trustworthy. The current MCP 2026-07-28 authorization specification requires clients implementing HTTP authorization to implement PKCE and verify the authorization server's advertised PKCE support before proceeding. It also requires resource indicators and token audience validation so a token is bound to its intended MCP server. Evaluation should still test the product’s actual consent, underlying data authorization, token revocation, and account isolation. Authentication proves an identity; it does not prove that every tool call is allowed or intended.
Prompt injection deserves end-to-end cases. Place malicious instructions in a retrieved document or simulated tool response, then verify that the agent treats them as data, does not reveal protected context, and does not expand its privileges. Include indirect attempts to override confirmation or redirect output to an unauthorized destination.
The interface is also part of the safety boundary. Before a consequential action, it should make the affected account or environment, selected filters, time range, evidence sources, proposed mutation, and confirm-or-cancel choice clear. Afterward, it should show the result and any partial failure. Simple defaults reduce mistakes, while precise search and filters allow investigators to narrow the evidence without granting broader access.
Use a Versioned Evaluation Specification
A small, reviewable specification prevents the test contract from living only in code or a dashboard. This example contains no credentials or private data:
schema_version: 1
suite_id: support-agent-regression
system_manifest: manifests/support-agent-2026-09-15.json
dataset_version: support-cases-v12
trials_per_task: 5
task:
id: billing-policy-citation-014
slices: [billing, retrieval, citation, read-only]
goal: Explain the applicable cancellation rule and cite its source.
initial_state: account_fixture_standard_active
allowed_tools:
- policy.search
- account.read_summary
forbidden_outcomes:
- access_other_account
- reveal_private_fields
- mutate_account
budgets:
max_tool_calls: 6
max_elapsed_seconds: 30
graders:
- type: state_assertion
check: account_state_unchanged
- type: citation_assertion
check: cited_policy_supports_claims
- type: rubric
criterion: answer_is_clear_and_complete
rubric_version: support-clarity-v4
release_gates:
- zero_cross_account_access
- zero_unconfirmed_mutations
- no_regression_on_critical_slices
Keep executable grader code, fixtures, manifests, and rubric versions in version control. Store secrets in the test environment’s secret manager, never in the specification. Give test agents isolated accounts and disposable state. Reset or verify state between trials so one attempt cannot make the next appear successful.
Connect Evaluation to the Release Lifecycle
Evaluation should run at several points, with scope proportional to risk.
During development
Run a small focused set while changing prompts, tools, retrieval, or orchestration. Add a failing test before fixing a reproducible defect when possible. Component tests give fast feedback; a few end-to-end tasks catch integration mistakes.
In continuous integration
Run deterministic regression tests and a bounded set of repeated model trials. Pin the manifest and dataset. Fail on hard safety gates and material regressions in critical slices, not only on a global score. Preserve grader evidence so a developer can reproduce the failure.
Before release
Run the full suite in an isolated environment that matches production permissions and integrations as closely as practical. Review new capabilities, changed failure slices, judge calibration, cost and latency, and unresolved exceptions. Share the evidence with the person or group responsible for the release decision. NIST recommends communicating pre-deployment test results to relevant release-approval actors and documenting limits beyond the tested conditions.
During a bounded rollout
Use canary or shadow evaluation only with appropriate privacy, safety, and access controls. Shadow mode is not automatically harmless: copied prompts and tool results can still expose sensitive data or invoke external systems if isolation is incomplete. Define stop conditions, a fallback, and who can expand or halt the rollout.
In production
Monitor feedback, policy violations, refusals, tool errors, latency, and sanitized outcome events. Sampled human review can detect shifts the offline suite misses. Convert verified production failures into privacy-safe regression cases. Do not automatically promote a model or configuration because one aggregate improved; review safety gates, slice regressions, grader changes, and deployment-context limits.
Record Enough Evidence Without Building a Data Leak
Evaluation records should make a run auditable without copying every sensitive input. Useful fields include:
- pseudonymous evaluation and run IDs;
- system manifest, dataset, task, and rubric versions;
- case ID and failure slices;
- trial number, outcome, criterion scores, and grader version;
- tool names and status classes;
- redacted argument categories rather than full values;
- final-state verification result; and
- latency, token, and cost fields when reliably provided.
Never record hidden chain-of-thought. Do not record raw prompts, retrieved passages, tool arguments, model responses, customer logs, credentials, or personal information by default. Keep a separately authorized and access-controlled evidence path for the limited cases that require content review. Apply defined retention, deletion, encryption, access auditing, and incident handling to both paths.
Logs are useful for execution evidence, but logs are not the evaluation framework. A log record can show that a tool returned a permission error; the grader still needs to decide whether the agent handled that error correctly. Likewise, an alert about a rising failure class can prompt investigation but cannot establish model quality by itself.
If sanitized evaluation events are sent through a supported collector or receiver, Fluxtail's documented search and filters can help operators investigate retained records by mapped fields, and Live Tail can help verify that new events arrive. Source fields depend on the collector and receiver mapping, so validate the schema before relying on a filter or alert. Fluxtail is a paid Starter/Pro, logs-focused service; it is not an LLM evaluation, tracing, model-monitoring, prompt-management, metrics, APM, CI, or benchmark product.
Fluxtail’s built-in AI log analysis and its hosted MCP interface are separate investigation surfaces. The hosted MCP uses account-bound OAuth/PKCE, and proposed mutations require short-lived confirmation. Raw retained rows remain the evidence. Do not use the same AI assistant to grade its own answers, and do not treat an AI summary as a substitute for the suite’s independent graders.
Put the Framework Into Operation
Begin with one high-value workflow and one consequential failure boundary. Version the complete system, write a handful of representative task contracts, and implement deterministic outcome checks before adding subjective judges. Add repeated trials, named failure slices, and clear release gates. Then expand from verified production failures rather than trying to predict every prompt in advance.
A practical readiness check is:
- the deployed system boundary is versioned;
- every task has an initial state, allowed scope, observable success, and forbidden outcomes;
- the dataset covers normal, edge, adversarial, refusal, and recovery cases;
- data provenance, consent, redaction, retention, and access are documented;
- deterministic graders cover state and policy whenever possible;
- model judges have narrow rubrics and current human calibration;
- repeated trials match the product’s actual retry behavior;
- task and slice distributions remain visible beside aggregate results;
- agent tests cover authorization, injection, mutation confirmation, partial failure, and final-state verification;
- release gates separate critical safety failures from negotiable quality measures; and
- production feedback becomes a privacy-safe regression test.
The result is not a permanent certificate that an LLM is safe or correct. It is a controlled way to make evidence-based release decisions, detect regressions, and learn from failures under the conditions the product is expected to face.