Skip to content

Evaluation

Ecosystem reference

This page covers evaluation from inside agents, with the agents_eval library. For scoring agents and models against labelled datasets as queued, containerised jobs, see Evaluate2.ai (GitHub) and Offline Benchmarks with Evaluate2.ai below.

Evaluation answers one question: is this output good enough? On AgentGr.id you describe what a good result looks like as an expectation, check outputs against it with an evaluator, and let agents self-refine: try, get evaluated, and try again with the feedback until the expectation is met or a budget runs out.

Everything on this page is in the agents_eval package (libs/agents-eval):

pip install ./libs/agents-eval

Expectations

An expectation is a named list of criteria with a pass score:

from agents_eval import Expectation, Criterion

summary_expectation = Expectation(
    name="contract-summary",
    description="A short, neutral summary that names the governing law and lists risks.",
    criteria=[
        Criterion.required_fields(["summary", "risks"], required=True),   # must pass
        Criterion.max_length("summary", 1200),
        Criterion.min_items("risks", 1),
        Criterion.regex("summary", r"(?i)governing law"),
        Criterion.custom("no placeholders", lambda out, ctx: "TODO" not in out["summary"]),
        Criterion.llm_judge("Is the summary accurate to the source, neutral and complete?",
                            threshold=0.7, weight=2.0),
    ],
    pass_score=0.85,
)

Criteria

Constructor Passes when
required_fields(fields) Every field exists and is not empty. Dotted paths are allowed ("result.items")
type(path, type_name) The value is a string, number, integer, boolean, object, array or null
regex(path, pattern) The pattern is found in the value
contains(path, value) value in the field (substring or list member)
max_length(path, n) / min_length(path, n) Length limits
min_items(path, n) A list has at least n items
range(path, minimum=None, maximum=None) A number is within bounds
equals(path, value) Exact match
custom(name, fn) fn(output, context) returns True, a score ≥ threshold (default 0.5), or (bool_or_score, message)
llm_judge(rubric, threshold=0.7) A model scores the output from 0 to 1 against the rubric, and the score is ≥ threshold
benchmark(dataset_id, category_id, metric="pass_rate", minimum=None, maximum=None) The output, a set of predictions, is evaluated on Evaluate2.ai and the metric is within bounds. See Benchmark Criteria

Every constructor also takes name=, weight= (default 1.0) and required= (default False).

Scoring

  • Each criterion scores 1 or 0, except llm_judge, benchmark and score-returning custom criteria, which score between 0 and 1.
  • The evaluation score is the weighted average of the criterion scores.
  • The output passes when the score is ≥ pass_score and every required=True criterion passed.

Expectations made of data-only criteria (everything except custom) can be stored and shared as JSON with expectation.to_dict() and Expectation.from_dict(...). For example, a requester can send the expectation along with the task.


Evaluating Output

from agents_eval import Evaluator

evaluator = Evaluator(llm=Evaluator.llm_from_client(client, name="openai"))   # model for llm_judge

report = evaluator.evaluate(output, summary_expectation, context={"source": contract_text})

report.passed        # True / False
report.score         # 0.0 – 1.0
report.failures      # failed CriterionResult objects (name, score, message)
report.feedback      # text listing what to fix, ready to send back to the agent
report.to_dict()     # for logs, traces or job_output_trace_data

context is shown to the judge model with the output, so include whatever it needs to check accuracy, for example the source document or the original question. The judge runs at temperature 0 and must answer with {"score": ..., "reason": ...}. The reason becomes the failure message.


Self-Evaluation Loops

self_refine lets an agent work towards an expectation. It calls your produce(feedback, attempt) function, evaluates the output, and calls it again with the evaluator's feedback until the output passes, or until a limit is reached:

from agents_eval import self_refine

def on_data(self, task):
    source = task.job_data["contract"]

    def produce(feedback, attempt):
        prompt = f"Summarise this contract as JSON with 'summary' and 'risks'.\n\n{source}"
        if feedback:
            prompt += f"\n\n{feedback}\nFix these problems."
        return self.ask_model_for_json(prompt)

    result = self_refine(
        produce,
        summary_expectation,
        evaluator=self.evaluator,
        max_attempts=4,          # compute limit: number of tries
        time_budget_s=90,        # wall-clock limit
        patience=2,              # stop after 2 attempts in a row without a better score
        context={"source": source},
    )
    return AgentResult(
        task_id=task.task_id,
        job_output=result.output,                       # best output seen
        is_error=not result.passed,
        job_output_trace_data={"evaluation": result.report.to_dict(),
                               "attempts": result.history,
                               "stopped_because": result.stopped_because},
    )
RefineResult field Meaning
output The passing output, or the best-scoring one if none passed
report Its EvaluationReport
passed Whether the expectation was met
attempts How many outputs were produced
stopped_because passed, max_attempts, time_budget or no_improvement
history Per attempt: score, passed, feedback, elapsed time

Use on_attempt=lambda n, out, report: ... to observe each attempt, for example to send progress events on a live task stream.


Offline Benchmarks with Evaluate2.ai

Everything above runs online: inside the agent, on one output, in milliseconds. Evaluate2.ai adds the offline half. It scores a whole dataset of outputs as a queued job in its own container, against labelled answers, and returns aggregate metrics.

Online (agents_eval) Offline (Evaluate2.ai)
When Every task, while the agent works Before a release, nightly, or when an agent tests a change
Question Is this one output good enough? How good is this agent, prompt or model across a dataset?
Ground truth An expectation A labelled dataset, plus optionally the same expectation
Speed Milliseconds to seconds, in-process A queued container job, seconds to minutes

The two share one definition of "good". The agent_task_eval evaluator for Evaluate2.ai runs the same Expectation objects with agents_eval inside its container. The same expectation therefore gates single outputs at runtime and whole datasets offline.

 dataset (tasks + expected answers)        agent                       Evaluate2.ai
 ───────────────────────────────           ─────                       ────────────
 upload once, with the expectation  ──►  runs every task  ──►  predictions.jsonl  ──►  agent_task_eval job
                                                                                         │
 Criterion.benchmark / CI gate  ◄──────────  pass_rate, f1, tool_calls, failures  ◄──────┘

Three pieces make this work:

Piece Where What it does
Evaluate2Client agents_eval.evaluate2 Uploads predictions, runs evaluations and waits for results
Criterion.benchmark agents_eval Turns an Evaluate2.ai metric into a criterion, for release gates and refinement loops
agent_task_eval Evaluate2.ai repository, agent-metrics/agent_task_eval/ The evaluator image that scores agent tasks

One-Time Setup

1. Build the evaluator image and push it to a registry that the Evaluate2.ai scheduler can pull from. The build bundles the Evaluate2.ai eval framework, agents_eval and the evaluator:

cd Evaluate2.ai/agent-metrics/agent_task_eval
./build.sh /path/to/agents-v1/services/agents/libs/agents-eval registry.example.com/agent_task_eval:1.0
docker push registry.example.com/agent_task_eval:1.0

2. Create a client and register the image as a category:

import os
from agents_eval import Evaluate2Client

e2 = Evaluate2Client(
    users_url="http://evaluate2-users:5000",
    jobs_url="http://evaluate2-jobs:5001",
    username="admin",
    password=os.environ["EVALUATE2_PASSWORD"],
    admin=True,                       # see Current Limitations
)

e2.create_category("agent-tasks", image_name="registry.example.com/agent_task_eval:1.0",
                   pkg_name="agent_task_eval", description="AgentGr.id agent task evaluation")

Evaluate2Client.from_env() reads the same settings from EVALUATE2_USERS_URL, EVALUATE2_JOBS_URL, EVALUATE2_USERNAME, EVALUATE2_PASSWORD and EVALUATE2_ADMIN.

Creating a Dataset

A dataset is a JSONL file (or JSON array) of tasks. Every field except id is optional:

{"id": "t1", "input": {"question": "Which state's law governs the RetailCo agreement?"},
 "expected": {"answer": "Delaware"},
 "expected_tool_calls": [{"name": "search_contracts", "arguments": {"party": "RetailCo"}}],
 "tags": ["governing-law"]}
Task field Meaning
id Unique task ID; predictions are matched to tasks by it
input What the agent is given. It is also passed to llm_judge criteria as context
expected The expected answer. A dict is compared field by field; anything else is compared as a whole
expected_tool_calls Tool calls the agent should make: names, or {"name", "arguments"}. Only the listed arguments are checked
context Extra context for llm_judge criteria, such as a source document
expectation An expectation for this task only, overriding the dataset's
tags Labels; the result reports the pass rate per tag

Upload the file, and register the dataset with its parameters: the expectation and the scoring options.

from agents_eval import Expectation, Criterion

answer_expectation = Expectation(
    name="contract-answer",
    criteria=[
        Criterion.required_fields(["answer", "citations"], required=True),
        Criterion.min_items("citations", 1),
        Criterion.range("confidence", 0, 1),
    ],
)

file_id = e2.upload_dataset_file("agent-tasks", "contract_qa_v1.jsonl")
e2.create_dataset("contract-qa-v1", "agent-tasks", file_id, parameters={
    "expectation": answer_expectation.to_dict(),
    "match": {"fields": ["answer"]},
    "pass_requires": ["expectation", "exact_match", "tool_calls"],
})
Parameter Default Meaning
expectation none An Expectation.to_dict(), applied to every output
match.fields the keys of expected Dotted paths compared with expected
match.normalize true Compare strings case- and punctuation-insensitively, ignoring articles
pass_requires see below Which checks a task must pass: any of expectation, exact_match, tool_calls
judge none Model for llm_judge criteria: {"base_url", "model", "api_key_env"}, for any OpenAI-compatible endpoint
max_failures 20 How many failing tasks to list in the result
include_tasks false Also return a per-task task_results list

By default, a task passes when its output meets the expectation, or, without an expectation, matches expected exactly. When the task lists expected_tool_calls, the agent must also have made each of them with the expected arguments. Custom (Criterion.custom) criteria are code and cannot be sent to Evaluate2.ai; use data criteria or llm_judge.

Check a dataset before uploading it

python run_local.py tasks.jsonl predictions.jsonl params.json, in agent-metrics/agent_task_eval/, runs the evaluator on local files and prints the result, without Evaluate2.ai. Install agents_eval first.

Collecting Predictions

Run every task through the agent and write one record per task:

{"id": "t1", "output": {"answer": "Delaware", "citations": ["retailco-msa.pdf#p12"], "confidence": 0.93},
 "tool_calls": [{"name": "search_contracts", "arguments": {"party": "RetailCo"}}], "latency_s": 2.1}
Prediction field Meaning
id The task ID
output The agent's result
tool_calls Calls the agent made: {"name", "arguments"} or OpenAI-style {"function": {"name", "arguments"}}
latency_s Time taken; reported as mean, p50, p95 and max
error Set when the agent failed; the task counts as failed

For example, sending the tasks to a deployed agent:

import json, time

def collect_predictions(tasks, call_agent):
    predictions = []
    for task in tasks:
        start = time.time()
        try:
            result = call_agent(task["input"])            # e.g. a Delegate API call
            predictions.append({"id": task["id"], "output": result["output"],
                                "tool_calls": result.get("tool_calls", []),
                                "latency_s": round(time.time() - start, 3)})
        except Exception as e:
            predictions.append({"id": task["id"], "error": str(e),
                                "latency_s": round(time.time() - start, 3)})
    return predictions

Running an Evaluation

result = e2.run("contract-qa-v1", "agent-tasks", predictions, timeout_s=600)

result.succeeded                  # True when the job finished with status "success"
result.output["pass_rate"]        # 0.0 – 1.0
result.output["failures"]         # the lowest-scoring failed tasks, with feedback
result.metrics                    # Evaluate2.ai phase timings (admin clients)

predictions can be a list of records (sent as JSONL), a dict, bytes or a file path. To queue the job and continue, for example in a workflow step, use eval_id = e2.submit(...) and later e2.wait(eval_id, "agent-tasks"). If the job fails, run and wait raise Evaluate2Error with Evaluate2.ai's failure message; pass raise_on_failure=False to get the result with status and log instead.

The result of an agent_task_eval job:

Field Meaning
tasks, passed, pass_rate How many tasks passed
mean_score Average task score: the expectation score, the answer F1, or their average when both apply
criteria Pass rate of each expectation criterion, by name
exact_match, f1 Answer accuracy against expected (token-overlap F1 for strings)
tool_calls precision, recall, f1, arguments_accuracy and sequence_accuracy
latency_s mean, p50, p95, max
by_tag Tasks and pass rate per tag
missing_predictions, agent_errors Tasks with no prediction, and tasks where the agent reported an error
failures Failed tasks with id, score and feedback, lowest score first

Benchmark Criteria: Release Gates and Refinement

Criterion.benchmark turns an Evaluate2.ai metric into a criterion. The output being evaluated is the set of predictions; the criterion uploads it, runs the job, and checks the metric. Give the client to the Evaluator, in the same way as the model for llm_judge:

release_gate = Expectation(name="release-gate", criteria=[
    Criterion.benchmark("contract-qa-v1", "agent-tasks", metric="pass_rate", minimum=0.9, required=True),
    Criterion.benchmark("contract-qa-v1", "agent-tasks", metric="latency_s.p95", maximum=5.0),
])

report = Evaluator(evaluate2=e2).evaluate(predictions, release_gate)
if not report.passed:
    raise SystemExit(report.feedback)          # e.g. fail the CI job
  • metric is a dotted path into the result, such as pass_rate, tool_calls.f1 or by_tag.dates.pass_rate.
  • A failing criterion scores partial credit (how close the value came to the bound), and its message names the job and the worst failing tasks, so report.feedback says what to fix.
  • report.results[i].details holds the job's eval_id, status, value and full output.
  • Each benchmark criterion runs its own job; criteria on the same dataset do not share one.

Because it is an ordinary criterion, a benchmark works with self_refine. This suits slower, offline improvement loops, for example an agent tuning its own prompt against a dataset:

def produce(feedback, attempt):
    prompt = improve_prompt(current_prompt, feedback)    # feedback lists the failing tasks
    return collect_predictions(tasks, lambda inp: run_with_prompt(prompt, inp))

result = self_refine(produce, release_gate, evaluator=Evaluator(evaluate2=e2),
                     max_attempts=5, time_budget_s=3600)

Keep benchmarks out of per-task loops: every evaluation is a container job.

Current Limitations

These come from the current Evaluate2.ai code. The client works around them where it can.

Evaluate2.ai fixes needed before use

  • Evaluation creation: createEvaluation looks the prediction file up by uniqueid, but the field is stored as uniqueId (frontend-apis/jobs-service/src/controls.js), so every evaluation is rejected with "Invalid evaluation file". Change the query key to uniqueId.
  • Service startup: the Users and Jobs Service index.js files use PORT, which is defined only in src/server.js. Define it in index.js from USERS_SVC_PORT / JOBS_SVC_PORT.
  • Node.js version: the service images are built FROM node:latest. On Node.js 26 the jsonwebtoken dependency fails to load. Pin node:18 (tested) or upgrade the dependency.
Limitation Effect What to do
Only the root account can create datasets and evaluations (non-root access checks fail, see the Evaluate2.ai API reference) Regular users are refused Use admin=True with a dedicated deployment's root credentials, kept in a secret
Results are polled; there is no completion callback wait polls getStatus every poll_s seconds Use submit and wait to keep the caller free
Tokens expire (5 minutes by default) and are per service The client logs in to each service and logs in again when a token is rejected
The scheduler passes only its own settings into evaluation containers llm_judge inside agent_task_eval cannot receive an API key from the environment Use a judge endpoint that needs no key inside your network, or bake the key into the image as AGENT_EVAL_JUDGE_API_KEY. Avoid judge.api_key in dataset parameters, which are stored in plain text
The framework's base image is Python 3.6 agent_task_eval cannot be uploaded as a dynamic zip package Use the container image built by build.sh (Python 3.8)

Where Evaluation Fits

Use How
Testing an agent before release Run a dataset through the agent and score it on Evaluate2.ai, with a benchmark release gate. Track the pass rate per version (see Testing Agents)
Self-checking inside the agent self_refine, as above
Checking another agent's work Evaluate delegated results before using them, and escalate to the Verification System or a human when they fail
Admission / output guardrails Call an evaluator from a post_processor policy (see Governance)
Learning from outcomes Store reports with the run's trace and feed them to the Learner (see Loop Improvement)
Comparing models or prompts Run the same dataset with each variant and compare their Evaluate2.ai results

Ecosystem reference

This page covers evaluation from inside agents, with the agents_eval library. For scoring agents and models against labelled datasets as queued, containerised jobs, see Evaluate2.ai (GitHub) and Offline Benchmarks with Evaluate2.ai below.