Skip to content

Testing Agents

Test an agent in three layers, from fastest to most realistic:

  1. Unit-test the agent class: call its hooks directly, with no platform services.
  2. Run the agent locally: start the real SDK runtime on your machine and send it tasks over WebSocket.
  3. Smoke-test the deployment: check the deployed executor with health and dry-run calls, then send a real task.

To judge the quality of an agent's output, rather than whether it runs, see Evaluation.


1. Unit-Testing the Agent Class

An agent class is a plain Python class (see Writing the Agent Code). Because on_preprocess and on_data take and return plain dataclasses, you can test them without Redis, NATS or a subjects service. Pass a stub in place of context and replace the parts your agent uses:

# test_uppercase_agent.py
from types import SimpleNamespace
from core.types import AgentTask

from agent import UppercaseAgent   # your agent module; do not call main() at import time


def make_agent(delegate_reply=None):
    context = SimpleNamespace(
        subject_id="uppercase-test",
        delegator=SimpleNamespace(submit_and_wait=lambda **kw: delegate_reply),
        direct=SimpleNamespace(submit=lambda **kw: None),
        p2p_manager=SimpleNamespace(send_and_wait_sync=lambda *a, **kw: delegate_reply),
    )
    return UppercaseAgent(subject=None, context=context)


def test_drops_task_without_text():
    agent = make_agent()
    assert agent.on_preprocess(AgentTask(task_id="t1", job_data={})) is None


def test_uppercases_text():
    agent = make_agent()
    [task] = agent.on_preprocess(AgentTask(task_id="t2", job_data={"text": "hi"}))
    result = agent.on_data(task)
    assert not result.is_error
    assert result.job_output == {"text": "HI"}

Tip

Keep main(AgentClass) behind an if __name__ == "__main__": guard, so that importing the module in tests does not start the runtime.


2. Running the Agent Locally

main() runs the same runtime that runs in the cluster. It needs:

Requirement Why
A registered subject The SDK loads the specification of SUBJECT_ID from SUBJECT_DB_URL at start-up. See Agent Registration
AGENT_DELEGATE_URL Required. The SDK refuses to start without it. Any URL works if the agent does not delegate
Redis on REDIS_URL (default redis://localhost:6379/0) The task queue worker runs in the main thread
MESH_LIST (optional) NATS meshes to join. Without it, P2P is disabled and a warning is logged
JOB_EXCHANGE_API_URL (optional) Needed only if results are reported to a task exchange
docker run -d --name agent-redis -p 6379:6379 redis:7-alpine

export SUBJECT_ID="uppercase-001"
export SUBJECT_DB_URL="http://<subjects-service-host>:<port>"
export AGENT_DELEGATE_URL="http://<delegate-host>:<port>"
export LOG_LEVEL=DEBUG

python3 agent.py

Sending a Task over WebSocket

The runtime serves WebSocket on port 8765 (WS_PORT). Each message is processed synchronously and the results are sent back on the same connection:

import json
from websockets.sync.client import connect

with connect("ws://localhost:8765") as ws:
    ws.send(json.dumps({"task": {"task_id": "t1", "job_data": {"text": "hello"}}}))
    print(json.loads(ws.recv()))
# {"results": [{"task_id": "t1", "job_output": {"text": "HELLO"}, ..., "is_error": false,
#               "origin_task_id": "t1"}]}

To send several tasks at once, use {"tasks": [ {...}, {...} ]}. If WS_BEARER_TOKEN is set, send the header Authorization: Bearer <token>.

Sending a Task through the Queue

This is the path the agent executor uses in production. Push a task onto the Redis list and watch the agent's logs:

redis-cli LPUSH jobs:inbox '{"task_id":"t2","job_data":{"text":"queued"},"output_ptr":{}}'

Results of queued tasks are routed by output_ptr (see Output Routing). With an empty output_ptr nothing is sent back; use "output_ptr": {"type": "redis", "host": "localhost", "port": 6379, "output_queue": "results"} and redis-cli BRPOP results 0 to see the result.


3. Smoke-Testing a Deployed Agent

Find the executor URL in the agent's runtime subject (runtime_info.executor, see Agent Deployment), then:

EXECUTOR=http://<executor-host>:<node-port>

# 1. Is the executor up?
curl -s $EXECUTOR/healthz          # {"ok": true}

# 2. Would a task be accepted? Runs the admission policies without executing the task
curl -s -X POST $EXECUTOR/dryRun -H "Content-Type: application/json" \
  -d '{"session_id": "smoke-1", "message_data": {"task_id": "t1", "job_data": {"text": "hi"}}}'

# 3. Submit a real task
curl -s -X POST $EXECUTOR/submitTask -H "Content-Type: application/json" \
  -d '{"event_type": "task", "task_id": "t1", "task": {"text": "hi"}, "origin": {}}'

/dryRun returns {"success": true, "message": "ok"} if the task would be accepted, or HTTP 429 with the blocking stage (pre_check or rate_limit) and reason. /submitTask returns HTTP 202 once the task is queued. To get the result back synchronously, call the agent through the Delegate API (context.delegator.submit_and_wait(...) or POST <delegate>/api/submit-and-wait).

Validating Workflows

For DSL workflows, use check_execute in the DSL Manager as a dry run before executing.