Skip to content

Observability

Ecosystem reference

This page covers how agents emit and use telemetry through the agents SDK. For more dedicated and comprehensive reading, see LedgerGr.id (docs, GitHub): the telemetry platform for metrics, tracing, logging and transaction ledgers, with its own instrumentation SDKs and Kubernetes deployments.

Observability means being able to tell what an agent did, why it did it, and how well it went, from the signals it emits: logs, metrics and traces. In a multi-agent system you also need to follow one task as it moves between agents, tools, models and workflows, and to keep the record of a run together with the feedback it received.

Signal What it answers Where it comes from
Logs What happened, step by step Standard Python logging in every service and agent
Metrics How many, how fast, how often Agent application metrics on port 9090. See Metrics
Traces How one task moved through steps and agents job_output_trace_data on every result; workflow-engine trace events
Live task streams What the agent is doing right now: tokens, tool calls, tool results, progress context.stream(task). See Agents Streaming
Evaluations Whether the result met its expectation, and why not EvaluationReport. See Evaluation
Feedback Whether the result was good Verification results, human responses, Incorporator feedback

Logs

Component Control
Agent instances (SDK) LOG_LEVEL (default INFO); format %(asctime)s %(levelname)s %(name)s: %(message)s
Agent specification execution.logging_level (CRITICAL, ERROR, WARNING, INFO, DEBUG)
Platform services (subjects service, deployer, scouter, Delegate API, chat server) LOG_LEVEL

Useful log lines to watch for:

Message Meaning
on_preprocess returned None (task_id=…) The agent dropped a task
on_data failed (subtask_id=…) The agent raised. The result carries is_error and error_data.stage = "data"
[p2p …] BLOCKED session_id=… An admission policy blocked a mesh task
failed to route result (type=…) A result could not be delivered to its output_ptr destination. See Output Routing
Recovered N unacknowledged task(s) Tasks left in flight by a previous run of the instance were requeued
Max deliveries reached; moving task to dead-letter list A task failed REDIS_MAX_DELIVERIES times and was parked in jobs:inbox:dead
[admission] … policy failed (fail-closed) A policy error blocked a task (POLICY_FAIL_MODE=closed)
[AutoScaler] scale_up / scale_down The auto-scaler changed the instance count
Guardrail policy '…' loaded An SDK guardrail policy is active on the instance
Exhausted retries for exchange_id=… A result could not be reported to the exchange

Logs go to standard output, so any Kubernetes log stack can collect them.

Metrics

Agents expose application metrics on port 9090, and the platform includes a metrics collector (k8s/global-services/metrics-collector.yaml). See Metrics for defining and publishing metrics from agent code.

Traces

Per-Result Trace Data

Every AgentResult has a job_output_trace_data dictionary. It travels with the result to the exchange, the calling agent or a Redis queue. Use it to record how the result was produced:

return AgentResult(
    task_id=task.task_id,
    job_output={"answer": answer},
    job_output_trace_data={
        "model": "gpt-4o-mini",
        "tool_calls": [{"tool": "search", "args": {"q": q}, "ms": 412}],
        "delegations": [{"to": "fact-checker", "task_id": sub_id}],
        "latency_ms": elapsed_ms,
    },
)

Use the same task_id and session_id when you delegate, so a run can be reassembled across agents.

Workflow-Engine Trace Events

AgentWorkflowEngine emits workflow_start, plan_cache_hit / plan_cache_miss, converter_executed, replan_triggered, replan_limit_reached, workflow_completed and workflow_failed. With enable_tracing=True the built-in tracer prints them. To send them to your tracing backend, pass your own tracer:

class OTelTracer:
    def __init__(self, tracer):
        self.tracer = tracer                       # e.g. an OpenTelemetry tracer
    def log(self, event_type, payload):
        with self.tracer.start_as_current_span(event_type) as span:
            for k, v in payload.items():
                span.set_attribute(k, str(v))

engine = AgentWorkflowEngine(llm_client=client, modules=modules, tracer=OTelTracer(otel_tracer))

The final WorkflowState.to_dict() contains every node's input, output, status, attempts and timings.

LLM Latency

The LLM interface keeps per-backend latency statistics (count and last, minimum, maximum and average latency). See Agent × AI Model Interface.


Feedback Paths

To improve agents, keep the record of a run together with the feedback on it:

  1. Return trace data with every result, as shown above.
  2. Store results and traces, for example in the Persistent Store or agent memory, keyed by task_id.
  3. Attach feedback as it arrives: verification outcomes (Verification System), human answers (Human-in-the-Loop) and lesson feedback (Incorporator & Feedback).
  4. Attach evaluation reports (report.to_dict()) to the trace data of results that were checked against an expectation.
  5. Feed the combined record to the Learner so it becomes lessons. See Loop Improvement.

Connecting to Your Observability Stack

Signal How to export it
Logs Standard output of every pod. Collect them with your cluster's log stack (Fluent Bit, Loki, Elasticsearch, ...)
Metrics Scrape port 9090 of agent instances, or use the platform metrics collector
Workflow traces A custom tracer= object, as above
Task traces job_output_trace_data on every result. Results routed to Redis (output_ptr.type = "redis") can be consumed by an exporter
Live activity Subscribe to live task streams on Redis

Ecosystem reference

This page covers how agents emit and use telemetry through the agents SDK. For more dedicated and comprehensive reading, see LedgerGr.id (docs, GitHub): the telemetry platform for metrics, tracing, logging and transaction ledgers, with its own instrumentation SDKs and Kubernetes deployments.