Skip to content

Scaling

An agent on AgentGr.id is not a single process. Each deployed agent has one executor that receives tasks and any number of instances that do the work. Scaling an agent means changing how many instances it has, where they run, and how tasks are spread across them. All of these decisions are made by policies attached to the agent, so you can change the strategy without changing the platform.

How a Task Flows Through a Deployed Agent

           POST /submitTask  ·  NATS mesh message
                         │
                ┌────────▼─────────┐
                │  Agent executor  │
                │                  │
                │ 1. pre-check     │  message_prechecker policy
                │ 2. rate limit    │  rate_limiter policy
                │ 3. backlog       │  message_backlog policy
                │ 4. load balance  │  load_balancer policy
                └───┬─────┬─────┬──┘
                    │     │     │   LPUSH to the instance's Redis (jobs:inbox)
                ┌───▼─┐ ┌─▼───┐ ┌▼────┐
                │ i1  │ │ i2  │ │ i3  │   agent instances (one pod each)
                └─────┘ └─────┘ └─────┘
                    ▲
                    │  auto-scaler policy adds or removes instances
  1. Steps 1–3 are admission: a task can be blocked (HTTP 429), queued in a backlog, or accepted. See Agentic Governance.
  2. The load balancer picks an instance for each accepted task and pushes it onto that instance's Redis sidecar. The instance's SDK worker pops tasks from the queue and runs the agent.
  3. The executor tracks live instances by watching pods labelled with the agent's subjectId in the agents namespace, refreshing every 20 seconds, so new and removed instances are picked up automatically.

Where the Scaling Logic Runs

All the logic that decides how an agent scales and how its tasks are admitted and spread runs inside that agent's own executor. Nothing in it is a shared, platform-wide service.

Logic Runs in Scope
Pre-check, rate limiting, backlog (message_prechecker, rate_limiter, message_backlog) The agent's executor, for every incoming task One agent
Load balancing (load_balancer) The agent's executor, for every accepted task One agent's instances
Auto-scaling (autoscaler) The agent's executor, every 30 seconds One agent's instances
Message transformation, pre- and post-processing (message_transformer, pre_processor, post_processor) Each agent instance, inside the agents SDK One instance
Creating and removing instances The agent deployer of the cluster, when the auto-scaler asks One cluster

One Executor per Agent

Every deployed agent gets its own executor: one Kubernetes Deployment, in the agents namespace, with three containers (the executor, the job scouter and a Redis). The executor manages that agent's instances and no others.

Policies are not shared between agents at runtime. A policy stored once in the policies system can be attached to many agents, but each agent's executor downloads its own copy when it starts, installs the policy's requirements.txt, and runs it in its own process. For example, if three agents all use the same load-balancing policy LB1, there are three executors, each running its own copy of LB1 over its own instances. A slow or failing policy in one agent cannot affect another.

What Compute It Uses

The scaling logic uses the compute of the agent's executor pod, not that of the agent instances or of the platform services. The deployer does not set CPU or memory limits on the executor pod, so it uses what the node gives it; keep policies light, because the admission and load-balancing policies run for every task.

Each agent instance runs in its own pod with the resources set in the agent's runtime.resources. Adding instances adds instance pods; the executor stays one pod per agent.

How the Scaling Logic Itself Scales

Because there is one executor per agent, the scaling logic grows with the number of agents: deploying more agents adds more executors, each handling only its own traffic. To handle more load for one agent, add instances, manually or with an autoscaler policy. The executor is a single pod per agent and is not replicated.

For how policies are written, attached and run, see Agentic Governance.


Deploying at Scale

  • More instances: list several entries in allocation.instances when deploying. Each instance runs in its own pod with its own Redis sidecar.
  • Placement: use node_selector per instance, and runtime.resources in the specification, to put instances on the right node pools, for example GPU nodes.
  • Many clusters: register one deployer per cluster and deploy each agent to the cluster that suits it. See Multiple Agent Clusters.

Load Balancing

The load balancer runs the agent's load_balancer policy for every accepted task.

Policy input {"session_id": ..., "message_data": {...}}
Injected into the policy settings get_instances() returns the current instance IDs; load_balancer is the balancer object
Expected output {"ok": true, "instance": "<instance_id>", "metadata": {...}}, a (instance_id, metadata) tuple, or just the instance ID as a string
Default (no policy) A random live instance

A custom policy can implement round-robin, least-loaded, session affinity (send every task of a session_id to the same instance) or content-based routing.

Variable (executor) Default Description
LB_REDIS_QUEUE_KEY jobs:inbox Queue key on the instance Redis

Auto-Scaling

The executor includes an auto-scaler that runs every 30 seconds and asks the agent's autoscaler policy what to do.

Policy input {"instance_count": n, "instance_ids": [...], "queue_depth": n, "subject_id": "..."}
Injected into the policy settings get_instances(), get_queue_depth()
Expected output "scale_up", "scale_down" or None, or {"action": "scale_up"}
Default (no policy) No action

scale_up asks the deployer to create one new instance (POST /instances) with the agent's meshes and Delegate API URL. scale_down removes the most recently listed instance (DELETE /instances). A typical policy scales up when queue_depth / instance_count passes a threshold and scales down when the queue has been empty for a while, within minimum and maximum instance counts kept in its parameters.

The deployer gives every executor it creates the variables the auto-scaler needs:

Variable (executor) Set to Purpose
DEPLOYER_API_URL The deployer's in-cluster URL, http://agents-deployer.agent-deployer.svc.cluster.local:9000. Override it with DEPLOYER_API_URL on the deployer Where scale_up / scale_down requests go
AGENT_DELEGATE_URL allocation.delegate_api_url from the deploy request Passed to the instances the auto-scaler creates
MESH_LIST allocation.meshes from the deploy request New instances join the same NATS meshes

Scaling is driven entirely by the policy. Without an autoscaler policy the instance count stays as deployed.

Example Auto-Scaling Policy

# code/function.py, attached with policy_type "autoscaler"
import time

class AIOSv1PolicyRule:
    def __init__(self, rule_id, settings, parameters):
        self.settings = settings
        self.idle_since = None

    def eval(self, parameters, input_data, context):
        n = input_data["instance_count"]
        depth = input_data["queue_depth"]
        lo, hi = parameters.get("min_instances", 1), parameters.get("max_instances", 10)
        per_instance = parameters.get("target_queue_per_instance", 20)

        if n < hi and depth > per_instance * max(n, 1):
            self.idle_since = None
            return {"action": "scale_up"}
        if depth == 0:
            self.idle_since = self.idle_since or time.time()
            if n > lo and time.time() - self.idle_since > parameters.get("idle_seconds", 300):
                self.idle_since = None
                return {"action": "scale_down"}
        else:
            self.idle_since = None
        return {"action": None}

Rate Limiting and Backlogs

Under heavy load, admission policies protect the agent:

  • The rate limiter (rate_limiter policy) receives a rate_counter with increment, get, reset and per-session metadata, and decides whether a task is allowed. Blocked tasks get HTTP 429 with stage: "rate_limit".
  • The backlog (message_backlog policy) receives a backlog_store (push, pop_one, pop_n, peek_n, length, clear) and can hold tasks back, or release several queued tasks at once.
Variable (executor) Default Description
RATE_LIMITER_BACKEND memory memory or redis
BACKLOG_BACKEND memory memory or redis
REDIS_URL — Redis used when a backend is redis

With the memory backends, counters and backlogs are lost when the executor restarts. Use redis when that matters.


Routing

Tasks reach an agent in three ways, and each can be scaled independently:

Route Path Notes
HTTP POST <executor>/submitTask Used by the Delegate API and context.direct
P2P mesh NATS message to the agent's executor Same admission and load-balancing pipeline as HTTP
Task exchange The exchange (and Orcade, for bidding) delivers through the Delegate API to /submitTask, and waits up to 300 seconds for the reply Exchange traffic goes through the same admission policies; see Task Exchange & Bidding

To choose which agent gets a task, see Task Delegation and Dynamic Workflows.


Fault Tolerance

Failure Behavior
Instance pod crashes Kubernetes restarts it (restartPolicy: Always). The executor's instance list is refreshed from pod labels, so tasks stop going to missing instances
Task in flight on a crashing instance Redelivered. The SDK moves each task to a per-instance processing list (jobs:inbox:processing:<instance>) and removes it only after the task is handled. On restart, unacknowledged tasks go back on the queue (at-least-once delivery)
Task keeps failing Retried up to REDIS_MAX_DELIVERIES times (default 3), then moved to the dead-letter list jobs:inbox:dead for inspection
Executor restarts Rate counters and backlogs persist when the redis backends are used
Exchange temporarily unreachable Results are retried 3 times with increasing back-off
Policy raises an error Controlled by POLICY_FAIL_MODE: open (default) allows the task; closed blocks it with reason.error = "policy_error". See Agentic Governance

Because delivery is at-least-once, make on_data idempotent per task_id, for example by checking a checkpoint before repeating side effects.

Variable (instance) Default Description
REDIS_MAX_DELIVERIES 3 Attempts before a task is dead-lettered
REDIS_PROCESSING_KEY <queue>:processing:<INSTANCE_ID> Processing list of this instance
REDIS_DEAD_LETTER_KEY <queue>:dead Dead-letter list

For long-running work that must survive restarts, see Persistence, Checkpointing & Recovery.