Scaling¶
An agent on AgentGr.id is not a single process. Each deployed agent has one executor that receives tasks and any number of instances that do the work. Scaling an agent means changing how many instances it has, where they run, and how tasks are spread across them. All of these decisions are made by policies attached to the agent, so you can change the strategy without changing the platform.
How a Task Flows Through a Deployed Agent¶
POST /submitTask · NATS mesh message
│
┌────────▼─────────┐
│ Agent executor │
│ │
│ 1. pre-check │ message_prechecker policy
│ 2. rate limit │ rate_limiter policy
│ 3. backlog │ message_backlog policy
│ 4. load balance │ load_balancer policy
└───┬─────┬─────┬──┘
│ │ │ LPUSH to the instance's Redis (jobs:inbox)
┌───▼─┐ ┌─▼───┐ ┌▼────┐
│ i1 │ │ i2 │ │ i3 │ agent instances (one pod each)
└─────┘ └─────┘ └─────┘
▲
│ auto-scaler policy adds or removes instances
- Steps 1–3 are admission: a task can be blocked (HTTP
429), queued in a backlog, or accepted. See Agentic Governance. - The load balancer picks an instance for each accepted task and pushes it onto that instance's Redis sidecar. The instance's SDK worker pops tasks from the queue and runs the agent.
- The executor tracks live instances by watching pods labelled with the agent's
subjectIdin theagentsnamespace, refreshing every 20 seconds, so new and removed instances are picked up automatically.
Where the Scaling Logic Runs¶
All the logic that decides how an agent scales and how its tasks are admitted and spread runs inside that agent's own executor. Nothing in it is a shared, platform-wide service.
| Logic | Runs in | Scope |
|---|---|---|
Pre-check, rate limiting, backlog (message_prechecker, rate_limiter, message_backlog) |
The agent's executor, for every incoming task | One agent |
Load balancing (load_balancer) |
The agent's executor, for every accepted task | One agent's instances |
Auto-scaling (autoscaler) |
The agent's executor, every 30 seconds | One agent's instances |
Message transformation, pre- and post-processing (message_transformer, pre_processor, post_processor) |
Each agent instance, inside the agents SDK | One instance |
| Creating and removing instances | The agent deployer of the cluster, when the auto-scaler asks | One cluster |
One Executor per Agent¶
Every deployed agent gets its own executor: one Kubernetes Deployment, in the agents namespace, with three containers (the executor, the job scouter and a Redis). The executor manages that agent's instances and no others.
Policies are not shared between agents at runtime. A policy stored once in the policies system can be attached to many agents, but each agent's executor downloads its own copy when it starts, installs the policy's requirements.txt, and runs it in its own process. For example, if three agents all use the same load-balancing policy LB1, there are three executors, each running its own copy of LB1 over its own instances. A slow or failing policy in one agent cannot affect another.
What Compute It Uses¶
The scaling logic uses the compute of the agent's executor pod, not that of the agent instances or of the platform services. The deployer does not set CPU or memory limits on the executor pod, so it uses what the node gives it; keep policies light, because the admission and load-balancing policies run for every task.
Each agent instance runs in its own pod with the resources set in the agent's runtime.resources. Adding instances adds instance pods; the executor stays one pod per agent.
How the Scaling Logic Itself Scales¶
Because there is one executor per agent, the scaling logic grows with the number of agents: deploying more agents adds more executors, each handling only its own traffic. To handle more load for one agent, add instances, manually or with an autoscaler policy. The executor is a single pod per agent and is not replicated.
For how policies are written, attached and run, see Agentic Governance.
Deploying at Scale¶
- More instances: list several entries in
allocation.instanceswhen deploying. Each instance runs in its own pod with its own Redis sidecar. - Placement: use
node_selectorper instance, andruntime.resourcesin the specification, to put instances on the right node pools, for example GPU nodes. - Many clusters: register one deployer per cluster and deploy each agent to the cluster that suits it. See Multiple Agent Clusters.
Load Balancing¶
The load balancer runs the agent's load_balancer policy for every accepted task.
| Policy input | {"session_id": ..., "message_data": {...}} |
| Injected into the policy settings | get_instances() returns the current instance IDs; load_balancer is the balancer object |
| Expected output | {"ok": true, "instance": "<instance_id>", "metadata": {...}}, a (instance_id, metadata) tuple, or just the instance ID as a string |
| Default (no policy) | A random live instance |
A custom policy can implement round-robin, least-loaded, session affinity (send every task of a session_id to the same instance) or content-based routing.
| Variable (executor) | Default | Description |
|---|---|---|
LB_REDIS_QUEUE_KEY |
jobs:inbox |
Queue key on the instance Redis |
Auto-Scaling¶
The executor includes an auto-scaler that runs every 30 seconds and asks the agent's autoscaler policy what to do.
| Policy input | {"instance_count": n, "instance_ids": [...], "queue_depth": n, "subject_id": "..."} |
| Injected into the policy settings | get_instances(), get_queue_depth() |
| Expected output | "scale_up", "scale_down" or None, or {"action": "scale_up"} |
| Default (no policy) | No action |
scale_up asks the deployer to create one new instance (POST /instances) with the agent's meshes and Delegate API URL. scale_down removes the most recently listed instance (DELETE /instances). A typical policy scales up when queue_depth / instance_count passes a threshold and scales down when the queue has been empty for a while, within minimum and maximum instance counts kept in its parameters.
The deployer gives every executor it creates the variables the auto-scaler needs:
| Variable (executor) | Set to | Purpose |
|---|---|---|
DEPLOYER_API_URL |
The deployer's in-cluster URL, http://agents-deployer.agent-deployer.svc.cluster.local:9000. Override it with DEPLOYER_API_URL on the deployer |
Where scale_up / scale_down requests go |
AGENT_DELEGATE_URL |
allocation.delegate_api_url from the deploy request |
Passed to the instances the auto-scaler creates |
MESH_LIST |
allocation.meshes from the deploy request |
New instances join the same NATS meshes |
Scaling is driven entirely by the policy. Without an autoscaler policy the instance count stays as deployed.
Example Auto-Scaling Policy¶
# code/function.py, attached with policy_type "autoscaler"
import time
class AIOSv1PolicyRule:
def __init__(self, rule_id, settings, parameters):
self.settings = settings
self.idle_since = None
def eval(self, parameters, input_data, context):
n = input_data["instance_count"]
depth = input_data["queue_depth"]
lo, hi = parameters.get("min_instances", 1), parameters.get("max_instances", 10)
per_instance = parameters.get("target_queue_per_instance", 20)
if n < hi and depth > per_instance * max(n, 1):
self.idle_since = None
return {"action": "scale_up"}
if depth == 0:
self.idle_since = self.idle_since or time.time()
if n > lo and time.time() - self.idle_since > parameters.get("idle_seconds", 300):
self.idle_since = None
return {"action": "scale_down"}
else:
self.idle_since = None
return {"action": None}
Rate Limiting and Backlogs¶
Under heavy load, admission policies protect the agent:
- The rate limiter (
rate_limiterpolicy) receives arate_counterwithincrement,get,resetand per-session metadata, and decides whether a task is allowed. Blocked tasks get HTTP429withstage: "rate_limit". - The backlog (
message_backlogpolicy) receives abacklog_store(push,pop_one,pop_n,peek_n,length,clear) and can hold tasks back, or release several queued tasks at once.
| Variable (executor) | Default | Description |
|---|---|---|
RATE_LIMITER_BACKEND |
memory |
memory or redis |
BACKLOG_BACKEND |
memory |
memory or redis |
REDIS_URL |
— | Redis used when a backend is redis |
With the memory backends, counters and backlogs are lost when the executor restarts. Use redis when that matters.
Routing¶
Tasks reach an agent in three ways, and each can be scaled independently:
| Route | Path | Notes |
|---|---|---|
| HTTP | POST <executor>/submitTask |
Used by the Delegate API and context.direct |
| P2P mesh | NATS message to the agent's executor | Same admission and load-balancing pipeline as HTTP |
| Task exchange | The exchange (and Orcade, for bidding) delivers through the Delegate API to /submitTask, and waits up to 300 seconds for the reply |
Exchange traffic goes through the same admission policies; see Task Exchange & Bidding |
To choose which agent gets a task, see Task Delegation and Dynamic Workflows.
Fault Tolerance¶
| Failure | Behavior |
|---|---|
| Instance pod crashes | Kubernetes restarts it (restartPolicy: Always). The executor's instance list is refreshed from pod labels, so tasks stop going to missing instances |
| Task in flight on a crashing instance | Redelivered. The SDK moves each task to a per-instance processing list (jobs:inbox:processing:<instance>) and removes it only after the task is handled. On restart, unacknowledged tasks go back on the queue (at-least-once delivery) |
| Task keeps failing | Retried up to REDIS_MAX_DELIVERIES times (default 3), then moved to the dead-letter list jobs:inbox:dead for inspection |
| Executor restarts | Rate counters and backlogs persist when the redis backends are used |
| Exchange temporarily unreachable | Results are retried 3 times with increasing back-off |
| Policy raises an error | Controlled by POLICY_FAIL_MODE: open (default) allows the task; closed blocks it with reason.error = "policy_error". See Agentic Governance |
Because delivery is at-least-once, make on_data idempotent per task_id, for example by checking a checkpoint before repeating side effects.
| Variable (instance) | Default | Description |
|---|---|---|
REDIS_MAX_DELIVERIES |
3 |
Attempts before a task is dead-lettered |
REDIS_PROCESSING_KEY |
<queue>:processing:<INSTANCE_ID> |
Processing list of this instance |
REDIS_DEAD_LETTER_KEY |
<queue>:dead |
Dead-letter list |
For long-running work that must survive restarts, see Persistence, Checkpointing & Recovery.