← Back to the summary

Google's Agent Substrate Packs 250 Stateful Agents onto 8 Pods by Treating Sandboxes Like Event Loops

📌 Core Summary: Agent Substrate integrates Serverless snapshot cold-start, Orleans Virtual Actor, and Knative Scale-to-Zero mechanisms, offloading related capabilities to the gVisor sandbox layer. It aims to solve the low utilization and high cost problems caused by resource residency for massive stateful Agents in Kubernetes (K8s) environments.

⚠️ Project Status: Currently in very early development, with API changes likely. The mentioned support for 1 billion Actors, 100ms activation latency, and 1000 wake-ups/second throughput are design targets. There is currently no public benchmark data, and direct use in production environments is not recommended.

1. Conceptual Model: Similarities and Differences between Event Loop and Agent Substrate

The design philosophy of Substrate can be understood through the JavaScript Event Loop mechanism:

Dimension	JavaScript Event Loop	Agent Substrate
Scheduling Unit	Callback Function / Promise	Actor (Sandbox Instance)
Execution Resource	Single Thread	Worker Pod Pool
`yield` (Relinquish Control)	`await` switches stack	Generate VM snapshot and save to object storage
`resume` (Resume Execution)	Callback function enqueued	Restore snapshot to a designated Worker
Wake-up Trigger	I/O and other events completed	Inbound request (intercepted by `atenet`)
Suspended State Storage	Heap Memory (RAM)	Object Storage (low-cost cold storage)

JavaScript handles high-concurrency connections through a single thread, while Substrate schedules massive Actors via a Pod pool. The essence of both is mapping N logical concurrency units onto M physical resources ($N \gg M$), relying on the characteristic that most units are in a waiting state to achieve resource oversubscription.

However, Substrate makes more aggressive trade-offs in system design:

Context Switching Overhead Difference: Code-level stack switching is typically at the nanosecond level, while VM-level snapshot and restore reach milliseconds to hundreds of milliseconds. Therefore, Substrate does not snapshot on every system call; it only suspends when an Actor enters a long idle state.
State Boundary Expansion: The suspended state needs to save complete memory state, file system, and kernel-mode information. Since the restore process may occur across nodes, the system must focus on solving the two major challenges of data locality and tiered storage.
Sandbox-Level Strong Isolation: The underlying layer is based on gVisor/Kata for strong isolation, capable of running untrusted code.
Storage Architecture Change: Traditional coroutine suspension is limited by memory capacity; Substrate transfers the suspended state to cheap object storage, making the scale of accommodatable Actors mainly limited by storage capacity and restore bandwidth. This is the underlying logic for its scale target of 1 billion.

2. Scenario Pain Points: Resource Utilization Bottlenecks of Agent Workloads

Substrate's service targets cover all Agent-like workloads (including AI Agents, sandbox code executors, MCP Servers, etc.). Such workloads have the following three significant characteristics:

High Burstiness: Long idle periods exist between consecutive requests (no inbound traffic); although processing a single request can take from milliseconds to days, the idle time between requests is much greater than the actual execution time.
Strong Isolation Requirement: Often need to run untrusted code, must rely on sandbox environments for single-tenant isolation, leading to explosive growth in the number of instances.
Stateful Dependency: Session context, file system state, etc., cannot be discarded.

If run directly on a standard Kubernetes system, the following bottlenecks will be encountered:

Architectural Pain Point	Kubernetes Limitation Reason
Idle Pods Occupy Resources	Adopts a 1:1 Agent-to-Pod binding model; CPU, memory, and node Pod quotas are long-term occupied by idle instances.
Control Plane Scale Bottleneck	etcd cannot support the concurrent management and high-frequency updates of millions of K8s objects.
High Scheduling Latency	Second-level latency caused by controller convergence, network route establishment, and image pulling is unacceptable for real-time response tasks.
Volume Management Overhead	PersistentVolume cannot support the high-frequency mounting and unmounting of millions of storage volumes.

Summary: Kubernetes was originally designed to manage long-running workloads and struggles to efficiently handle massive, short-lived, bursty, and stateful Agent workloads.

3. Core Architecture: Multiplexing Actors on a Worker Pool

The core mechanism of Agent Substrate can be summarized as:

Multiplexing massive Actors (Agent instances) onto a small number of pre-warmed Worker Pods; suspending via snapshot to release resources when idle, and restoring from snapshot within seconds when a request arrives.

3.1 Actor Lifecycle Example

gantt
    title Actor Lifecycle: Long-term Suspension, Instantaneous Activation
    dateFormat X
    axisFormat %s
    section Worker (CPU/RAM)
    Burst Processing 1 (Inbound Request) :active, 0, 1
    Burst Processing 2 (Inbound Request) :active, 6, 7
    Burst Processing 3 (Inbound Request) :active, 14, 15
    section State (Cheap Storage Persistence)
    Identity and Snapshot Persistence :done, 0, 20

Worker resources are only occupied during the burst processing triggered by inbound requests, and released to other Actors for the rest of the time; the Actor's identity and snapshot state persist in the storage layer at extremely low cost.

By interleaving the burst processing of multiple Actors in the time dimension, high-density multiplexing can be achieved using a small number of Pods.

The Actor lifecycle is as follows:

stateDiagram-v2
    [*] --> RUNNING : CreateActor
    RUNNING --> SUSPENDING : Idle / Active Suspend
    SUSPENDING --> SUSPENDED : Snapshot complete, release worker
    SUSPENDED --> RESUMING : Inbound request arrives
    RESUMING --> RUNNING : Restore from snapshot
    SUSPENDED --> [*] : DeleteActor

3.2 Three Key Design Elements

Suspension Mechanism (VM Snapshot) Utilizes the Checkpoint/Restore technology of gVisor (runsc) or Kata Micro-VM to freeze memory state, file system, and kernel mode and write them to object storage. After suspension, the Actor no longer consumes CPU and Worker memory.
Restore Mechanism (Currently Traffic-Driven) The lightweight network proxy atenet is responsible for intercepting requests destined for an Actor (identifying the target entity via the Host request header) and triggering the snapshot restore process. The target node for restoration is dynamic and can land on a Worker on a different physical machine.
Bypassing the K8s Control Plane The Actor lifecycle is not managed through the K8s API Server and native scheduler but is uniformly managed by an independent high-QPS control plane (ate-api-server based on Redis/Valkey). This strips K8s scheduling overhead from the critical path, reducing activation latency.

3.2 Architectural Design Goals (Metrics to be Verified)

Performance Metric	Target Setting	Architectural Intent
Single Cluster Actor Capacity	1 Billion	Ultimate design metric for system scalability
Wake-up Throughput	1000 times/second	Support large-scale concurrent switching from `SUSPENDED` to `RUNNING`
P95 Activation Latency	100ms	Reduce the overall latency from inbound request arrival to Actor resuming execution

4. Technical Evolution and Architectural Precedents

Substrate's core technologies have mature precedents in both academia and industry. Its essence is the engineering integration and offloading of known technical solutions, rather than a breakthrough in fundamental theory.

Technical Category	Academic / Industrial Precedent	Substrate's Engineering Difference
Snapshot Cold Start Optimization	Catalyzer (ASPLOS'20), SEUSS (EuroSys'20), vHive/REAP (ASPLOS'21), FaaSnap (EuroSys'22), AWS Lambda SnapStart	Offloaded to gVisor sandbox granularity
Warm-up Pool and Micro-VM	Firecracker (NSDI'20), SOCK (ATC'18)	Superimposed with stateful Checkpoint/Restore recovery
Sandbox-Level Checkpoint-Restore	CRIU, gVisor `runsc` C/R, `userfaultfd` on-demand paging	Optimized for high-density Agent multiplexing scenarios
Virtual Actor Model	Microsoft Orleans (MSR-TR'14), Erlang, Cloudflare Durable Objects	Offloaded to sandbox instance level, not process/thread level
Scale-to-Zero Traffic Wake-up	Knative Activator	Introduced stateful snapshot recovery, optimizing latency performance
Idle Resource Reclamation	Harvest VMs (OSDI'20), Memory-Harvesting VMs (ASPLOS'22)	Combined with object storage for cross-node stateless migration

The system's core engineering innovation points lie in:

Building an independent low-latency control plane, stripping away K8s scheduling dependencies;
Applying gVisor's Checkpoint/Restore mechanism to high-density Agent scenarios;
Using Redis/Valkey instead of etcd as the storage medium for Actor runtime state;
Optimizing snapshot routing based on data locality.

4.1 Currently Known Limitations

The project is in an early stage, API backward compatibility is not guaranteed;
Lacks public benchmark verification data;
gVisor's Checkpoint/Restore mechanism still has known defects in supporting long-lived network protocols;
Currently deeply bound to the GCP/GKE ecosystem (snapshots heavily depend on GCS), general environment support is still in the planning stage.

5. Storage Selection: Why Choose Redis Instead of etcd Directly?

The storage selection reflects the core trade-off between consistency and throughput in the system.

5.1 etcd's Design Positioning: Strongly Consistent Configuration Database

Kubernetes uniformly persists state in etcd. etcd provides multi-replica strong consistency based on the Raft consensus algorithm, positioned to store small-capacity, high-value, read-heavy, write-light cluster configuration data.

This positioning determines its architectural bottlenecks:

High Write Overhead: Each write operation must go through the Raft protocol for disk persistence and achieve majority consensus. Write throughput has a physical upper limit (limited by single-leader serial commits) and cannot be horizontally scaled by adding nodes.
Capacity Limit: The officially recommended database capacity limit is about 8GB.
Watch Event Storm: Massive objects and high-frequency changes can cause the API Server's Watch monitoring mechanism to carry excessive events, thereby dragging down the control plane.

5.2 Storage Challenges of Agent Workloads

Scenarios etcd Optimizes For	Actual Characteristics of Agent Workloads
Ten-thousand-level object scale	Million to billion-level Actor scale
Read-heavy, write-light	Single Actor high-frequency changes (state, physical location, snapshot pointer)
Long lifecycle, low-frequency updates	High-frequency bursty suspension and restoration
Strong consistency priority	100ms-level activation, latency-sensitive

If the K8s native API Server/etcd architecture is forced to host massive Agents, the K8s storage layer needs to be restructured (e.g., abandoning strong consistency assumptions, rewriting the event distribution mechanism), which contradicts K8s's design philosophy of ensuring cluster stability.

5.3 Substrate's Decoupling Solution: Data Hot-Cold Tiering

Substrate adopts a tiered data storage architecture:

K8s + etcd     →  Only manages low-frequency, high-value control plane data: CRDs like WorkerPool, ActorTemplate
                  (Small overall quantity, very low change frequency)

Redis/Valkey   →  Hosts high-frequency, massive runtime data: real-time status, physical location, snapshot pointer of each Actor
                  (Supports 100,000-level single-machine write QPS, can be horizontally scaled based on Hash Slots)

Redis/Valkey can meet high concurrency requirements because it makes different CAP trade-offs:

Weakened Strong Consistency: Based on in-memory reads/writes and asynchronous replication mechanisms, avoiding the overhead of cross-node consensus and disk fsync for every write.
Memory-Level Read/Write: Actor metadata is small in size, and memory operations are naturally suitable for high-frequency reads and writes.
Horizontal Sharding Capability: Can be sharded and scaled based on key, with write throughput increasing linearly with node expansion.

The corresponding architectural cost: The system gives up strong consistency guarantees at the storage layer; consistency control, fault recovery, and state calibration logic must all be implemented by Substrate at the application layer.

6. Kubernetes Native Evolution and Substrate's Positioning

With the evolution of the K8s ecosystem, some underlying primitives have gradually become native, but the core control logic will still maintain layered isolation.

6.1 Existing Primitive Supplements in the K8s Community

Primitive Capability	Evolution Status	Relationship with Substrate
Container Checkpoint/Restore	KEP-2008 (v1.25 Alpha, based on CRIU)	Currently mainly used for fault forensics; Restore capability stays at the runtime layer (containerd/CRI-O), not entered K8s core API
In-Place Resource Resizing	KEP-1287 (v1.27 Beta)	Simplifies the implementation complexity of dynamic Worker resource adjustment
Task Suspend/Resume	Job's `.spec.suspend` already supported	Provides a declarative suspension concept but does not cover sandbox-level snapshots
Sandbox Runtime Support	gVisor/Kata accessed via RuntimeClass	Infrastructure layer is fully mature

6.2 Technical Paths Not Easily Absorbed by the K8s Core Layer

etcd Protocol Stack Limitation: API Server/etcd cannot directly host millions of high-frequency changing objects.
Controller Asynchronous Convergence Latency: K8s's "controller convergence" model, based on state comparison and eventual consistency, trades off for extremely high generality and self-healing capability but cannot meet sub-second real-time activation needs.

Substrate's architectural core lies in stripping the critical path of Agent scheduling from the K8s control plane. This design has an essential difference from the evolution path of the K8s core architecture, making it more suitable to exist as an upper-layer extension (Add-on).

6.3 Predicted System Architecture Division of Labor

A more reasonable trend for future architecture division of labor is:

Kubernetes focuses on underlying infrastructure management—providing compute scheduling, network storage access, underlying Checkpoint/Restore, and sandbox isolation primitives; while Agent-specific logic such as high-density multiplexing of massive Actors and sub-second routing wake-up is carried by a dedicated orchestration layer (like Substrate) built on top of K8s.

This positioning is consistent with the evolution path of Knative (Scale-to-Zero) and KEDA (event-driven scaling): not changing the K8s core, but providing domain-specific scheduling capabilities as an upper-layer ecosystem component.

📎 Related References

Google Agent Executor (google/ax): A distributed Agent runtime built on Substrate

Agent Substrate Code Repository (Note: API may change during the development phase)