mrkeyoor.com_
Mon 21 Sept 11:52 UTC
Open Source6 min read

Google's AX v0.3 Draws 500 Points and 10 Open Bug Reports

Google rebuilt AX around Redis and Agent Substrate. The launch found a large audience, while early reports exposed the distance between its design and production use.

Google's AX collected 500 Hacker News points and 220 comments in just under three hours. The consequential detail was buried in the code: a five-commit v0.3 diff replaced the project's earlier embedded harness with a Redis-backed control plane for agent tasks. The Hacker News thread supplied the attention. The rewrite explains why developers should care.

That distinction matters because AX is easy to mistake for another library that tells a model how to call tools. Its current job starts one layer lower. AX accepts declarations for isolated tasks, workspaces, network rules and model settings, then reconciles those resources onto Google's separate Agent Substrate runtime. The project README describes the goal as billions of tasks per cluster, while warning that the concepts and protocols may break before a stable release.

The five-commit rewrite behind v0.3

Google published AX v0.3.0 on September 20 with no release notes or attached binaries. The comparison with v0.2.3 is more informative. One large commit reorganized the project around three Go services: ax-server, ax-controller and ax-task-runner. It removed the legacy Python harness, SQL event log and earlier command structure, then added new manifests, documentation and a Redis store.

That is a sharp change from Google's May introduction of Agent Executor. The original Google Cloud announcement focused on durable event logs, connection recovery and branching an agent run from a checkpoint. In v0.3, the public interface is narrower and more infrastructural. The current concepts document defines four resources called Task, Workspace, Gateway and Model. Current v0.3 documentation does not expose those earlier features as AX's main interface.

A Task is deliberately smaller than an agent. AX does not model planning, delegation or retries. The docs say an agent may use one task for an entire job or create a tree of tasks as work expands. That boundary lets teams keep an existing harness while asking AX to handle isolation and lifecycle operations, a division spelled out in the Task definition.

A Kubernetes-shaped interface with different storage

The manifest reference uses an object familiar to anyone who has written Kubernetes YAML. A minimal task names an image, a command and the resources AX should bind to it:

apiVersion: ax.io/v1alpha1
kind: Task
metadata:
  name: code-check
spec:
  image: ghcr.io/example/code-agent
  command: ["python", "agent.py"]
  workspaces:
    - name: project
  gateway:
    name: restricted-egress

The manifest also supports CPU and memory limits, environment variables and several workspaces. The CLI follows the same mental model: ax apply, get, describe, watch and delete resemble kubectl, while suspend, resume and ssh cover the stateful parts of an agent session. AX follows the active Kubernetes context and can open a tunnel to its gRPC server, according to the CLI guide.

The similarity stops at storage. Recording millions of short jobs as Kubernetes custom resources would put pressure on etcd, so AX stores resource records in Redis and sends reconciliation work through Redis Streams. A stateless gRPC server writes each change. Controller workers consume the stream and call Agent Substrate to create actors, assign workers and apply egress rules. The architecture document says more controller replicas provide horizontal scale.

A local trial still requires a Kubernetes cluster, ko, a container registry and a reachable Agent Substrate Control API. make deploy installs Redis and AX's control-plane services in the ax-system namespace. The quick-start prerequisites put v0.3 in reach of teams already comfortable operating clusters. Local agent developers do not get a one-binary setup.

The security controls have real edges

AX puts useful policy beside the workload. A Gateway has an outbound host allowlist, a Task has compute limits, and a Model refers to a credential stored as a Kubernetes secret. Workspaces can declare Git repositories, MCP servers and skill registries. The resource examples make those boundaries visible in the same YAML that launches the task.

Interactive access is explicit. Setting debug: true turns on guest services that can execute processes and read or write files inside the sandbox. The default is off. The task runner remains PID 1 after the child command exits so metadata and debug access keep working. AX sends SIGTERM to the child's process group during suspension, waits ten seconds, then kills anything left, as documented in the sandbox guide.

Plain-language workspace setup adds a less deterministic path. When a workspace binding includes a goal, the runner asks an Antigravity agent to install tools and prepare the directory before the actual command starts. The operation needs a Gemini credential in the container and has a ten-minute default timeout. That behavior is convenient for experiments, but the same runner documentation gives teams enough reason to prefer a pinned image when repeatable setup matters.

One launch-day report identifies a separate credential gap. AX's Model object can reference a Kubernetes secret for AX's own model calls, but a task environment variable currently accepts a literal value and the task schema has no model reference. The external reporter demonstrated that a literal credential came back through ax get task. Issue 348 remains open, so this is a reported defect rather than a maintainer-confirmed security advisory.

Ten reports probe the new control plane

Issues 345 through 354 arrived on September 21 and were still open during reporting. Four came from a user testing v0.3 against a two-node kind cluster. Six more included executable regression tests against the controller or an isolated Redis server. None of the reporters is marked as a repository member, and the threads had no maintainer confirmation at the time checked. The reports are evidence worth testing, not settled findings.

The network cases pull in opposite directions. Issue 345 says a hostname allowlist blocked TLS traffic even to permitted hosts, while wildcard and CIDR rules worked in the reporter's cluster. Issue 350 says a task that references a missing gateway instead receives wildcard egress and starts. The second report used an in-process Substrate fake and checked the control RPC, so it does not claim a packet-level cluster test.

Status reporting also needs scrutiny. In issue 346, a task stayed Running and Ready: True after its child command exited, including one case where Substrate already considered the actor crashed. Issue 347 describes a Git setup that produced an empty repository while AX reported WorkspaceReady: True. Both reports came from a deployed test cluster and include commands or logs for reproduction.

Redis supplies three more failure modes in the queue. One regression says a new consumer group skips tasks written before the first controller starts. Another says an event delivered to a dead worker remains in Redis's pending list because the subscriber reads only new entries. A third test says a status write can overwrite a concurrent task-spec edit. These are exactly the ordering and recovery paths that matter once Redis becomes the source of truth.

The scale claim belongs to two projects

The AX project site says one cluster can run billions of concurrent sessions and resume suspended work in under a second. Its linked documentation does not provide an AX benchmark for the billion-task figure, and the repository's standard test target uses a mock Substrate gRPC server. The figure should be read as a design target until operators can reproduce it with a published workload and cluster configuration.

Agent Substrate carries the lower-level performance claims. Its README reports sub-500-millisecond resumes, more than 500 suspend or resume activations per second, and a demo that multiplexes about 250 stateful actors across eight pods. The same Agent Substrate status note says the software is in early development, is not ready for production and offers no backward-compatibility guarantee. AX depends on that runtime for actor creation, checkpointing, routing and network enforcement.

AX therefore supplies the vocabulary for agent infrastructure while Substrate handles actor state and placement. The v0.3 rewrite makes that separation easier to see. Its first issue batch identifies the seams to test before real credentials or unattended code enter a cluster.

Before deploying AX, watch the open failure reports for fail-closed gateway resolution, Redis event recovery, accurate task state and secret references for workloads. Published end-to-end numbers would make the scale claim easier to evaluate. The next useful count will come from the repository: how many of those ten reports gain regression tests and fixes before another tag lands.

We reviewed this

  1. servers — our honest review
  2. kubernetes — our honest review
  3. sharp — our honest review

Sources

  1. AX discussion on Hacker News
  2. AX v0.2.3 to v0.3.0 comparison
  3. AX v0.3.0 release
  4. AX repository and README
  5. Google Cloud Agent Executor announcement
  6. AX core concepts
  7. AX design
  8. AX manifest reference