mrkeyoor.com_
Thu 24 Sept 04:43 UTC
LLM Toolsevaluationupdated 24 Sept 2026

agent-beacon review

Beacon records coding-agent sessions from Claude Code, Cursor, Codex, OpenCode, and other harnesses in one local event format. It lets you inspect what agents did, forward those records to security tools, and turn reviewed lessons into project memory that other agents can retrieve.

Verdict

Our run installed 72 Go packages and passed all 18 tests, but it covered Beacon's unreleased sandbox contributor tool rather than the endpoint binary users install. Beacon is worth a controlled trial for a team running several coding agents and prepared to govern the resulting prompts, commands, diffs, and tool output. A single-agent developer who only wants chat recall is taking on more collection and configuration than the problem calls for.

We ran it

Lab card: what happened when we ran agent-beaconScreenshot of agent-beacon (beacon.sh)
Install✓ · 25s72 packages
Build✓ · 47s
Tests✓ · 10s18 passed · 0 failed of 18 (go test)
Repo1467 files~207,931 lines of source · 52 MB · 6 CI workflows · tests dir

Answers from our run

Does agent-beacon build from source?

Dependencies installed in 25 seconds (72 packages), and the build succeeded in 47 seconds. We cloned commit 239eefa into a clean Debian container with 3 CPUs and no project-specific setup.

Do agent-beacon's tests pass?

Yes: 18 of 18 passed when we ran the project's own test command (go test). Some failures need services or credentials a bare container does not have.

Who should not use agent-beacon?

Teams that cannot retain agent content on developer machines: the browser extension defaults to full retention, including prompt and response text, until an operator changes it.

What are the alternatives to agent-beacon?

Langfuse, Phoenix, Mem0. Our run installed 72 Go packages and passed all 18 tests, but it covered Beacon's unreleased sandbox contributor tool rather than the endpoint binary users install.

Setup3/5Source checks pass; endpoint setup changes services and agent configs
Docs5/5Detailed coverage, retention, install, and forwarding documentation
Community4/51,326 stars, a September 2026 push, and active pull requests
Maturity4/5Frequent releases, with an open uninstall restoration fix

Discussed on

  1. hnShow HN: Beacon - The open-source layer for local AI agent visibility21 points

Who it’s for

Developers who switch among several coding agents and want one searchable session history.
Platform or security teams that need command, tool, file, approval, and token records from agent work.
Teams willing to review extracted lessons before exposing them to future agents through MCP or Agent Skills.
Operators already equipped to set retention rules for prompts, responses, diffs, and tool output.

Who it’s NOT for

Teams that cannot retain agent content on developer machines: the browser extension defaults to full retention, including prompt and response text, until an operator changes it.
Developers who want identical telemetry from every agent: the README's coverage table has missing or partial fields for several harnesses, including file, approval, MCP, and token data.
Anyone who expects a passive binary that leaves agent settings alone: endpoint installation can add hooks, plugins, extensions, OTLP settings, and a local service.
Teams that require memory extraction to stay entirely offline: reviewing and retrieval remain local, but the documented evaluation command sends a bounded redacted trace to Jev or a compatible internal endpoint.
Buyers treating our passing source checks as an end-to-end product trial: the measured beacon-sandbox directory is a contributor tool that the project says is absent from releases.

Setup reality

Our sandbox installed 72 Go packages in 25 seconds inside beacon-sandbox, then built in 47 seconds. All 18 Go tests passed in 10 seconds. That directory is a contributor test tool, not the released Beacon endpoint, so this run confirms its source checks rather than a full installed-agent workflow.

A normal endpoint install adds a collector service and configures detected agent integrations. Local-only use needs no hosted account, while managed forwarding requires sign-in. Memory evaluation needs a TypeSafe API key or a compatible internal evaluator; SIEM and object-storage destinations need their own credentials.

The repo was 52 MB with 1,467 files and about 207,931 source lines. It has 6 CI workflows and a tests directory, but no Dockerfile. Coverage varies by harness, and interactive setup preselects managed forwarding unless you choose Local on the confirmation screen.

Beacon records more than a chat transcript

Beacon sits beside coding agents and turns their activity into a shared event stream. Depending on the harness, that can include prompts, responses, tool calls, shell commands, file changes, approvals, MCP activity, and token use. The local dashboard reads the resulting JSONL, while forwarding packs can send the same records to systems such as Splunk, Datadog, Elastic, Sentinel, S3, or GCS. That makes Beacon closer to an endpoint recorder for agent work than a simple conversation archive.

The breadth comes with real weight. Our checkout contained 1,467 files and about 207,931 lines of source, spread across the endpoint, integrations, documentation, packaging, and a cloud testing tool. The README lists more than 20 agent harnesses, but its own table shows that capture is uneven. Codex exposes no file events in the listed path, Gemini lacks session and command fields, and several agents do not report tokens. One normalized schema cannot create data a harness never emits.

Local storage still needs an explicit data policy

Beacon writes local JSONL by default, and choosing Local during onboarding keeps forwarding off. That is a useful boundary, though local does not mean low sensitivity. The documented inventory includes prompt text, assistant output, command lines, tool arguments and results, file diffs, usernames, repository context, and raw payloads for some integrations. Secret-pattern redaction and size limits run before records are written, but a developer laptop still holds an unusually detailed work log.

The 52 MB checkout is small beside the records a busy team may retain over time. Browser capture deserves special attention because its default retention mode is full, which stores complete prompt and response text locally. A metadata option drops that text before it leaves the page, and managed forwarding has a metadata-only mode that strips content fields before upload. Those controls are useful only after somebody chooses them, documents retention, and limits access to the files and downstream destination.

What happened when we ran it

Our sandbox installed 72 Go packages in 25 seconds, and the build succeeded in 47 seconds. The tests then finished in 10 seconds with 18 passed and 0 failed. Pip or npm audit results do not apply here, and the supplied run reported no vulnerability scan. The repository had 6 CI workflow files and a tests directory, with no Dockerfile.

There is an important scope limit. We ran the Go project under ./beacon-sandbox at commit 239eefa in an unprivileged Debian container with 3 CPUs, 8 GB of RAM, and no secrets. Its README calls it a testing tool for Beacon contributors and says it is not included in any release. The passing 18-test result supports that contributor utility. It does not show that hooks captured a live Claude Code session, that a service survived reboot, or that managed forwarding worked.

Installation changes both services and agent settings

The packaged route is friendlier than building the whole repository. macOS users get Homebrew commands, while Linux users download a DEB or RPM and Windows users get an MSI. Running beacon endpoint install adds a collector, creates a launchd job or systemd unit where available, and configures detected harnesses. An explicit harness list narrows those edits, and --dry-run prints them first. Those options matter on a workstation whose agent configuration is already managed.

Six CI workflows provide more visible automation than many young agent tools, but the absence of a Dockerfile matches the product shape: this is an endpoint service tied to host agents, files, hooks, and service managers. Interactive setup signs in through beacon.sh and preselects Beacon Managed, then explains Standard and Metadata-only forwarding before confirmation. Choose Local if records must stay on the machine. Automated and package installs remain account-free and noninteractive.

Memory becomes reusable only after review

The memory workflow is sensibly separated from raw capture. Beacon selects traces, sends a bounded redacted projection to Jev or a compatible internal evaluator, and stores candidate lessons in a local SQLite database. A person can inspect, approve, reject, or supersede each candidate. Approved memory is available through read-only MCP tools, and a separate command can write it as an Agent Skill inside the current project. Hooks do not approve candidates or install skills on their own.

Our 10-second test run did not exercise that network evaluation because the sandbox had no secrets. Teams should therefore test the exact redaction and evaluator endpoint they plan to use before sending a real trace. The project scopes approved memory by repository, which reduces accidental cross-project retrieval. It does not decide whether a captured command, diff, or response was appropriate to preserve. Human review is the part that keeps one bad session from becoming standing guidance.

Rapid releases coexist with an open uninstall repair

GitHub recorded a push on September 23, 2026, and release v1.3.22 landed the same day. The repository had 1,326 stars and 4 open issues and pull requests when fetched; all four returned items were pull requests. That is current activity, not proof that every integration is settled. Release v1.3.22 bundled Vector into Linux artifacts and included fixes around sign-in and forwarding.

One open pull request describes an uninstall case where user-mode setup could leave Codex hooks pointing to a dead log, plus a Homebrew upgrade that needs beacon endpoint repair --user. The proposed fix adds backups and restoration. Our 47-second build did not cover that path because it targeted beacon-sandbox, so workstation rollout should begin with a dry run, one test machine, a retention decision, and a verified uninstall. Beacon earns that trial when cross-agent evidence is the problem you actually have.

Alternatives

ProjectWhat it isPick it when
Langfuse gh↗An open-source tracing, evaluation, and observability platform for LLM applications.pick this instead when application traces and model evaluations matter more than developer-machine agent history.
PhoenixA self-hostable observability and evaluation system for AI applications.pick this instead when you need model and retrieval diagnostics rather than hooks across coding-agent harnesses.
Mem0 gh↗A memory service for agents and applications that need persistent user or task context.pick this instead when your product needs an application memory API rather than endpoint telemetry and session replay.

What people are saying

  1. [github-trending] Asymptote-Labs/agent-beacon

Sources

  1. Agent Beacon repository and README
  2. Beacon data inventory
  3. Beacon endpoint install documentation
  4. Beacon sandbox contributor tool README
  5. Beacon v1.3.22 release
  6. Open uninstall and Homebrew upgrade fix

More llm tools reviews

MiMo-Code · pi-claude-bridge · treg · train-llm-from-scratch · mistral.rs · flue · the whole board →