mrkeyoor.com_
Tue 11 Aug 19:10 UTC
Dataevaluationupdated 11 Aug 2026

tempo

Grafana Tempo stores and searches distributed traces, the records that show how one request moves through multiple services. It gives operations teams a lower-cost tracing backend built around object storage, with OpenTelemetry ingestion and close links to Grafana, Prometheus, and Loki.

Verdict

Tempo is a strong tracing backend for organizations already committed to OpenTelemetry and Grafana, especially when cheap object storage and high ingest volume outweigh the appeal of an all-in-one product. The monolith is approachable, but the serious deployment is not object storage alone: Tempo 3 high availability means Kafka and a distributed system. Use it when your platform team can own that stack, not merely because the Docker demo looks simple.

Setup3/5Easy monolith trial, demanding secure and highly available operation
Docs5/5Excellent architecture, sizing, migration, query, and operations guides
Community5/5Active Grafana project with current commits and focused issue traffic
Maturity4/5Proven at scale, while the major 3.0 architecture is still settling

Who it’s for

Platform teams already using Grafana and OpenTelemetry that need durable, searchable trace storage.
Organizations ingesting enough traces that object-storage economics matter.
Operators who want to correlate traces with metrics and logs through the Grafana stack.
Teams prepared to run either a simple single process or a Kafka-backed microservices deployment as volume grows.
Engineers willing to learn TraceQL for targeted searches beyond the point-and-click drilldown interface.

Who it’s NOT for

Teams looking for a complete tracing product in one binary: Tempo is the backend, while visualization comes through Grafana and useful data still depends on instrumentation or collectors.
Anyone planning to expose Tempo directly: the official operations guide says it includes no authentication layer and requires an authenticating reverse proxy.
High-availability operators who want the README's object-storage-only simplicity: Tempo 3 microservices mode requires a Kafka-compatible system and several independently operated components.
Organizations that cannot accept AGPL-3.0-only for the server, subject to the project's documented Apache exceptions.
Tempo 2 operators who need a reversible in-place upgrade: the 3.0 guide says there is no in-place downgrade after the old deployment is removed, and microservices migration runs both versions in parallel.
Teams depending on unusual S3-compatible storage without validation: an open 3.0 report shows startup failing against stores that implement only ListObjectsV2.

Setup reality

A local monolithic Tempo process and Docker Compose demo are reasonable ways to prove ingestion and Grafana queries, but they do not represent a production tracing system. You must instrument applications, normally place an OpenTelemetry Collector or Grafana Alloy in the path, configure object storage, connect Grafana, secure every API behind a proxy, choose retention, and monitor ingestion and query health. High availability moves you to microservices mode, which adds Kafka, distributors, live stores, block builders, query services, backend workers, caches, and careful capacity planning.

Trace storage without an Elasticsearch-sized bill

Tempo's core bet is that distributed traces can live economically in object storage instead of a large indexing cluster. It accepts OpenTelemetry, Jaeger, and Zipkin data, assembles spans into traces, stores long-term blocks in Amazon S3, Google Cloud Storage, Azure Blob Storage, or a compatible service, and retrieves them by ID or through TraceQL. Local disk exists for development and testing, but the documentation recommends object storage for production.

That design is attractive because trace volume grows quickly. Object storage offers cheap capacity and familiar durability, while Parquet blocks, bloom filters, compaction, caching, and dedicated attribute columns reduce the amount queried. Costs do not vanish. Retention drives stored bytes, searches produce read operations, block maintenance produces list and write calls, and broad TraceQL scans consume query capacity. Tempo makes the storage layer simpler and cheaper than many indexed systems, not free.

The backend also fits naturally into Grafana's observability stack. Grafana displays individual traces and correlates them with Prometheus metrics or Loki logs. Traces Drilldown offers point-and-click views of rates, errors, duration, and attribute comparisons for users who do not know TraceQL. TraceQL remains available when an engineer needs precise structural and attribute filters. TraceQL metrics can aggregate trace queries into metrics, although the README still labels that feature experimental.

A backend, not an observability appliance

Tempo does not instrument an application. Teams still need OpenTelemetry SDKs, sensible sampling, and usually a Collector or Grafana Alloy to receive, batch, enrich, and forward spans. It also does not bundle its main analysis interface. You add Tempo as a Grafana data source and install or enable the relevant Grafana experience. Poor attributes, missing context propagation, or aggressive sampling will produce poor investigations regardless of the backend.

Security is another deliberate omission. Grafana's official operations guide states that Tempo has no included authentication layer. It recommends putting an authenticating reverse proxy in front of its services. In multi-tenant mode, that proxy must set the trusted X-Scope-OrgID header. Exposing an ingestion or query endpoint without this boundary is unsafe, and accepting a tenant header directly from an untrusted client defeats isolation.

The AGPL-3.0-only license also needs review, with Apache-2.0 exceptions documented separately by the project. For many internal deployments this is manageable. Organizations modifying or offering the service should have their legal team assess the actual use rather than assuming all Grafana infrastructure projects share one license.

Two deployment modes, very different jobs

Monolithic mode runs the required pieces in one process and needs no Kafka. It is the right evaluation path and can support meaningful workloads. Grafana's guide positions it below roughly 25 to 35 MB per second, or 55,000 to 80,000 spans per second, while warning that actual sizing depends on trace shape and queries. Query spikes and writes share resources, components cannot scale independently, and running multiple all-in-one instances is unsupported because the backend scheduler is a singleton. That last point rules it out for teams requiring high availability.

Microservices mode separates distributors, live stores, block builders, queriers, query frontends, backend scheduling, and workers. Tempo 3 uses a Kafka-compatible system as the durable queue between ingestion and consumers. This provides failure isolation and independent scaling, but it makes the README's claim that only object storage is required incomplete for a highly available deployment. Operators must size Kafka partitions with matching live-store and block-builder replicas, provide scratch disk and upload bandwidth, cache hot object data, and monitor each stage.

The sizing guide is refreshingly candid: memory ranges for live stores, queriers, and query frontends can span 4 to 20 GB depending on traces and queries. These are starting points, not promises. Use real production-shaped spans and search windows in a load test before purchasing a cluster.

Tempo 3 is a meaningful migration

Version 3 replaces the old ingester path with Kafka, live stores, and block builders in microservices mode. It removes scalable single-binary mode and the old compactor target. Monolithic users can migrate configuration and update one binary, but distributed users must run Tempo 2 and 3 in parallel, share object storage, switch traffic, verify data, then retire the old system. There is no in-place downgrade after decommissioning 2.x. Older block formats must move to vParquet4 or later first.

The project provides a configuration migration command and Tempo Vulture for end-to-end write and read checks, which is good operational tooling. Still, inspect generated files before replacement. An open report shows tempo-cli migrate config producing a nearly empty output file when redirected while hiding a legacy-format error that appears without redirection. Preserve the old configuration and validate the new file before any rollout.

Current 3.0 edges also deserve staging. An August report says Tempo cannot start against an S3-compatible backend that supports ListObjectsV2 but not the legacy V1 call. Another report describes duplicate attribute keys losing earlier values when mapped to a dedicated Parquet column. A live-store report reproduces a panic during concurrent ID queries for a large trace. Each is configuration-specific, but together they justify pinned versions, representative canaries, and end-to-end trace checks.

Strong project, serious operating commitment

Tempo was pushed on August 11, 2026. The latest release, 3.0.2, shipped June 9 with an updated Go toolchain carrying security and bug fixes. GitHub showed 174 open items, including 91 issues and the remaining pull requests. Current work spans storage, querying, Kafka, backend maintenance, security dependencies, and documentation. This is healthy activity around a major architecture transition.

Tempo is easy to recommend as the storage core of a Grafana-centered tracing platform. It is harder to recommend to a small team seeking one install, one UI, and no platform ownership. Start monolithic, prove that traces answer real incidents, then move to microservices only when availability or measured volume demands it. That path preserves Tempo's economic advantage without volunteering for Kafka and a fleet of components too early.

Alternatives

ProjectWhat it isPick it when
JaegerA CNCF distributed tracing platform with OpenTelemetry foundations and its own established query UI.pick this instead when you want a long-established tracing project and prefer Jaeger's UI and storage choices over Tempo's Grafana-centered workflow.
ZipkinA focused distributed tracing server with a simple UI and a long history in service diagnostics.pick this instead when your trace volume and query needs are modest and operational simplicity matters more than large object-storage scale.
SigNozAn OpenTelemetry-native observability platform combining traces, metrics, logs, dashboards, and alerts.pick this instead when you want a more integrated open-source application rather than assembling Tempo with separate Grafana-stack components.

What people are saying

  1. [github-trending] grafana/tempo
  2. [theverge] Spotify Running Mode helps match tunes to tempo

Sources

  1. Grafana Tempo repository and README
  2. Grafana Tempo documentation
  3. Tempo deployment modes
  4. Tempo 3.0 migration guide
  5. Tempo v3.0.2 release
  6. ListObjectsV2-only storage startup report
  7. Dedicated column duplicate attribute report
  8. Live-store concurrent query panic report