The backend connects ESP32 speech to several AI services
xiaozhi-esp32-server receives voice and device traffic over WebSocket or MQTT plus UDP, then routes work through speech recognition, a language model, speech synthesis, and optional vision or memory components. The management side handles users, devices, agents, configuration, and authentication. Tool calling includes device IoT commands, client and server MCP paths, an MCP endpoint, and custom functions. This breadth makes it closer to an assistant platform than a thin relay for one ESP32 board.
The project is Chinese first. Its main README, deployment guidance, release notes, and most issue discussion are Chinese. A substantial English README is available and the management interface supports Simplified Chinese, Traditional Chinese, and English. That is enough to understand the architecture and deployment choices, but an English-only operator may still need translation when debugging a current issue or following a linked guide. Language support should be evaluated as part of operations, not only the user interface.
The README explicitly rejects production deployment
The maintainers state that functionality is incomplete, the system has not passed a network security assessment, and it should not be used in production. That warning is unusually direct and should control the buying decision. A public learning deployment still needs protective measures. The stack handles device identity, voice data, model credentials, management accounts, webhook-like tool traffic, and potentially knowledge-base content, so exposing it casually creates more risk than a local demo server.
Release v0.9.6 was published on July 24, 2026. Its notes describe server-side acoustic echo cancellation, agent snapshot history, timezone fixes, and voiceprint prompt changes. GitHub recorded 10,432 stars, 715 combined open issues and pull requests, and a push on August 21. That is active development with a large support surface, not evidence that the production warning is outdated. The combined count also includes proposed changes, so it must not be read as 715 confirmed bugs.
What happened when we ran it
Our fresh Debian sandbox installed commit c02becd's main/manager-mobile project in 73 seconds. Pnpm added 1,407 packages and used 464 MB on disk. The build succeeded in 21 seconds. The repository checkout was much larger: 1,585 files, about 180,563 lines of source, and 193.3 MB. We found an npm-workspaces monorepo, 2 CI workflow files, no Dockerfile at the measured scope, and no tests directory.
There was no test script or target, so our harness skipped tests. The result verifies that the mobile management client could install and compile at that commit. It does not verify the Python server, Java management API, database migrations, ESP32 protocol, microphone and speaker behavior, or any hosted model provider. We also had no secrets and no device in the unprivileged 3-CPU, 8 GB container. A deployment trial must exercise a full spoken request and a management action on real hardware.
Simplified and full installs serve different owners
The simplified route supports conversation and single-agent management with configuration-file storage, so it avoids a database. The full route adds multi-user and multi-agent management, a graphical control console, and database persistence. Docker and source deployment guides exist for both. The English README gives a 2-core, 2 GB baseline when every AI component is an API, while local FunASR raises the simplified recommendation to 4 GB and the full setup to 4 cores and 8 GB.
Provider configuration is the larger variable. The project lists local and hosted choices for speech recognition and synthesis, language and vision models, voiceprints, memory, and retrieval. Some paths need paid credentials; local paths need models and more compute. A free home configuration can combine local services with free tiers, but provider terms, quotas, latency, and data handling remain separate decisions. Build a configuration inventory before opening firewall ports or asking users to enroll voiceprints.
MCP expands the device, with a documented timeout edge
MCP can expose tools to the assistant and send commands from the console to ESP32 devices through MQTT. That can turn a voice endpoint into a controller for appliances or custom functions. It also widens the trust boundary: tool descriptions enter model context, calls may reach local devices, and credentials may connect to external servers. Grant a small tool set, log the selected tool and arguments, and keep physical actions behind explicit checks where a mistaken call could matter.
Issue 3290 reports that MCP calls are limited to 10 seconds and that slower tools time out, with a request for configurability. Issue 3268 separately asks how to handle poor recall when too many tools inflate context. Neither report proves every deployment suffers the same behavior, but both identify realistic design limits. Test the slowest tool and the largest intended catalog before promising hands-free reliability. Long-running work may need an asynchronous pattern outside the current request window.
Hardware pairing is part of setup, not an optional demo
The README targets people who already own ESP32 hardware, have connected it to an existing Xiaozhi backend, and now want an independent server. That assumed experience matters. Issue 3323 describes a second NodeMCU-ESP32 appearing online in one view but unlinked in the agent view despite reflashing and rebinding. Another report describes repeatable disconnection of an ML307 cellular module after about 10 seconds against a self-hosted endpoint. These are reports, not universal results, but they show why one successful browser build is insufficient.
Use a staging device with the exact firmware, network path, microphone, speaker, and transport planned for deployment. Verify enrollment, reconnection, interruption, OTA checks, an MCP action, and a provider outage. The 21-second mobile build makes source inspection easy; it does not reduce the system into a mobile app. The honest fit is a home lab, classroom, or contributor setup where failures can be investigated. The project's own security warning rules out a production recommendation today.