v1 splits tokenization into smaller Rust crates
The v1.0.0-rc.2 line separates inference, JSON reading, legacy conversion, and training concerns. The tk-encode crate owns the hot path. tk-serialize reads canonical tokenizer.json files, tk-convert upgrades older files, and tk-train is the training half. A thin tokenizers crate preserves a convenient import point. This layout lets a serving binary avoid linking trainers and legacy conversion code it never calls.
Our checkout at commit fb11e51 contained 307 files, about 41,333 source lines, and 4.3 MB before installation. That is a compact source tree for a project serving Rust, Python, and Node.js users. The size also makes the split inspectable: a buyer can follow which crate enters an inference binary instead of accepting one large package with every historical feature attached.
The 102-package install built cleanly
Our sandbox installed 102 Rust packages in 41 seconds, then completed the build in 76 seconds. The environment was an unprivileged container with 3 CPUs, 12 GB of RAM, no secrets, and the lab-rust:1 image. Those numbers describe repository mechanics. We did not run the project's published performance comparisons, encode a production corpus, or compare output IDs against another tokenizer.
The repository had 13 CI workflow files, which is substantial coverage infrastructure even though our scan found no top-level tests directory or Dockerfile. Source contributors work from the tokenizers subdirectory, one level below the repository root. Python and Node bindings have their own build and test routes, so a successful Rust compile does not prove either package is ready on every wheel or npm target.
What happened when we ran it
Our cargo test step exited 0 after 56 seconds and reported 0 passed, 0 failed, and 0 total tests. That is a successful command with no discovered cases, not evidence that Tokenizers passed its full suite. The supplied log contains no failing test tail and no vulnerability result, so there is nothing else to infer from that step.
The rest of our run was straightforward: 102 packages installed in 41 seconds and the build finished in 76 seconds. The project lives in ./tokenizers/, and that is where the harness worked. Across the whole checkout we counted 307 files and about 41,333 source lines. A buyer can reproduce those mechanics without model credentials, but fixtures and Hub-backed examples add network requirements beyond this run.
The documented test path is wider than our 0-test command
The contributor guide uses make test from the Rust workspace. That target downloads pinned fixtures through the Hugging Face CLI before exercising the workspace. Python contributors use uv, maturin, and pytest, while Node contributors use Yarn. Our direct cargo command did not run those paths, so its 0-test report should send a maintainer back to the project Makefile.
This distinction matters because tokenization bugs often return valid-looking integers. A 76-second successful build cannot tell you that offsets still match source text, a saved tokenizer reloads identically, or a multilingual normalizer agrees with its reference. For a production change, use the documented fixture path, then add your own tokenizer.json files and strings that exercise the model families your application actually serves.
Official support covers Rust, Python, and Node.js
The README names 3 official surfaces: the Rust reference implementation, Python bindings, and Node.js bindings. Ruby support lives in an external community repository. C, C++, Java, and Go remain roadmap items, with Swift mentioned after v1 through future C bindings. Teams standardizing on one of those languages would be adopting a plan rather than a released interface.
The v1 candidate is inference-first and feature-incomplete
The v1.0.0-rc.2 candidate targets faster, smaller inference while the README explicitly lists features that have not returned. Python training, batch encoding views, DLPack exchange, mutable pipeline composition, save, paired inputs, offsets, stride windows, and streaming decode appear in the v1 work list. The project also says training is unavailable in the candidate until trainers emit the new pipeline models directly.
Our 102-package build proves that the checked-out Rust target compiles. It does not make those missing interfaces appear. Existing Python applications may depend on setters, trainers, serialization, or legacy module paths even if ordinary encode and decode calls look familiar. Inventory those calls before installing the pre-release package, and keep the stable v0.23.2 line available until the migration test covers real tokenizers.
Two open reports deserve focused regression cases
Two open issues describe correctness and safety failures with concrete inputs. Issue 2094 reports that a crafted BPE tokenizer.json can trigger a panic while loading v0.23.1, which may end a Rust or FFI process depending on panic settings. Issue 2116 reports normalization differences from SentencePiece on decomposed Unicode input, including Hangul and Vietnamese examples. We did not reproduce either report in the 56-second lab test.
The response is version-specific testing, not a blanket rejection. If users can upload tokenizer files, parse them outside the main serving process and set resource boundaries until the load-time report is resolved for your release. If multilingual offsets or token classification matter, add NFC and NFD forms to your fixtures. The zero-test lab result leaves both questions open, while the issue examples give a buyer useful cases to run.
September releases show activity and migration risk
GitHub showed 11,077 stars and 207 combined issues and pull requests on September 22, 2026. The last push was September 21. Stable v0.23.2 arrived September 3 as the final v0 release, and prerelease v1.0.0-rc.2 followed on September 21. The dates show active work on both the existing package and its replacement path.
The 4.3 MB checkout comes with detailed contribution and compatibility rules. It is also in a rewrite where old authoring surfaces are being replaced. New adopters who only load known model files have a credible v1 trial. Existing users with training, mutation, offsets, or broad binding needs should make the migration earn its place one tokenizer at a time.