The upcoming v1 of the tokenizers library is a refactor centered on performance: the authors wanted tokenization to be lightweight and scale with workflows so that GPUs never idle waiting for CPU tokenization. The release candidate was benchmarked against v0.23 and other widely used alternatives.
Why tokenizer performance matters
Tokenization has not historically been the slow part of ML pipelines, but as models get faster and workloads scale, tokenization can become the bottleneck. Large-scale training, many concurrent requests, or repeatedly processing long inputs can starve models of data if tokenization cannot keep up.
Headline results
Across the ten model families covered, v1's encode path runs 3 to 30 times faster than v0.23 on an Apple M4 Max using a single thread; the low end of the range was t5-base and the high end gpt2. With eight workers it scales at about 76% of ideal linear speed. Throughout these changes, v1 produces exactly the same token IDs as the released v0.23 library.
What changed
Tokenizers performs encoding in four stages: normalization, pre-tokenization (splitting), the model step (turning pre-tokens into tokens and mapping to IDs), and post-processing. The v1 work targeted all stages, but most heavy optimizations were in the model step. Key changes include:
- Workspace split: the single crate was divided into tk-encode, tk-serialize, tk-convert and tk-train so applications link only what they use.
- No-alloc model: the merge working set lives in a scratch buffer owned by the caller so the merge loop does not touch the allocator.
- Bitcannon: the split pattern became Boolean operations over bitstreams using SIMD instructions instead of invoking a regex engine.
- Merge-loop rewrite: pieces being merged live as an intrusive doubly-linked list inside one preallocated buffer, so a merge updates two indices instead of moving data.
- Word cache: a thread-local memo maps pre-token bytes to finished IDs so repeated words are merged only once.
- Native parallelism: one shared tokenizer encodes from many threads; each thread draws its scratch buffer and word cache from a sub-pool, removing queueing on a single lock.
Bitstreams instead of regex for splitting
Byte Pair Encoding (BPE) models split input into pre-tokens using a fixed regular expression parameter that accompanies the model. Because that pattern never changes at runtime, a hand-written splitter can replace a general-purpose regex engine. The bitcannon approach views input bytes as parallel bitstreams and uses SIMD to operate on 64 bytes per register, deriving split boundaries via boolean operations rather than per-character scans. This yields large speedups when a model's split grammar matches the supported set; if it does not, the implementation falls back to the regex path and the speed-up is lost.
The word cache
Real text contains many repeated words. Since BPE produces the same token IDs for a given pre-token, v1 stores results in a thread-local cache mapping pre-token bytes to token IDs, letting later occurrences skip the merge process. Caching helps most when inputs contain repeated pre-tokens; inputs with few repeats can pay for lookups without receiving many hits.
The merge loop
Previously, each pre-token allocation and priority queue construction caused repeated allocations. v1 reuses a caller-owned scratch buffer, stores symbols in a flat array and links adjacent symbols by indices. Candidate pairs are packed into a single 64-bit value with the merge rank in the high bits, so comparing candidates is an integer comparison; the “no merge” sentinel is represented by the largest possible value to avoid branches.
Benchmark methodology
To keep comparisons consistent, the measurements follow rules such as: a single timing loop that all engines run identically, vocabulary load timed separately, verification that the output IDs' FNV-1a hash matches the baseline, medians computed only over cells run and verified by every engine, each repeat starting in a new process, pinning workers to distinct physical cores (no sibling SMT threads), and separate jobs to measure host-to-host variation. The authors note the difference between repeatedly encoding the same document (cache-warm) and encoding distinct documents (larger corpus with less cacheability) and base headline results on distinct documents where the corpus is too big to fit in cache.
Why these additions add up
The 3–30× single-thread speedups and the multi-thread scaling are the result of several combined changes: a hand-written splitter replacing a regex engine, a thread-local cache, a merge loop that avoids allocations, and batched model calls processing multiple pre-tokens per model invocation. Each change reduces work at a different pipeline stage.
Roadmap to 1.0.0 and beyond
Before 1.0.0 the team plans to move additional model families onto the new merge loop so training and inference share the same encoding implementation. Other planned items for 1.0.0 include optional computation of offsets and masks only when requested, reworking normalizers (including bitnorm support), simpler Python bindings with reduced locking, and inference-only C/C++ bindings for ExecuTorch and llama.cpp. After 1.0.0 they will explore tok-devices: GPU encoding and batch decoding that keep text and token IDs on-device, with vocabulary upload and parallel output-position computation — an optional component aimed at large batches pending further prototyping.
How to try v1
A v1 release candidate is available on crates.io: cargo add tokenizers --pre. The public API you call is unchanged, so switching to the pre-release is just a build/install change. Training is behind a default-on feature that pulls a C++ dependency; if you only need encoding, disable training to exclude that implementation. Encoding and encode_batch calls remain the same; encode_batch is the call the multi-core scaling measurements exercise.
Acknowledgements
The work built on many active open-source tokenization projects (gigatoken, tiktoken, kitoken, tokie, fastokens, wordchipper, ai-tokenizer and others). IBM, NVIDIA and the ExecuTorch team contributed patches and hardware testing that broadened platform support. The post and its figures are drawn from tokbench measurements and will be updated as support expands.
