GigaToken: A Rust Tokenizer Claiming ~1000x Faster Than HuggingFace

TL;DR: Marcel Rød open-sourced GigaToken, a Rust tokenizer claiming up to ~1000x faster encoding throughput than HuggingFace Tokenizers on the same hardware — reaching gigabytes per second. It hit ~1,300 GitHub stars within a day. The speedup comes from SIMD-accelerated pretokenization and aggressive caching, not just "rewrite it in Rust" (HuggingFace's tokenizer and tiktoken are already Rust-based). It matters most for pretraining data pipelines and, more modestly, time-to-first-token in inference.

What GigaToken claims

From the project README:

"~1000x faster than HuggingFace's tokenizers, drop-in replacement. Tokenize your text data at GB/s!"

That claim is notable specifically because both HuggingFace Tokenizers and tiktoken — the two comparison points — are already Rust-based with Python bindings, not naive pure-Python implementations. A 1000x speedup against an already-compiled, already-optimized baseline is a much stronger claim than beating a slow interpreted implementation, which is why the project's own FAQ addresses the skepticism directly:

"Did you just way over-optimize for a specific CPU and tokenizer? How is it so fast? No, I way over-optimized for every combination of these!"

TL;DR — GigaToken at a glance

Where the speed actually comes from

Rød's own explanation, quoted directly from the FAQ, names three specific techniques:

"The major improvements are in optimizing heavily an implementation that usually is outsourced to a Regex engine (pretokenization) using SIMD, minimizing branching and other tricks, as well as heavily optimizing caching of pretoken mappings (if a word has been seen before, look it up its encoded tokens efficiently). Caching is a very hard problem in this domain since the cache grows very quickly, and pretoken distributions are very long-tailed. Some gains are also achieved from minimizing interactions with Python, and avoiding communication between threads."

Breaking this down for readers less familiar with tokenizer internals:

• Pretokenization is the step that splits raw text into word-like chunks before byte-pair-encoding merges are applied — most tokenizer implementations run this through a general-purpose regex engine, which is flexible but comparatively slow. Replacing it with hand-tuned SIMD (single-instruction-multiple-data) code exploits CPU vector instructions to process multiple bytes per cycle instead of one.
• Pretoken caching exploits the fact that natural text is highly repetitive at the word level — common words appear constantly, so once you've computed a word's token encoding once, you can look it up instead of recomputing it. The stated difficulty is that this cache grows very quickly and word-frequency distributions are long-tailed (a few words dominate, but the "tail" of rare words is enormous), making cache design genuinely hard to get right.
• Minimizing Python interop and avoiding inter-thread communication are lower-level systems optimizations — every crossing between Python and Rust, and every point where threads need to coordinate, costs real overhead that compounds at gigabyte-per-second throughput targets.

The benchmark numbers

The project's headline benchmark, run on an 11.9GB sample of the OpenWebText dataset:

Note the comparison methodology: HuggingFace's number is measured on a 100MB subset for practicality, while GigaToken's is measured on the full 11.9GB file — the author validates that both produce matching output on the shared subset before reporting the speedup ratio, addressing the most obvious methodological objection (are they actually tokenizing the same thing correctly?).

At the EPYC throughput rate, the README notes you could tokenize the entirety of Common Crawl — often described as effectively "the entire internet," roughly 130 trillion tokens — in under 6.5 hours.

Does tokenization speed actually matter?

This was the most-debated question in the Hacker News discussion, and the honest answer is "it depends on what you're doing."

For a single inference request, tokenization genuinely is a tiny fraction of total compute — commenters cited figures around 0.1%. If you're serving one chat completion, GigaToken's speedup is close to irrelevant to end-to-end latency.

For pretraining data pipelines, it's a different story. Rød, responding directly on the thread:

"In my case it's mostly pretraining experiments, where you might want to change your data mixture/filtering/processing of training data, and splits are usually done at a token-level instead of a text level. In this case we usually run for days on a huge number of CPUs to finish tokenizing something like DCLM."

A multi-day tokenization job that becomes a multi-hour job directly speeds up how fast researchers can iterate on data mixture changes — a meaningfully different value proposition than single-request latency.

For time-to-first-token (TTFT) specifically — not total inference time — the author shared preliminary numbers from testing against sglang on an 8B Qwen3 model on a single B200 GPU:

These are preliminary, not yet in the published README, and TTFT gains are smaller than raw tokenization-throughput numbers would suggest — because tokenization is only one part of the prefill pipeline, and the GPU-bound work still dominates total time. But the gains are real and grow with input length, which matters increasingly as long-context agentic workloads become more common.

One security-adjacent use case surfaced too:

"I run an AI platform and we need to tokenize fast and early to make a lot of decisions on the subsequent steps (things like routing, rate limiting and such). Its really important to do this efficiently even though its not a large % of total end to end time for the request." — a Hacker News commenter running production AI infrastructure

For platforms that tokenize before routing a request to a specific model or backend, tokenization sits on the critical path even though it's a small fraction of overall compute — the same "latency-critical, not throughput-critical" distinction that shows up across systems engineering generally.

Current limitations

The project is transparent about what isn't finished yet:

• WordPiece (used by BERT-style models) is not yet supported.
• SentencePiece-based tokenization is meaningfully less optimized than BPE — deprioritized since it's used mainly by Google/BERT-style models rather than the more common frontier LLM tokenizers.
• File sinks aren't implemented in the native GigaToken API yet — output currently goes through Python.
• Windows support is largely untested; WSL is recommended in the meantime.
• Compatibility mode, which matches HuggingFace/tiktoken output exactly for drop-in replacement, costs meaningfully more overhead than the native API — expect ~200-300x rather than ~1000x in that mode.

What this means for teams running LLM infrastructure

If you run pretraining data pipelines:

• This is worth a serious look if tokenization time is currently gating your data-iteration speed — the difference between a multi-day and multi-hour tokenization run changes how many data-mixture experiments you can realistically run per week.
• Check the validation command against your specific tokenizer before switching production pipelines — the project recommends running the built-in validator to confirm output matches your current tokenizer exactly.

If you're optimizing inference latency:

• The TTFT gains are real but modest (5-8% in the author's early numbers) — worth adopting if you're already optimizing aggressively at the margins, but not a silver bullet for overall latency.
• If tokenization sits on a routing or rate-limiting critical path before your model call even starts, the latency-critical (not throughput-critical) framing applies more directly than raw MB/s numbers suggest.

If you're evaluating build-vs-adopt for infra like this:

• This is a reminder that "already Rust" doesn't mean "already fast" — HuggingFace Tokenizers and tiktoken are both compiled, optimized libraries, and GigaToken's gains came from domain-specific techniques (SIMD pretokenization, pretoken caching) layered on top of that baseline rather than a language-level rewrite.

Related on explainx.ai

• Bun's Zig-to-Rust rewrite — Fireship's take on infra performance
• Google's Frozen v2 TPU chip and Gemini 4 pretraining
• What is llama.cpp? Run models locally
• AI benchmarks — complete guide

Primary sources: github.com/marcelroed/gigatoken · fastokens (TTFT benchmark tool)

Benchmark figures reflect the GigaToken project's published README and author comments on Hacker News as of publication. Speedup ratios vary by hardware, tokenizer, and workload — run the project's own validation and benchmark commands against your specific setup before adopting in production.