OmniToken: Pure-Go OpenAI tokenizer for Go: ~15.84x faster than tiktoken-go , near OpenAI Rust performance

API
, , ,
1

I built OmniToken, a pure-Go tokenizer library for local token counting, encoding, and decoding in Go services.

It supports OpenAI-compatible BPE encodings:

  • cl100k_base
  • o200k_base
  • o200k_harmony

Why I built it:

I wanted local prompt sizing, context-window planning, and cache-boundary analysis in Go without requiring Python, Rust, CGO, or a hosted API call.

Benchmarked on Windows 11 / Intel i7-1250U / Go 1.24.2:

  • CountTokens vs tiktoken-go: 15.84x faster geomean
  • EncodeOrdinary vs tiktoken-go: 13.09x faster geomean
  • Decode vs tiktoken-go: 2.29x faster geomean
  • CountTokens vs OpenAI Rust tiktoken: 0.96x, near parity
  • EncodeOrdinary vs OpenAI Rust tiktoken: 0.75x, competitive pure Go

The CountTokens path avoids materializing the token slice and hits 0 B/op / 0 allocs/op on supported OpenAI BPE workloads.

Correctness:

For supported OpenAI ordinary-text encodings, OmniToken includes parity tests, including a deterministic 50,000-case reference corpus.

It also includes:

  • CountTokens / EncodeOrdinary / Decode APIs
  • prompt-cache alignment planner
  • benchmark harness comparing against tiktoken-go and OpenAI Rust tiktoken
  • optional adapters for Gemini, Llama 3, Mistral, Hugging Face tokenizer.json, OSS SentencePiece models, and Anthropic message counting

The repo is public on GitHub under:

ron2111 / omnitoken

The Go package is available on pkg.go.dev as:

github.com/ron2111/omnitoken

I’d appreciate feedback from Go developers using OpenAI APIs:

  • Are the model mappings useful?
  • Any tokenization edge cases I should add?
  • Should special-token-aware encode be next?
  • Would streaming token counting help your workflows?
2

You cannot really do streaming token counting, especially were it multibyte Unicode. You’d get a count of 5 tokens for " Pennsylvani" but then “stream” a byte longer, the final letter “a”, and it becomes 1 token. An estimate would only be an estimate, usually incorrect.

There’s pretty much a single token encoder being used by OpenAI models these days. The large exception is embeddings, where still cl100k_base is used. A token encoder is of value to have for embeddings length measurement, because the input is limited, and 2000 of them can be included in a request for only one overage to fail. A pattern there is encode to token numbers, create some chunks and overlaps, and then stringify again.

To answer the burning question of others (or only me): the token dictionaries for “base” are compiled-in and are in the repo. They are the same format as the blob downloads that tiktoken employs but doesn’t include - getting up to 7 bytes x 100000 wasted for the second half of the file by including an index number that can be obtained positionally.

ID 200018 has two aliases: base o200k_base already has <|endofprompt|> = 200018, and the Harmony generated reserved range also creates <|reserved_200018|> = 200018.

In my using AI to inspect whether the recursion can go nutty (like tiktoken), its harmony implementation is noted:

Important mismatch vs tiktoken

The attached repo does not include the <|reserved_200018|> alias that current tiktoken creates. It keeps only:

<|endofprompt|> = 200018

This is probably intentional or forced by its engine design: newEngine(...) rejects duplicate special-token IDs. Since tiktoken has two string keys pointing to ID 200018, this Go repo cannot represent that exact alias set without changing its duplicate-ID rule.

For ordinary text counting, this alias difference usually does not matter. For exact special-token compatibility with tiktoken, it does matter.

Bigger limitation in the new repo

The repo stores Harmony special tokens in the decoder, but its public API is still ordinary-text-oriented:

EncodeOrdinary(text string)
CountTokens(text string)
Decode(tokens []int)

So if you pass the literal string:

<|start|>

it does not encode as special ID 200006; it encodes as ordinary text pieces. I verified locally that EncodeOrdinary("<|start|>") returns five ordinary tokens, not one Harmony control token.

That means the repo can decode Harmony control IDs into readable strings, and it can count ordinary text with the same BPE as o200k_base, but it is not enough by itself for exact gpt-oss Harmony prompt counting unless the caller separately accounts for inserted special tokens. A real Harmony renderer would need to construct token sequences with IDs like 200006, 200008, 200005, etc., not just concatenate their printable marker strings and call EncodeOrdinary.

Also, you’ve got a go 1.24 line, but it seems to work in 1.23 okay.

3

Thanks, this was very helpful.

You were right on the Harmony special-token gap. EncodeOrdinary intentionally treats marker strings like <|start|> as ordinary text, matching tiktoken’s encode_ordinary behavior, but OmniToken needed an explicit special-token-aware path for exact Harmony prompt accounting.

I fixed that in v0.1.5:

  • EncodeOrdinary keeps literal ordinary-text behavior.
  • Encode(…, EncodeOptions{}) rejects known special markers by default.
  • Encode(…, EncodeOptions{AllowAllSpecial: true}) encodes allowed special markers as special IDs.
  • CountTokensWithOptions follows the same policy.
  • o200k_harmony now includes the <|reserved_200018|> alias for ID 200018.
  • Decode canonicalizes ID 200018 to <|endofprompt|>.

On streaming: agreed that summing counts over arbitrary chunks is not exact. BPE merges can cross chunk boundaries, and partial UTF-8 makes that even easier to get wrong. Your Pennsylvania example is a good illustration.

I would phrase the limitation slightly differently tho: exact streaming token counting is not impossible, but it cannot be stateless. A correct streaming counter needs to retain boundary state / a rolling suffix buffer and only finalize tokens once future input can no longer change them. OmniToken does not expose that API yet, so current counting APIs are full-buffer APIs.

Also lowered the root module minimum to Go 1.23+. The benchmarks were still run on Go 1.24.2, but the root package itself does not require 1.24.