The developer published ttok 1.0 after discovering an unexpected default tokenizer when upgrading from the previous 0.4 release. During the upgrade procedure — described by the author — they ran uv tool upgrade ttok and piped a file into the new version, then noticed the tool defaulted to the GPT‑4 tokenizer even though they considered the GPT‑5/GPT‑6 family should be the default.
Changing the default tokenizer provided a reason to finalize and ship the 1.0 release.
GPT‑6 tokenizer compatibility remains unconfirmed by OpenAI
OpenAI has not officially confirmed whether GPT‑6 uses the same tokenizer as the GPT‑5 family. The topic has generated some friction in discussions (the author notes an "angry issue"), so a formal confirmation from OpenAI is still pending.
The developer points to a commit by William Liu reporting an experiment: seven GPT models (5.5, 5.6 Sol/Terra/Luna and 6 Astra/Sol/Luna) produced the same token count — 44,794 tokens — on the tested corpus and matched each other on all 31 fixtures used for comparison. The commit also notes that GPT‑6 introduces no input-count change on that corpus.
Why this matters
Changing the default tokenizer can affect systems tuned for specific models, token counting, and cost estimates. If GPT‑5 and GPT‑6 share the same tokenizer, migration and compatibility are simpler; if they differ, mismatched defaults could lead to incorrect token counts or other integration issues.
The ttok 1.0 release is therefore both a practical fix and a response to the observed defaulting behavior, while definitive technical confirmation about GPT‑6’s tokenizer awaits an official statement from OpenAI.
