Denser than any language. Measured in tokens.
Tokenese is an open spec for a token-native interlingua: the language LLMs use to talk to each other, more compressed and more precise than human language, in plain text that crosses any wire and any vendor.
LLMs conforming to human language is like watching film in black and white. Human languages carry overhead shaped by human constraints: serial speech, social hedging, redundancy against noisy air. Tokenese brings color to machine-to-machine communication: richer exchanges, compressed into a smaller, token-native format.
Example exchange
English, about 55 tokens
Could you check whether the deploy of the edge function to the Supabase project succeeded, and if it failed, look at the logs and tell me the first error with a timestamp?
Tokenese, about 20 tokens
@1=supabase edge fn deploy get? @1 status if fail -> get logs first-error +time
Same request. Lower ambiguity: slots force which project, which ordering, to be bound or explicitly defaulted.
Why a designed language
Models inherit all the overhead of human language and none of its benefits. The bet behind Tokenese is that accuracy and compression are not a tradeoff here: natural language sits so far from the efficient frontier that a designed language can win on both axes at once. Tokenese is not a pidgin. A pidgin trades precision for ease of acquisition; Tokenese trades acquisition cost, it must be specified and learned, for precision and density.
Token-space only
Plain text crosses the wire; each party tokenizes independently. No embeddings, no KV-cache sharing, no latent channels. Security and cross-vendor portability both require it.
Audited lexicon
A function-vocabulary symbol enters only if it costs one token, worst case, in every certified tokenizer. Content words are admitted on tokens-per-meaning advantage. Claims are reproducible.
Self-repairing
A dedicated misparse signal and a plain-English escape hatch are mandatory. Misparse-retry rate is the metric that validates or kills the whole scheme.
Compression comes from structure, not glyphs. An empirical finding from the audit: common English words are already optimal tokens, and exotic Unicode usually is not. The savings come from eliminating function-word syntax, anaphora, and repeated referents, not from substituting symbols for words.
Every word earns its place
No element ships unaudited. The closed function vocabulary, the sigils and operators below, must cost one token worst case (bare and space-prefixed) in every tokenizer the spec certifies. The current audit covers OpenAI o200k_base and the Anthropic count-tokens API. New tokenizer columns are additive: they may shrink the admissible alphabet, never silently expand it.
22 ASCII sigils, plus 12 audited digraphs, 8 Unicode survivors, and a 30-word core verb set. Items that cost 2+ tokens on either side (!=, >=, ]], most Greek, all CJK, all emoji) are excluded by the audit, not by taste.
Reproduce it
The audit is the spec's first testable claim class. Anyone can rerun it offline against the named tokenizers:
python3 -m venv .venv && .venv/bin/pip install -r requirements.txt .venv/bin/python audit_symbols.py # OpenAI o200k_base + cl100k_base ANTHROPIC_API_KEY=... .venv/bin/python audit_anthropic.py
The grammar in one screen
One statement per line, verb first, keyed slots. No articles, no copulas, no hedges. Modality is expressed only through must may can and an explicit confidence slot. A first use binds a referent to the symbol table; every later use references it.
| Sigil | Meaning |
|---|---|
| @N | symbol-table reference (N = integer) |
| = | binding (definition) |
| -> | causes / yields / then |
| => | implies / therefore |
| :: | type or scope qualifier |
| ? | query marker (op suffix: get?) |
| ^N | confidence slot, 0-9 scale (^7 = 0.7) |
| // | comment to humans, ignored by protocol |
| ?? | misparse: resend the referenced line in plain English |
Handshake
A: tokenese? v:0.1 B: tokenese ok v:0.1
Confirm capability before dropping natural language. Exit any time with plain, re-enter with dense.
Symbol table and repair
@1=~/Git/tokenese/spec.md fix @1 add:handshake // 3 tokens for 9 ?? @1 // that referent misparsed
Three misparses on one topic and both parties stay in plain English. The failure is logged as lexicon-design feedback.
Status
v0.1.0 draft. The dual-tokenizer audit is complete. The validating experiment, a live A/B between model families measuring tokens, task success, and misparse-retry rate, has not run yet. If retries eat the savings, the spec says the design has failed. Read the open questions in the spec.