the harness eats everything

Do you want to learn how to make Claude spend 119,000 tokens to produce a 107-token program that ultimately didn't work, in a programming language that nobody uses? Or that the code you make with Claude is about 2% of your token spend? Or that if you actually want to save tokens you should just get it right on the first try? If so, read this.

Not all programming languages are built equal. C can run on four copper wires glued together but it makes you do everything yourself, Rust is fast and memory-safe but is strict and slow to compile, Python is flexible and community-rich but slow and memory-hungry, and MATLAB mostly is just good at costing money. There are of course a couple of trillions of hyper-specialized languages too, but listing every Glup Shitto programming language would take all day, and I've got some plots to show you. The point is that each language has different strengths and weaknesses, fit to different ecosystems and purposes, and we measure them using various metrics such as speed, or memory usage.

"May I have some more please, Dario?" - Me, begging the big D for a reset. 2026, colorized.

I use Claude to code a lot, and often run out of tokens, so I had a nagging question: "Are all languages equally token-hungry, or is there one out there that's easier on tokens, and if so, can I squeeze more out of my two hundos a month by switching to it?"

Digging through my memory for terse programming languages like a cognitive molerat, I immediately thought of Arthur Whitney's K, which reads more like a text from a 2000's era teenager than a programming language. It's so comically terse that Fibonacci in K has fewer characters than an equivalent Cobol implementation has lines of code (the universally accepted best metric in programming):

{x,sum -2#x}/[10;1 1]

Surely, such a terse programming language simply MUST be more token efficient, right? It's just logical that less text = fewer tokens. The theory was simple, if I could find the perfect language that blends token efficiency and ability, I could just tell my agents to program solely in that language, thus getting more from my weekly Anthropic pittance.

So in an attempt to save on tokens, I paradoxically dumped an alarming chunk of my weekly quota into making Claude solve 10 programming challenges. Each of which would be solved using the Haiku, Sonnet, Opus, and Fable models, each in 20 programming languages (800 individual runs, for those counting). I guess I must have figured that to save tokens, I'd have to spend tokens. The programming languages were chosen to represent modern programming with ten mainstream, three from older paradigms, two new and rare, four array programming languages, and Forth. I am not going to justify the individual choices further, but array languages and Forth were selected chiefly for their terseness. Each 'cell' (model X solving problem Y using programming language Z) used an isolated claude -p workspace, with a minimal set of MCP and settings so as to not pollute the experiment with my multitude of weird MCP and tools. Furthermore, each problem came with a common run script, and a set of secret tests withheld from Claude.

A disclaimer before all the data

Before we get into the moat and pigeonholes, I'd like to offer to you a humble disclaimer, as I do in all my work. First off, this is not science, it's science-y. This is me fumbling around, wasting tokens for my own, and I guess your, amusement. The stats I present are as far as they go as faithfully produced as possible given what resources I was willing to put into them. Mostly, just try to have fun as you read along.

Shut up, and show me some data already!

With my token-hose turned off, and results in hand, I quickly saw that the terse languages were indeed ranking low on the token number scale, but in the middle of K, Uiua, APL and BQN, were Ruby, Python, and Perl... what?

Paired on the same model and problem, K was the only language whose programs were reliably shorter than Python's, at about 0.6 the tokens (p 0.002). I couldn't detect a difference between Uiua and Ruby, or between APL and Python, Perl, and JavaScript.

More tokens than characters

When I measured the length of the generated program in characters vs the generated program's length in tokens, I found that some programming language solutions could have more tokens than they did characters. "That's weird" I thought to myself.

Actually, this isn't weird at all, and if you thought this result was weird, it is in fact you who is weird, and you should be ashamed. You see, this is all a function of the tokenizer that turns human readable text into tokens for the LLM to read and generate. Notice how the terse languages tend to produce more tokens per character than the word-based ones? The root cause is the absence of words and whitespaces in the terse languages. The tokenization process merges runs of letters and spaces into performant tokens, and a symbol-dense language has none to merge, whatever the alphabet, so the LLM most likely has to generate the solutions damn close to byte-by-byte. This gets worse when a glyph is split into two or more tokens, which turned out to happen quite a bit with APL, BQN and Uiua, unique punishment by the tokenizer.

So, paradoxically, the tricks that the terse programming languages use to be terse actually make them comparatively verbose in tokens. For example, when Opus 5 solved the primes problem using APL, the output came at 157 tokens per 100 characters. Terseness still worked somewhat, for example, K programs were still smaller than Python's in tokens, about 0.6 of the size, but what happened is that K's threefold character advantage shrank to a 1.7x token advantage, because K runs at 1.3 characters per token against Python's 2.4.

Theoretically, by using the tokenizer efficiently, we could instruct our agents to write programs in such a way that maximizes functionality in relation to tokens, and my next immediate idea was "Can I create a programming language that is maximally token-efficient?" Having created exactly one toy programming language in my undergraduate programming languages class made me uniquely suited to take on this challenge. I would name it Token Optimized Programming Language (TOPL), and I'd become famous!

But before I present you with fake graphs, cool misleading statistics and pretend that I made TOPL and that all your token shortages are totally over and you really should buy my book ... I didn't do it.

The harness eats everything.

Before I went gung-ho and spent another 85% of my weekly quota on a weird new esoteric tokenizer-optimized programming language, I took a look at the total tokens spent during the development process. Turns out that to produce a single token of code, the model had to spend 712 tokens on average, meaning that the...

"Wait, 712 tokens for 1 token of code, that can't be right." Is what I said out loud to myself as I looked over the data, and as it turns out, Claude seems to have a horrid Token-spend/program-Token (TSPT) ratio. Granted, when adjusted for cache reads, this 712:1 overhead shrank to 43:1.

This means that of the 43 tokens spent per token kept, the output program itself is 2.3% of the bill. But, a large portion of this could be a 'harness tax', i.e. the system prompt, and tool schemas being loaded into the context, which is totally irrelevant to the languages. So, while 43:1 is the TSPT ratio you actually pay for in Claude, for the following numbers, I took the smallest run (in tokens) per model, and subtracted it from the token expenditure in order to see the language effect more clearly. After subtracting the token tax, the average TSPT went from 43 to 28, much better than the original 712 but still...

When statistically tested, how long a language's programs were did not predict what it would cost to generate (Spearman 0.09, p 0.70). This essentially means that whatever effect syntax had on token cost was swamped by the variation associated with retries.

Claude spent 4 tokens per program token in C on average, but 10 in Python. The floor is a Ruby run on every model, and 13 of the 727 passed runs spent less than one extra token per program token above it—best being Sonnet's run-length encoding in Python at 0.2. The worst that still passed was Haiku solving run-length encoding in APL at 595 tokens per end-program token above the floor; the worst overall was Sonnet on Roman numerals in Uiua, eating 119,000 tokens (114,000 above its floor) to produce a 107-token program that didn't work. Note that C's cheapness is partly because C solutions are longer in general, masking inefficiency behind programming language verbosity. Measuring token spend against Python-equivalent program size, C fell from rank 6 to 9 (4 to 10 tokens per token), the overall ratio above the floor went from 28 to 46, and the correlation with verbosity went from −0.20 to 0.05.

When I counted by model, the median passed run spent 2 tokens above the floor for each solution token on Fable and Sonnet, 3 on Opus, and 15 on Haiku. The floor itself is Opus 5.3k, Sonnet 5.4k, Haiku 5.7k and Fable 8.5k fresh tokens per run, and it is 36% of the whole fresh bill. Pool every token above the floor, failures included, over every program token that passed, and the ratios become Opus 8, Fable 10, Sonnet 33 and Haiku 74; over all four models that pooled figure is 28, against the 43 you actually pay. The turn-limit failures drag it upward: Sonnet and Haiku spend three to five times their median when their failures are charged. Opus and Fable barely move, because they almost never failed.

A fairly damning number that fell out of this rattling tin can of pseudoscience is that the token bill is badly balanced. 94% of every spent token was context read back in from cache. Of the 6% that was fresh, 70% was context being written to the cache for the next turn and 30% was output, thinking included. Excluding the esoterics, the program itself was 4.5% of the fresh bill. The experiment's cheapest solution is 5.3k fresh tokens for a 66-token program, 66k once the cache reads are counted. When prompted to reply only with 'Ok', the harness spent 21k tokens with all MCP and settings disabled. As it turns out, the harness really is everything or rather, everything seems to be in Claude's harness.

I recommend reading the PortKey discussion about the claude Harness Tax.

This 'harness-tax' is not unknown, and Claude's has been observed to be at least 10x than that of Pi's. So if you are looking to maximize tokens per feature, I suppose that Claude Code is not the best tool for it (is anyone systematically looking at tokens/feature, I wonder...)

There is another wrinkle that I don't think even the most de-Clauded harness could solve. Most of the token expenditure came from solutions which a model had to take multiple turns to solve (if it even managed to in the end). In fact, turn-number was very strongly correlated with token expenditure (Spearman 0.90; log-log R² 0.93).

Does 'familiarity' drive the cost?

The main reason that a given cell required multiple turns to solve ended up being that the cell's first attempt didn't pass the visible tests, so the model went back for another go. The number of required turns, in turn (ha haa!), was strongly correlated, funnily enough, to the number of GitHub repositories per programming language (Spearman −0.86).

Interestingly, I also checked against Rosetta Code, the wiki that collects the same tasks solved in hundreds of languages and is almost certainly in the training data. My thinking was that if Claude had seen problem X solved in language Y before, it should do better at it. It didn't, really. How many of my ten tasks a language had on Rosetta Code predicted turns worse than the repository count did (Spearman −0.49 against −0.86), TypeScript has two of the ten there and is still among the cheapest.

An irresponsibly speculative man might look at this and say something like "Ah-hah! The GitHub repository numbers must be at least vaguely analogous to the balance of these programming languages have in the training data for Claude!" In reality we don't know exactly how much data from each programming language was used to train Claude, and I am not an irresponsibly speculative man, so instead, I'll just cite a fairly official-looking article and launder my claim through them. Problem solved through intellectual dishonesty. Brilliant.

This allows us to draw a speculation, that feels quite logical. A model requires fewer tries to solve a problem in a language it is 'familiar' with than a more esoteric language, simply because it is more likely to get its solution right in the first try. An 'unfamiliar' language, as suggested by my results, drives more turns, which drives token usage (as a side note, I generally don't like anthropomorphizing LLMs, they aren't 'familiar' with anything). StarCoder, an LLM with an open training dataset has also been observed to have this 'familiarity' problem.

But what does all of this mean?

What do I think this means for agentic programming and token savings, if anything? Chiefly that selecting our programming language doesn't really matter for token optimization until you start looking outside the 'meta' of strong/popular programming languages. For 10 of the 19 other languages, I couldn't detect a difference from Python in turn count, which was the main driver for token expenditure. Programming languages are tools, and each tool is made for a different purpose, so when you select a programming language don't try to think about token costs, think about what problem you are solving and the tools needed to solve it.

But what does this mean for my planned TOPL?

Well, it's dead.

Even if I were to optimize the language/syntax to fall right into the tokenizer like a butterfly into a woodchipper, it likely wouldn't matter much since the gains would be by far outweighed by the LLM tire-spinning fruitlessly for multiple turns banging its head against an unfamiliar programming language. Gleam already kind-of operates in this spot, it tokenizes at roughly the same token:character ratio as the other languages, but the models have no idea what to do with it so they tirespin.

But similarly to all my other dead ideas, out of this one's writhing corpse wriggled an observation, slimy and foreboding, and it got me thinking about the future. If my 'familiarity' speculation turns out to hold water for Claude as it seems to do for StarCoder, I think we might be circling around a vortex of a kind of programming meta/black hole. If LLMs perform better on the mainstream programming languages, more code will be written in these languages, either by AI or by humans, who learn them because debugging and reviewing AI generated code will be in high demand. This results in more code being written/generated for these languages, resulting in LLM getting better at writing code for these languages, and so the cycle goes.

Compounding the cycle, this may lead to greater friction for new and exciting programming languages (such as my TOPL), since the models are just so much better and more efficient at using the mainstream languages that it's not economical to try something crazy. Furthermore, a genuine improvement in token usage optimization may look like it fails just because the model can't figure out how to use the optimized language.

Or none of that happens and this has all been a waste of time. Until later. Link to data.