RC RANDOM CHAOS

Claude 4.7's new tokenizer quietly inflates session costs by 20-30%

· via Hacker News

Original source

Measuring Claude 4.7's tokenizer costs

Hacker News →

An independent measurement of Anthropic’s Claude Opus 4.7 tokenizer finds it consumes about 1.33x more tokens than 4.6 on real Claude Code content, and up to 1.47x on technical documentation - near the top of Anthropic’s stated 1.0-1.35x range. English prose and code take the biggest hit (chars-per-token dropped from 4.33 to 3.60 on English, 3.66 to 2.69 on TypeScript), while CJK, emoji, and symbols barely moved. The pattern suggests Anthropic shortened sub-word merges for Latin-script content while leaving non-Latin vocabulary largely intact.

The practical cost: an 80-turn Claude Code session that ran ~$6.65 on 4.6 now runs $7.86-$8.76, and Max-plan users hit their 5-hour rate limit proportionally sooner. Cold-start cache writes are also more expensive because model-partitioned caches invalidate on migration, and the new prefix being written is 30-45% larger.

Anthropic’s justification is tighter instruction-following - smaller tokens force attention over individual words. A 20-prompt IFEval sample showed 4.7 scoring roughly 5 percentage points higher on strict mode, driven mostly by one multi-constraint prompt. The effect is real but modest, and the sample is too small to isolate whether the tokenizer, weight changes, or post-training drove the improvement.

Read the full article

Continue reading at Hacker News →

This is an AI-generated summary. Read the original for the full story.