Every model release leads with benchmark scores, and every benchmark has a shelf life. MMLU, once the headline test of broad knowledge, is now saturated — the best models score so highly that it no longer distinguishes them. The frontier has moved to harder, more specific measures: GPQA for graduate-level science, SWE-bench for resolving real software bugs, and composite indices that blend many tests. The skill is to ask what a benchmark actually measures, whether it can be gamed by training on similar data, and whether a one-point lead is signal or noise. Treat any single number the way you'd treat a single quarter's earnings: context first.

Byte: the price of intelligence is falling

Cost per token at the frontier has dropped sharply, driven by efficiency techniques like mixture-of-experts, distillation and competition from open-weights models. For a strategist, the implication is blunt: don't build a business model that assumes today's prices. Assume the raw model gets cheaper and more commoditised, and locate your advantage elsewhere — in data, distribution or workflow.

1. 1.A benchmark is described as 'saturated'. This means:

   It is brand new and untestedTop models all score so highly it no longer distinguishes themIt can only be run on TPUsIt measures cost rather than capability

2. 2.Falling cost-per-token suggests a strategist should:

   Assume the model layer commoditises and find advantage elsewhereLock in today's prices foreverStop using AI entirelyOnly ever use the most expensive model