Prefill eats the clock at long context

A heavy day for local inference: a 21-hour benchmark sweep upends the conventional wisdom on what makes models fast at long context, while Simon Willison ships a release mostly written by Claude Fable ahead of the July 7 price hike. Elsewhere, new benchmarks keep finding that agents fail at asking questions, not searching, and a KAIST paper puts hard numbers on the agent power tax.

Long-context benchmark: prefill is 94-99% of your wait, and KV head count beats parameter count

A 13-model sweep at 65K-128K context on an RX 7900 XT found that for agentic workloads with short outputs, prefill (prompt processing) dominates wall-clock time while token-generation speed is nearly irrelevant. The dominant architectural factor for long-context prefill was KV head count, not parameter count: a 9B model with 4 KV heads ran 4.4x faster at 128K than a 15B model with 8 KV heads. Mamba2 hybrids (Granite-4.0-H-Small) held near-flat prefill scaling, and F16 KV cache beat Q8/Q4 quantization by 20-53% on MoE and small dense models due to dequantization overhead.

Why it matters: If you deploy local models for tool use or coding agents, this reframes the metric that matters: benchmark pp65K/pp131K, check n_kv_heads before parameter count, and stop reflexively quantizing your KV cache.

Simon Willison ships sqlite-utils 4.0rc2 mostly written by Claude Fable, for ~$149 of tokens

Willison used Claude Fable in Claude Code for web to do a final pre-release review of sqlite-utils 4.0, and it flagged five release-blocker bugs including a delete_where() call that never committed and poisoned the connection, silently discarding subsequent writes. Over 37 prompts, 34 commits and +1,321/-190 lines, the two reworked transaction handling; GPT-5.5 xhigh via Codex Desktop then caught two more P1 issues in db.query(). AgentsView estimated the unsubsidized cost at $149.25.

Why it matters: A concrete data point on cross-model review (having one lab's model check another's work) and on the July 7 'Fablepocalypse' when even Max subscribers lose subsidized Fable access and pay full API cost.

Mistral leans into sovereignty, promises open-weight summer model as Mensch attacks closed labs

In the wake of a Trump directive that pushed Anthropic to pull its latest models offline in some contexts, Mistral CEO Arthur Mensch published a LinkedIn broadside arguing that proprietary models give labs a 'front-row seat' to customers' business processes, urging companies to control their own weights. He confirmed a new open-weight model with July early access, and TechCrunch reports Mistral is raising ~$3.5B at a $23.15B valuation with ARR past $400M. Mensch conceded Mistral does not yet own the best language models but claims SOTA in voice, vision and document processing.

Why it matters: Mistral is Europe's only serious frontier contender, and its Palantir-style forward-deployed, sovereignty-first pitch is a genuine alternative model for enterprises wary of US-hosted APIs, even if Mensch is talking his own book.

DiscoBench: search agents don't fail at searching, they fail at asking

A benchmark from Tencent Hunyuan and Tsinghua (211 tasks, 463 ambiguous points) tested whether agents spot ambiguity and ask clarifying questions rather than plowing ahead. Even top models stayed below 50% end-to-end: Doubao Seed 2.0 Pro led at 43.1%, Gemini 3.1 Pro at 40.8%, Claude Opus 4.7 at 39.8%. Agents that searched then asked hit 93.4% success, while searching repeatedly but still guessing dropped to 51.9% (worse than guessing outright), and a warning prompt raised detection but barely moved end-to-end accuracy.

Why it matters: For anyone building deep-research or multi-step agents, the lesson is that more tool calls don't fix an underspecified query; the missing primitive is turning uncertainty into a user question.

KAIST puts a number on the agent power tax: up to 136x a simple chatbot query

A KAIST study led by Prof. Yoon Min-soo quantified the compute cost of tool-using agents, finding they make on average 9.2x more LLM calls than step-by-step reasoning, push response times up as much as 153.7x, and leave GPUs idle up to 54.5% of execution time waiting on external tools. An agent on a 70B model averaged 348.41 Wh per query. At a hypothetical 13.7B daily agent requests, data-center demand could hit ~198.9 GW, roughly half average US power consumption.

Why it matters: Agent orchestration overhead, not just model size, is becoming the dominant cost driver, and the idle-GPU-during-tool-calls figure is a direct argument for better scheduling and cheaper accelerators.

Better models, worse tools: newer Claude models fumble third-party edit schemas

Armin Ronacher reports that while hacking on Pi, newer Anthropic models (Opus 4.8, Sonnet 5) call his custom edit tool with invented extra fields in the nested edits[] array, causing schema rejections, while older models handle it fine. He theorizes the SOTA models were RL-trained to use Claude Code's built-in search-and-replace edit tools, degrading their ability to use custom harness tools. OpenAI's Codex has a similar story with its apply_patch mechanism.

Why it matters: If model training is optimizing for the vendor's own coding harness, third-party agent builders may need to implement multiple edit-tool variants and select per-model, a real portability tax.

Zig formalizes a no-LLM contribution rule, citing reviewer scarcity

Zig's Code of Conduct now bars LLM-generated or LLM-assisted contributions, covering code, prose, editing, translation, brainstorming and bug-finding. Coverage from Business Insider, TechSpot and The Register ties it to Andrew Kelley's comments that AI submissions waste scarce review time, with roughly 200 open PRs at the time. The framing is less anti-AI sentiment than a reviewer-capacity policy for a small systems-language project with a high correctness bar.

Why it matters: This is an early governance template: as AI shifts work from contributors to reviewers, more upstream projects will formalize provenance rules, constraining AI coding adoption by review economics rather than model quality.

Also worth a look

Browse previous days →