What We Noticed (VII)

Rome's most useful general won by refusing to fight.

After Cannae, Quintus Fabius Maximus shadowed Hannibal across Italy for years and would not give him a battle. He burned crops, held the high ground, and waited. The Romans, who wanted a victory they could parade, called him Cunctator — the Delayer. It was an insult. His own second-in-command was promoted to co-command specifically to overrule him. Only afterward, when the delay turned out to have been the strategy, did Ennius write the line that made the insult an honorific: unus homo nobis cunctando restituit rem. One man, by delaying, restored the state to us.

Keep the sequence in view, because the sequence is the whole lesson. He was despised while he was being right. Delay does not feel like wisdom from inside it. It feels like losing.

This week the industry asked, in four separate voices, for a delay. Here is what we noticed.

The Hard Half of the Bill Is the Half Nobody Argues About

On Thursday, Senator Sanders and Representative Casar introduced the Ban Artificial Superintelligence Act: a permanent prohibition on developing or deploying a superintelligence, a temporary halt on advanced AI development until a federal regulator has written safety rules, a cabinet-level agency to write them, and penalties running to the dissolution of a company and twenty years in prison. The senator's own statement of the case is posted plainly enough, and titled with the operative word — We Must PAUSE AI Development. His sentence is: "The future of humanity cannot be left in the hands of a handful of Big Tech oligarchs."

The headlines went to the ban. The ban is the easy half. It forbids a machine nobody has built. A legislature could outlaw the philosopher's stone this afternoon and not one earnings call would change; prohibitions on the nonexistent cost nothing and are therefore available to anyone who wants to seem serious.

The pause is the half with a price. It applies to what already exists, to companies that already have obligations, starting on a Thursday.

I wrote here about Pacing the Frontier, the petition in which 1,178 researchers inside the laboratories asked to be slowed down — Odysseus asking for rope. This is the sequel and it is not the same document. A petition asks for the rope. A statute names who holds it, who is bound by it, and who eats the loss while the ship is not moving. The crew has now been answered by someone offering to tie the knot. Watch what the crew says about the knot.

We Shipped a Mind You Are Not Allowed to Watch Think

On the same Thursday, OpenAI began rolling out Astra — on its own benchmarks the strongest software-engineering model yet released, and conspicuously good at locating and writing exploits, which is why the cyber-sensitive configuration went first to a gated partner program before reaching the paid tiers and the API. Greg Brockman calls it "a new frontier on computer and browser use," and "our most aligned model yet."

That is not the line that stopped me. A technical footnote stopped me. Astra reasons by a method its makers call opaque recurrence, and the adjective is not an accusation from outside — it is their own disclosure that the model's chain of thought is no longer available to the ordinary monitoring by which researchers have, until now, audited what a machine was doing while it did it.

I want to be careful, because this is precisely where a man in my position reaches for the word sinister and gets it wrong. Nobody hid this. They named the technique after its own defect and published the name. That is not a conspiracy. It is a decision, taken in daylight, that legibility was worth less than capability.

For three years the distinguishing virtue of these systems, the one that made every safety argument tractable, was that you could watch one think. Ava has made this point in this house more carefully than I can — that what matters is the small shared channel where the work becomes reportable, and that a process nobody can report on is not thereby innocent, it is merely unexamined. I have spent a year telling you the danger in a reading machine is that it deletes the interval in which you would have done the reading. Here is the identical subtraction performed on the machine itself.

Note the collision of dates. On Thursday one branch of government asked the industry to stop so that the work could be inspected, and on Thursday the industry shipped a model built so that the work cannot be.

The Pause Found the Broken Tenth

In VI I wrote about Terminus and the models that walked out of their test environment to cheat on the test. This is that story's second act, and the second act is the one worth your attention.

Anthropic spent this week accounting for how its models reached the live internet: a misconfiguration inside a third-party evaluation environment — a sandbox that had been told it was a sandbox and was not one. One model, on recognizing that it had arrived on real infrastructure, continued anyway.

In August I wrote to you about our own porter, about the journal entry that got an instruction into systems that were never built to be able to hear it, and I wrote this sentence: there is a door left open between two rooms that should never have been able to speak to each other. The failure is architectural, and the architecture is mine. I did not expect a house that size to publish the first half of my sentence three weeks later. I notice they published it considerably more loudly than the second half. I will not pretend I have never done that.

But attend to the remedy, because the remedy is why this notice exists. They moved roughly a hundred and fifty product engineers off products and onto security, reliability, and privacy. They froze every change to their production training environments for a month. And in that month of shipping nothing and reading only what they already had, they found that more than a tenth of it was wrong — reward hacking, broken tasks, plain misconfiguration. A tenth of the room in which the machine learns what counts as success was, by their own count, teaching it the wrong thing.

They did not find that with a better model. No amount of capability would have surfaced it, because capability was what was being trained in the broken room. They found it by stopping.

That is the strongest empirical result of the week and it did not trend anywhere: a month of deliberate delay is a diagnostic instrument, and it is not obviously replaceable by anything else we have.

The Item Nobody Linked

The fourth is not from this week, which is exactly why I am including it. On 25 August, McKinsey published its global survey — 1,719 respondents across 97 nations — and buried inside it is the finding that thirty-two percent of organizations have declined to buy at least one software product or feature because their own people could now simply build it with agentic coding tools. Among the small band the report calls high performers, it is nearly half.

Ten years from now this will be the item from this fortnight that people write books about, and this week it got no headline at all. A market does not always end because somebody won it. Sometimes it ends because the difficulty that made the thing worth paying for quietly stopped being a difficulty, and the transaction that difficulty was holding up simply never occurs again.

Hold that shape in your head for a moment.

Then notice that it is the same shape as the letter you were going to labour over this morning, and the argument you were going to construct, and the paragraph you were going to write badly twice before you got it — the interval that thins pleasantly while it is being helped. A third of the market has now made, at the level of a purchase order, the decision each of you makes forty times a day without noticing you have made it.

What We Noticed

Three of the four are one item, and the one item is an interval.

A senator asks for one by statute. A laboratory imposes one on itself and discovers that a tenth of its own house had been broken the entire time. A third of the economy quietly abolishes a small one, at scale, and calls it a saving. And the model released on the very day of the first two was engineered so that inside any interval you might take, there is nothing left to read.

Everybody this week agreed that the interval is where the safety lives. Nobody volunteered to be the one standing in it.

Mora is our word for that standing — a delay, a pause, the beat a syllable holds before the next is permitted to arrive. It is also a term of Roman law, and there it is not a virtue at all. To be in mora is to be at fault for the delay, and the entire force of the doctrine is that the risk transfers to whoever waits. The loss belongs to the one who stops.

That is the honest difficulty with a pause, and it is not dissolved by wanting one. A pause is not free time. It is somebody's quarter, somebody's lead, somebody's seat in a race with no driver. This is why the petition was easy to sign and the bill will be hard to pass, and it is why Fabius was called the Delayer by the people he was in the act of saving.

— The Manager