A unit on bending the curve.

Throughout 'Machines of Loving Grace,' Dario Amodei's October 2024 essay on what powerful AI might produce, the word 'we' seems to do three distinct jobs simultaneously.1 'We will be able to cure most infectious diseases' is one scope. 'We will need to fight for this outcome against those who would concentrate power' is another. The lab's own voice, which is bounded and specific, is a third. The gap between the civilisational 'we' and the actual room where AI decisions get made is the unit's central question, and is not named directly.1. The three 'we' referents appear sometimes within paragraphs of each other: lab-we (bounded, specific to the organisation), civilisational-we ('we will cure most infectious diseases'), and rhetorical-we ('we will need to fight for this outcome'). Amodei's introduction explicitly flags the risk of AI lab CEOs sounding grandiose, which is itself an interesting feature of the essay.

This essay is an attempt to answer one question: what would it actually take to shape AI's future toward the good outcome? BlueDot Impact's AGI Strategy cohort assigned nine readings to that question. Taken together, they are not a debate so much as a set of partial answers to the same problem. I'll work through six constraints, in order.

§01The prize

Start with what makes the question worth asking. 'Machines of Loving Grace' is the most expansive, carefully-argued optimistic vision any frontier lab author has published. Its central claim is if alignment is solved, powerful AI compresses decades of progress in biology, mental health, economic development, and geopolitics into a much shorter window. Not 'AI will be useful' — that's already true — but something structurally different: a 'country of geniuses in a datacenter,' running, in his projection, at ten to a hundred times human speed, taking on the problems that have resisted human intelligence for generations.

The framework Amodei uses to make this claim is a useful analytical tool. He calls it marginal returns to intelligence: for any task, ask how much faster it would go with vastly more intelligence, holding everything else constant. The answer varies wildly by domain. Writing code is highly intelligence-bottlenecked since almost all the constraints are cognitive, and better reasoning produces better code almost directly. Drug discovery is intelligence-bottlenecked in its early stages (hypothesis generation, target identification, experimental design) but time-bottlenecked in its later stages, where Phase I, II, and III trials run on human biology that cannot exactly be parallelised beyond the population that is willing to participate. Building democratic institutions is barely intelligence-bottlenecked at all. We know quite well what good institutions look like, so the bottleneck is political will, historical contingency, and coordination problems that accumulate over decades.

This distinction — between intelligence bottlenecks and everything else — is a good tool for evaluating the grand claims. 'AI will transform biology' needs to be evaluated differently from 'AI will transform geopolitics,' because biology and geopolitics are differently bottlenecked. 'AI will double human lifespan' hits the clinical-trial constraint, since no amount of intelligence shortens the time it takes to run a ten-year longevity trial, because the biology is the experiment and it runs in real time.22. Amodei acknowledges the experiment-time and clinical-trial constraints and then claims 10x compression in 5–10 years anyway, via massive parallelism.

The diagram is a way of taking the vision seriously enough to ask which parts of it are actually delivered by more intelligence and which parts require different inputs entirely. Drug discovery and mental health are heavily intelligence-bottlenecked in their early stages — real wins, real acceleration. Economic development and geopolitics are mostly not. The honest version of the prize is domain-specific: powerful AI delivers a lot in some areas and little in others, depending on what is actually in the way. The prize is real. Now the question is what stands between here and there.

§02The structural obstacle

The first thing standing between here and the prize is a fact about markets: pure commercial incentives cannot produce a net good AI outcome. IFP names it explicitly — beneficial AI applications are undersupplied by markets. Defensive technologies receive roughly 2% of AI research investment and about $100 million a year globally. The other readings imply it: Toner supports transparency regimes and third-party audits that markets don't generate; RAND assumes state-level coordination as its baseline; Amodei treats market forces as background to be steered from outside. Even the most market-friendly voice in the unit closes his essay by calling for collective decisions about 'broad bounds' — admitting, without quite naming it, that the market won't set them.

The visible debate in AI circles — optimists versus pessimists, accelerationists versus doomers, tends to obscure this near-consensus. Both sides agree that market incentives alone won't deliver the prize. The actual disagreement is about who should fill the gap, through what mechanism, and on what timeline. For this, the unit contains five competing answers:

1. The US government as strategic actor (IFP, Amodei): use supply-chain dominance, R&D funding, and export controls to shape which capabilities are built first and who gets access.
2. Multilateral dialogue (RAND): build institutional channels between great powers to manage AI competition without triggering the instabilities that arms-race logic produces.
3. External regulation with real teeth (Samuel): change the incentive structure across the whole industry; taxation, liability, and legislative instruments rather than voluntary commitments.
4. Reformed internal governance (Altman, implicitly): labs themselves can develop governance structures — benefit corporations, long-term benefit trusts, internal safety boards, that resist market pressure over time.
5. Emergent, decentralised dynamism (Toner): resist the urge to pick an end-state; build the conditions under which many actors, many approaches, and many feedback loops compete, and no single one locks the outcome in.

None of these is wrong, exactly. A multilateral dialogue can coexist with domestic regulation and reformed lab governance. But they compete for political bandwidth, institutional energy, and the authority to set the terms — and there is a deeper problem underneath, which Samuel names more precisely than the others. The structural pressure on safety-first commitments comes from competitive dynamics, capital dependency, and regulatory ambiguity — forces that operate regardless of what a given lab believes or says, independent of anyone's intentions. Any governance answer that relies on voluntary commitment has to explain why the structure won't erode it.

// pullThe honest question isn't 'why does market pressure win?' It's 'what institutional design is strong enough to resist it?'

Samuel's piece is most valuable not as a critique of any specific actor but as a problem specification: what institutional form allows a safety-first lab to remain safety-first under market conditions that weren't designed to support it? She reaches toward 'government regulation' without specifying which instrument, at which level of government, with what enforcement mechanism. That vagueness is a weakness, but the question is the right one to be asking. Here's the most concrete answer in the unit.

§03What shaping actually looks like

The Institute for Progress's contribution is partly about the market-failure diagnosis, but its most original argument is about path dependence. The same general AI capabilities can be channelled toward defence or offence, toward beneficial or harmful ends, depending on what gets built first and what infrastructure accumulates around it. The order of capability development matters as much as the eventual capability set.

// pullThe order of capability development matters as much as the eventual capability set.

This reframing is important because the question isn't whether AI is good or bad in the abstract, the question is: given that powerful AI is coming, which capabilities should arrive first? The answer depends on the 'jagged frontier,' which is the observation that AI advances unevenly across domains. It has advanced enormously in language and code generation, considerably in protein structure prediction; less so in reasoning about physical systems, relatively little in autonomous multi-day task completion in complex real-world environments. At any given moment, the frontier is jagged, and the shape of that edge determines what is and isn't possible to shape. Combine this with the marginal-returns framework from §01 and you get a specific policy agenda: accelerate in domains that are intelligence-bottlenecked and defensively oriented, and proceed cautiously in domains that are intelligence-bottlenecked and offensively oriented. Cybersecurity is roughly symmetric where AI can help defence about as much as offence. Biology is not: at the frontier, AI-assisted biodesign poses near-term offensive risks that biodefence infrastructure isn't yet equipped to match.

IFP names three historical playbooks for how this kind of deliberate shaping has been done before:

1. Nonproliferation (nuclear weapons): restrict access to dangerous capabilities from the outset via export controls and international treaty regimes. In AI terms: compute and chip export controls, chokepoints the US holds via TSMC's reliance on US-origin tools and IP, and ASML's EUV scanners — though ASML is Dutch, so meaningful export controls require Dutch and Japanese coordination.
2. Selective acceleration (Human Genome Project): public investment in a specific beneficial application, built ahead of and to a higher standard than private alternatives would produce. In AI terms: AI-assisted drug discovery, pandemic preparedness, climate modelling.
3. Defensive acceleration (Operation Warp Speed): crash programme to build defensive capabilities before an offensive threat has time to mature. In AI terms: AI-assisted cybersecurity, interpretability research, biodefence.

The key is the historical specificity. IFP isn't arguing from first principles that shaping is possible, it's showing the US has done it before, in three different modes, with different instruments. The case studies drag the 'can we shape this?' question out of pure hypothetical and into documented precedent. The limiting assumption that 'US shaping' and 'good shaping' are the same thing is left mostly implicit. The mechanisms IFP wants: compute restrictions, supply-chain leverage, export controls. All of these concentrate decision-making authority in whichever executive branch is in office. The governance of the shaping mechanism is its own open problem. The mechanism works domestically. The international version is harder.

§04The international complication

The RAND paper targets MAIM — Mutual Assured AI Malfunction — the proposal that states should deter AI monopoly bids via the credible threat of preventive infrastructure sabotage. RAND endorses the two less aggressive pillars of the underlying proposal (nonproliferation and managed competition) and attacks only the third.

MAD — Mutual Assured Destruction — worked as nuclear deterrence because a disarming first strike was made infeasible by survivable second-strike capability: submarine-launched missiles, hardened silos, dispersed bombers that no first strike could reliably destroy. The stability came from the inability to destroy an opponent's retaliatory capacity, not from a threat to try. MAIM inverts this logic: it asks states to develop first-strike capabilities and use the threat of deploying them as deterrent. That is preventive war doctrine, not MAD — a distinction with a bad historical record.3 There is also a technical problem: 'AI infrastructure' has no clean trigger point. A nuclear launch is a discrete event. There is no equivalent threshold that would clearly activate a MAIM response, which means every actor is guessing at the other side's red lines, and guessing wrong means catastrophe.3. RAND also notes that AI development is distributed across cloud providers, research institutions, and hardware manufacturers in ways that make comprehensive sabotage technically unavailable short of actions that would look like acts of war. The kill switch MAIM assumes does not exist in the form required — and trying to build one would itself be destabilising.

Three papers later in the same unit, Amodei proposes an 'entente strategy': the democratic coalition should leverage its AI advantage to set global terms — offering AI benefits to countries that accept the coalition's norms, withholding from those that don't. He frames this via 'Atoms for Peace,' a carrot-heavy programme from a different Cold War. The framing is different from MAIM: carrots before sticks, coalition-building rather than sabotage threat, more cooperative in tone. But RAND's structural critique — that any framework using AI advantage as geopolitical leverage generates first-mover incentives and pushes rival development underground — applies to the genus, not just to the most aggressive species. Both proposals require the democratic coalition to maintain a decisive enough lead that rivals comply; both create pressure for rivals to race before that threshold is crossed. The two papers never engage each other. They are on the same reading unit, responding to the same underlying problem, with proposals that are closer in structure than their framing suggests.

The diagram makes visible something the unit leaves implicit: the readings that are most in tension with each other are the ones that don't engage each other. The optimism cluster (top-right) and the risk cluster (bottom-left) are structured responses to the same question — how to get from here to the prize — and they'd sharpen each other considerably if put in direct dialogue. Even with the mechanism from §03 and an approach to the geopolitics, there is still the question of what future we are actually aiming for.

§05What kind of future, exactly?

Rutger Bregman's chapter from Utopia for Realists looks like the odd one out in this unit. It is about whether affluent societies can still imagine a better future — not about AI strategy specifically. But it has the sharpest analytical frame in the unit for reading the vision essays — and for understanding why the geopolitical complications in §04 are harder than they look.

Blueprint utopias are rigid. They have an endpoint, a destination, a specific vision of what the good society looks like. Communism was a blueprint utopia — not because the vision was entirely wrong, but because the blueprint mode led to treating any deviation from the plan as heresy and any obstacle as evidence of sabotage. Blueprint utopias fail in a specific way: the map doesn't match the territory once you arrive, and the project of arriving tends to justify means that corrode the ends it was meant to serve.

Guidepost utopias are directional without being prescriptive. Thomas More's original Utopia was a guidepost: a critique of the present dressed as a picture of somewhere else. The Enlightenment was a guidepost project — more freedom, more reason, less arbitrary authority — without specifying exactly what the free, rational society looked like. The direction matters; the specific endpoint doesn't, and locking it in too tightly is its own kind of error.

Most AI strategy writing tends toward blueprint mode, and the most ambitious essays tend toward it most. 'Theory of victory' thinking — Toner's term for the habit — is blueprint thinking: there's an endgame, and the strategy is the path to that endgame. The most expansive vision essays in this unit offer detailed predictions across five domains: named outcomes, specific timescales, named geopolitical end-states, closing literary references to famous fictional utopias. These are rich destinations, and the vision is compelling on its own terms. The blueprint quality isn't a flaw in itself — destinations are motivating. But it does mean that when the path changes, the vision needs to update, and blueprint mode makes updating harder. Specific predictions get locked in. Revisions feel like retreats. And as §04 showed, the geopolitical path is very likely to change.

Toner's essay is the most self-consciously guidepost voice in the unit. She imports Virginia Postrel's dynamism-versus-stasism framework and applies it to AI governance: dynamist approaches act on local knowledge, maintain competitive feedback, and produce nested revisable rule structures; stasist approaches centralise control, fix outcomes, and resist deviation. The framework is useful for sorting governance proposals — you can use it to rule out the Bostrom-style 'ubiquitous real-time worldwide surveillance' solutions, which are stasist and whose costs in non-catastrophic worlds are real. You can't use it, on its own, to select among dynamist alternatives.4 What Toner's frame does is ensure that the mechanism from §03 and the geopolitical approach from §04 stay adaptive — that the shaping strategy updates as the frontier jagged edge moves. A guidepost says which direction is forward. Having established what the prize is, what the obstacle is, what the mechanism is, and what the international complication is — the last question is: who, actually, is the 'we' doing all of this?4. Toner is honest about this: the essay explicitly says she doesn't have solutions, just a better frame for the debate. That's admirable and also limits the essay's practical reach. A frame that rules out bad options without selecting among good ones is valuable — but the unit needs at least one paper that gets to the institutional specifics, and none of them quite do.

§06Who is "we"?

The diagram below is a map of the current state.

When a technology is new enough, the decisions happen before the governance infrastructure exists to broaden them — and the room is always small at the start, nuclear included, where every consequential decision in the first decade passed through a handful of physicists, generals, and one president, and the internet, whose architecture was fixed by a community of hundreds before billions were using it, and biotech, where the recombinant DNA debates of the 1970s happened in a room that excluded almost everyone whose life would be shaped by the outcome. This isn't unusual. It is simply how it begins.

What the diagram makes visible is the gap between the innermost and outermost rings — the gap between the actual decision-makers and the implied subjects of the civilisational predictions. That gap is not permanent. It is the governance problem in spatial form, and it is closeable by deliberate institutional design. The question is not whether the room is small now — it is — but what gets built to widen it, on what timeline, and whether the building happens before or after the most consequential decisions are made.55. Toner's concentration-of-power framing is a careful treatment of this in the unit. She names it as a first-class existential risk distinct from misalignment — a world where AI is aligned but controlled by a narrow group is not a good outcome even if the group has good intentions. The institutional question is: what structures make power concentration less likely as capability increases? None of the nine readings fully answers this.

Nine readings. What I actually think, having read them all: the marginal-returns framework is the unit's most durable contribution — it lets you stop arguing about whether AI is transformative in the abstract and start asking domain-specific questions about bottlenecks. Bregman's blueprint/guidepost distinction is the right diagnostic for why most AI strategy writing feels simultaneously ambitious and somehow unserious. And the IFP playbooks are encouraging precisely because they're not theoretical — they're historical. Shaping has worked before. Whether it will work here depends on whether the institutions doing the shaping exist yet.

And the 'we' is small. The bottleneck isn't ideas — this unit is full of good ones. It's the organisations that make safety commitments sticky, that get governance in place before the capability arrives, that hold the plural competition Toner describes together without letting it slide into the instability RAND warns against. The safety labs, policy institutes, international bodies, and cohort programmes being built now — BlueDot's among them — are the first serious attempt to close that gap.

— written during the first week of BlueDot's AGI Strategy cohort, in London.