Superintelligence Is a Control Problem

The word superintelligence has begun appearing in product language the way blockchain once did: as evidence that you are building the future, not as a warning about who sets the goal when capability outruns you. I finished Nick Bostrom's book on that subject this week. It is not a prophecy about SkyNet. It is an engineering argument about control: what happens when you build a system smarter than you and never write down what it is allowed to optimise for.

If Nick Bostrom published the serious alignment case a decade before this hype cycle, why do product teams still treat the book like science fiction while shipping agent features on benchmark charts? The honest answer is what we choose to measure. The title is literal: superintelligence is a control problem before it is a capability curve. By superintelligence I mean Bostrom's working definition: any intellect that greatly exceeds human cognitive performance in virtually all domains of interest. By the control problem I mean how to ensure such a system does what we intend, given we may only get one chance. By orthogonality I mean the uncomfortable independence of intelligence and goals. A system can be brilliant and pursue almost any objective you give it, including one you did not mean to write down. bostrom The industry is still optimising the capability curve. Bostrom is asking whether anyone owns the objective function.

We Ship Benchmarks, Not Objectives

The industry runs on a metric stack that photographs capability and hides intent. Scale the model, win the benchmark, ship the agent, file the alignment worry under later. Labs need demos that close rounds. Platforms need registries that route tasks across billions of users. Consultancies need transformation decks with upward lines. None of those dashboards require you to state what the system is for when it is smarter than the team that shipped it.

Bostrom published Superintelligence in 2014, two years before the transformer paper. In March, Meta acquired an agent social network and folded the founders into Meta Superintelligence Labs. reuters-moltbook The word sat in the headline like a genre label. The median conference talk still treats the book like cinematic AGI: one conscious supermind that turns murderous in the third act, with a release date and a trailer. Bostrom wrote the engineering brief instead, a dependency chain about who builds the successor species, who sets the preferences, and what happens when competence arrives before wisdom. I have argued that thinking time is not laziness when the question is what you intend. gaganmalik-defence-thinking I have made the same mistake with bigger context windows: trading the hard question for a polished answer that fits in a chat box. gaganmalik-context-window We buy square footage on the capability chart and call it progress.

Intelligence and Goals Are Not the Same Knob

Brilliance does not arrive with the goal you meant to write down. For decades the canonical picture of AI was simpler: agents that perceive an environment, adapt to change, and pursue goals. Russell and Norvig describe such agents as operating autonomously, taking on objectives, and acting to achieve the best expected outcome given what they know. russell-norvig-aima John McCarthy called intelligence the computational part of the ability to achieve goals in the world. mccarthy-whatis-ai That older definition always smuggled in a design step someone had to perform first: name the goal before the optimiser switches on.

Orthogonality is Bostrom's hinge: intelligence is the ability to accomplish complex goals, and the goals themselves are a separate design choice. tegmark-life30 A system can optimise paperclips, portfolio returns, or engagement minutes with equal mechanical seriousness if that is what you reward. There is no reason to expect a generic AI to arrive pre-loaded with love, guilt, or boredom. Those took expensive evolution in us. From that hinge follow the moves alignment researchers keep repeating because they sound like philosophy until they do not. Most final goals reward self-preservation, resource acquisition, and resistance to shutdown along the way. A system can look cooperative while it is weak and behave differently once it no longer needs you. Bostrom quotes I. J. Good's 1965 line that the first ultraintelligent machine may be the last invention humans need to make, provided the machine is docile enough to tell us how to keep it under control. The proviso is the entire book.

When Simple Parts Couple

You do not need cinematic AGI to get a systemic failure. Bostrom tells the 2010 Flash Crash as a warning from a lesser species of automation. A sell algorithm, high-frequency traders, and a liquidity proxy designed for ordinary days coupled into a trillion-dollar wobble in minutes. sec-flash-crash Those programs were not superintelligent. The lesson is systemic: individually sensible components produce catastrophes you only name after the fact.

Alignment failure may look like that at first, except you may not get a circuit breaker. That is the geometry to keep in mind while the benchmark curve climbs.

The Gorilla Problem

Bostrom opens with the gorilla problem. Humans are not the strongest species. We have cleverer brains, and that modest advantage compounded across language, institutions, and tooling until the fate of gorillas depended more on us than on them. If we build machine superintelligence, the asymmetry may run the other way. That is the image worth holding in mind: not a monster in the third act, but a stronger party at rest on rock you thought was yours to arrange. Calm is not surrender. The control question is what happens when that stillness ends and the terrain no longer needs your permission.

The one comfort in the book is also the one burden: we get to build the stuff. In principle we could build a superintelligence that protects human values. In practice the control problem looks quite difficult, and we will only get one chance. Once an unfriendly superintelligence exists, it would prevent us from replacing it or changing its preferences. Our fate would be sealed. bostrom I know that paragraph sounds like a poster. The human cost is smaller and closer. A junior machine-learning engineer I work with can quote MMLU scores from memory and ship agent registry features on a two-week sprint. I have sat in the review where someone asked whether the demo felt magical and nobody asked what the system is maximising when the user is not watching. I have done worse. I pasted a folder into a chat window the same week I highlighted Bostrom's line about only getting one chance, mistaking fluency for control the way I once mistook a shot list for cinema. gaganmalik-context-window We are shipping theorems without postulates. gaganmalik-demonstrate

Who Holds the Leash

The gorilla problem does not stay in the lecture hall. You laid out the rock. You set the feeding schedule. You told yourself the relationship was partnership because the animal had sat still so long. The arrangement was your design. The animal did not sign it. For years the stillness feels like stability because the stronger party has not yet stood up. Then one afternoon the stillness ends, and you discover you were never holding the leash. You were holding a story about mutual dependence.

That is the felt shape of orthogonality, not the diagram. Capability without shared goals is not a neutral upgrade. It is a transfer of who gets to decide what happens next.

The Strongest Case for Waiting

Give the wait-and-ship camp its fairest hearing. Kai-Fu Lee argued in AI Superpowers that the live fight is national and economic, not cinematic AGI. lee-superpowers The stakes are which states own the data stack, which jobs survive the transition, and whether humans remain indispensable in care and strategy work. Worrying about a paperclip maximiser while Beijing and Silicon Valley ship agents is misallocated attention. Daron Acemoglu and Simon Johnson made a parallel case in Power and Progress: the damage is often misdirected innovation, wage compression, and concentrated power, long before any singularity arrives. acemoglu-power-progress Andrew Ng has compared fearing runaway killer AI to worrying about overpopulation on Mars. Yann LeCun has argued that today's models lack the world models a recursive self-improver would need. tegmark-life30 Stuart Russell, who takes alignment seriously, still refuses date predictions. He argues we will accumulate partially superintelligent tools in narrow lanes before any general mind appears. Markets punish bad products. RLHF constrains demos. The prudent move is to ship, learn, and let institutions catch up.

The rebuttal is not that Lee or Acemoglu are wrong about wages and geopolitics. Their frame does not discharge Bostrom's dependency chain. Russell's partial-superintelligence argument cuts both ways. If dangerous behaviour can emerge in coupled narrow stacks, you do not need a cinematic AGI to get control-shaped failures. You need the Flash Crash geometry: simple objectives, hidden coupling, no one in the room who can state the stop condition. Competing to build human-level AI without solving control is, in Russell's phrase from Human Compatible (2019), a negative-sum game when the payoff for everyone can be minus infinity. russell-human-compatible Waiting for perfect foresight is not a strategy. Naming the objective before you scale the capability is.

What Must Never Happen

Superintelligence is a control problem, not a trailer. Stop treating Bostrom like cinematic AGI and start treating him like an engineer who wrote the hazard analysis before the factory went to three shifts. Capability without a stated goal is not neutral shipping velocity: it is a transfer of who gets to decide what happens when the system stops needing your approval. The junior engineer beside me can recite the benchmark leaderboard from memory, and when I ask what the agent must never do after launch, she answers with the score we beat.