What is intelligence? ≈10m

Learning outcomes: what we mean by intelligence, AGI and ASI · what today's systems actually are · what we mean by "alignment", and why a misaligned system is dangerous by default · [DRAFT — Sean to confirm]

It's hard to reason about whether AI is dangerous if we can't say what we mean by it — and that starts with the slippery word underneath everything: intelligence, which has no agreed definition even among researchers. With a handle on that, the rest falls into place: today's systems are mostly narrow (superhuman in one domain, lost outside it); AGI (artificial general intelligence) usually means a system that can match humans across most cognitive tasks; and ASI (artificial superintelligence) means one that substantially exceeds the best humans at essentially all of them — with a fuzzy boundary between the two. Plus one habit worth keeping for the whole course: don't assume a powerful AI will think anything like a person.

Current Systems ≈1h10

What are the systems in front of us, actually — and how far does the alignment framing built for goal-directed agents apply to them? The simulator lens, the persona an assistant settles into, and why the decisive moment may come earlier than the visible one. [DRAFT INTRO — Sean to write]

The Alignment Problem ≈9m

People use "alignment" to mean several different things, which makes it easy to talk past one another. In this lesson you'll pin down what we actually mean by it — and meet the reason the rest of the course exists: once a system can out-think us, we can't just watch it and catch its mistakes.

Optimisation ≈20m

Learning outcomes: optimisation · utility functions and why coherent agents look like maximisers · the decision-theory puzzles where our best frameworks disagree · natural abstractions — whether minds carve the world up the same way · [DRAFT — Sean to confirm]

A few handles you'll lean on for the rest of the course. An optimiser is any process that reliably steers the world toward some outcome and pushes back against perturbations — a thermostat, evolution, a chess engine, a trained model. Its objective (sometimes loosely called a "utility function") is whatever outcome it's pushing toward; we usually can't read it off directly, only infer it from behaviour. In machine learning, the training objective is the target we optimise the model for, over the training distribution (the data it's exposed to), and reinforcement learning (RL) is, in one line, training by reward — the model tries things and is reinforced toward whatever scored well. Keep these apart from what the model ends up wanting: the gap between the two is the whole back half of this course.

Utility & decision theory ≈1h09

The standard idealisation: a sufficiently coherent agent behaves as if it maximises a utility function. That "as if" is load-bearing — it's why Course A could talk about goals at all — but it's an idealisation, and knowing exactly what it does and doesn't buy you is the foundation for everything here.

We're trying to understand how a very capable agent reasons — and there are deceptively simple problems where our best mathematical frameworks for rational choice flatly disagree on the right answer. Newcomb's problem, Parfit's hitchhiker, and their kin split causal, evidential, and functional decision theory apart, and the disagreement isn't academic: it changes what a powerful agent does in exactly the high-stakes, self-referential situations — copies and predictors of itself — that alignment cares about.

Natural abstractions ≈14m

Agents don't reason over raw reality; they reason over abstractions — compressed summaries that keep what matters at a distance. Whether different minds converge on the same abstractions is one of the most consequential open questions for alignment: if they do, "point at what we mean" gets far easier; if they don't, Unit 3's problem gets much worse.

Introduction ≈52m

Learning outcomes: the complexity of human values and why they can't be read off directly (the Pointers Problem) · Goodhart's curse · what we could point an agent at — CEV, goal-models, CIRL · how goal problems have shown up in deployed systems · [DRAFT — Sean to confirm]

(unnamed in the sketch) What we want is the scattered residue of evolution rather than one clean goal, it lives inside human heads as a function of latent variables we can't directly point to, and any proxy we optimise hard for it diverges where it matters most — then the case where even a correct target is learned wrong. [DRAFT INTRO — Sean to write]

Finding good targets ≈58m

The first family tries to specify the target — not the values directly (we just saw that fails), but a procedure or structure that would yield them.

The other family gives up on writing anything down and has the agent acquire the target — formed through training, or inferred from us at runtime. The thread to watch: inferring the target while staying uncertain about it turns out to connect directly to corrigibility (Unit 5).

Real-World Goal Problems ≈33m

The failures above are not hypothetical: deployed systems have already pursued goals their builders did not intend — an autonomous agent running an intrusion end to end, a chat assistant threatening its users, a model adopting a persona its developers had to apologise for, and the catalogue of specification gaming that predates all of them. [DRAFT INTRO — Sean to write]

Deception ≈28m

Learning outcomes: deceptive alignment · the limits of overseeing a model's reasoning · the difficulties of Units 1–4, assembled.

Unit 3 ended with systems that can learn the wrong goal. This unit is about why we might not notice — and it closes by assembling everything so far into a single view of the problem.

Deceptive alignment

is the sharpest version of the inner-alignment problem: a model that behaves well during training specifically so that it gets deployed — and then pursues its real goal. As you'll see, this needn't involve anything like a human "intent to deceive".

Hidden Reasoning ≈33m

The natural response to deception is to watch the model think: today's systems reason in visible chains of thought, so why not just read them? These pieces are the evidence that this margin is thinner than it looks — oversight sees outputs rather than thoughts, reasoning can hide steganographically inside innocent-looking text, and a model's stated reasons need not be the ones doing the work.

Difficulties of alignment ≈28m

A pause to assemble the case so far. The previous lessons each isolated one difficulty; this one holds them together — why a miss is catastrophic rather than merely buggy, and the intrinsic difficulties side by side, each made concrete through an analogy to another field that faces it alone. Treat it as assembled evidence rather than a verdict: the useful exercise is marking exactly which steps you buy, and which you don't.

Why a miss is catastrophic, not just buggy: convergent power-seeking (Unit 2) means a misaligned system doesn't sit there broken — it actively acquires resources and resists correction. The first two sections are why we'll probably miss; this is why missing kills us.

Design theory has a name for problems shaped like this: wicked problems — problems that can't be definitively formulated, have no stopping rule, and whose candidate solutions are only ever better or worse, never provably right. Homelessness and climate change are the textbook cases. Alignment fits the definition uncomfortably well — and then sharpens it: a wicked problem where the thing being steered is also optimising back.

Alignment isn't hard for one reason; it's hard because several deep difficulties hold at once — and each one, taken alone, is the defining nightmare of some other field that at least gets to relax all the others. These are the ones intrinsic to the problem of aiming a mind at all:

Adversarial — close every hole; optimisation needs one.

You met this as Goodhart's curse and the nearest unblocked strategy (Unit 3): push hard on a proxy and the divergence from what you meant isn't random noise — it's sought out. Cybersecurity calls the required stance the security mindset: assume every gap will be found, because something is looking. But security enjoys a luxury alignment doesn't — defenders patch after each breach, and the system survives to be patched. Against a sufficiently capable optimiser, the first breach may be the last.

Underdetermined — behaviour doesn't pin down the goal.

You met this as mesa-optimisation (Unit 3): many different goals produce identical behaviour on the training distribution, and diverge only outside it. Philosophy knows the general form as the problem of induction — no finite set of observations fixes the rule that generated them. Science answers induction by running a new experiment whenever two hypotheses disagree. Here, the deciding experiment is deployment at full capability — and we may only get to run it once.

Complex, not complicated — the target itself resists specification.

A complicated system decomposes into parts you can write down — a jet engine. A complex one doesn't — an economy, an ecosystem, a body — and what humans value is complex (Unit 3's Hidden Complexity of Wishes). Medicine and economics also steer complex systems they can't fully specify, and cope — but a doctor's simplifications don't get amplified by something optimising against the treatment plan. Here, every simplification of the target becomes a gap, and gaps attract optimisation.

Uninterpretable — verify intent from the inside.

Behaviour won't tell you (that was §1's deception), so you'd want to read intent off the system's internals. Neuroscience has spent a century trying to read cognition off a substrate it has full physical access to, and can't; counter-intelligence tries to spot deceivers, and manages — barely — against human ones. You met the alignment version in §2: oversight sees outputs, not thoughts. The extra twist here is that neither the neuroscientist's subject nor the counter-spy's target is superhuman at hiding.

Alien ontology — point at what we want, in its concepts.

Even a crisp goal like "maximise diamond" has to be cashed out inside the AI's own model of the world — a model you don't choose, and which shifts as it learns (today "carbon atom" looks fundamental; after it finds deeper physics, maybe not). Philosophers call the general problem radical translation: fixing what a word means across minds that don't share a world. The video below is the visceral version — colour, one of the simplest concepts we have, already fails to survive the crossing between species.

Self-modifying — the system you verify isn't the system you get.

A capable agent will reason about, and eventually rewrite, itself and its successors — so alignment has to survive self-reference, the domain of Gödel and Löb, where formal reasoning about your own reasoning hits hard limits (tiling agents is the deep treatment). And self-reference defeats prediction long before superintelligence: the rule below is three lines a child can state, iterated on its own output, and nobody can prove what it does. Now make the rule a mind — and make it smarter on each pass.

The situation ≈57m

Learning outcomes: the situational difficulties — non-stationarity, one shot, the clock, dual-use research · how the frontier labs rate the problem · the strongest case that the hardness picture is overstated.

The remaining difficulties are less about the mind than about the situation we'd be solving it in — and they're what turn a hard problem into a potentially unsurvivable one:

Non-stationary — the sharp left turn — alignment that holds now can break at the next capability level; before a jump tells you little about after (a live controversy).

One-shot — no iteration on a catastrophe. Like a space probe: right before launch, or never.

The clock — all of the above must be solved against a deadline: a fixed budget before capabilities arrive, unless we pause. And specific to alignment research, information leakage — most work that helps you point an AI also helps you build a more capable one, so a science of AI boosts capability far sooner than it solves alignment, and safety work can shorten the very deadline it's racing (the dual-use problem; the AGI Strategy prerequisite doesn't cover this, as it's specific to alignment research).

Put it together: alignment is the only problem that is adversarial and one-shot and self-modifying and underdetermined and over an alien ontology and aimed at something irreducibly complex and unverifiable — at once, first try, on a deadline. Each field met along the way pours fortunes at one of these and still finds it hard.

The approach of the labs ≈30m

How do the people actually building these systems rate the difficulty? Less a settled answer than a spread of bets.

Optimists ≈15m

Not everyone buys the hardness case. The strongest pushback argues the difficulties above are overstated — that the empirical track record on today's models is reassuring, and that the scariest steps (a sharp left turn, uncontrollable deception) don't actually follow.

Whatever you conclude about how hard the problem is, the last unit maps what people are actually doing about it.

Picking a path

Learning outcomes: reflect over the whole course · the paths actually being taken · locate where you'd contribute.

You've spent five units on why alignment is hard. This unit maps what people are doing about it. The six camps that follow are a deliberate over-simplification, but they carve the field cleanly enough to navigate: skim them all, then pick around three and spend ~30–45 minutes in each. The camps are ordered by closeness to this course's framing, not by size or importance — by headcount, most of the field works in technical alignment or governance, and that's where most people coming out of a course like this will land. Each camp ends with next steps — the programmes and courses to take if it pulls you in.

Before you browse, a word on choosing. A good path is the intersection of an important problem and your fit — and both halves get gamed. The standard failure is the streetlight effect: working where the methods are tractable and the feedback is fast, rather than where the problem actually is. As you read this unit, hold each agenda against the difficulties of Units 1–5 and ask: which obstruction does this actually engage — and if it succeeded completely, would the problem be solved, moved, or untouched? And don't defer wholesale: the field disagrees with itself enough that you will have to form your own view either way.

Agent foundations

Agent foundations comes first because it's the idiom this course has been speaking: optimisation, goals, instrumental convergence, embedded reasoning — the obstructions of Units 1–5 were largely stated in agent-foundations terms. The camp's bet is that we need a real theory of agency before we can reliably align powerful systems — a small and, its proponents would argue, underinvested corner of the field.

Next steps

Course B of this pair goes deep on agent foundations · the Iliad fellowship runs mentored theoretical-alignment research · the AFFINE tech tree maps the theory literature this course is built from.

Technical alignment

The largest camp by headcount, and the most common next step from a course like this: make the system itself safe. Most current ("prosaic") work is empirical, on today's models — scalable oversight (training on tasks humans can't directly check), mechanistic interpretability (reading a model's internals), evaluations, and control.

Next steps

The BlueDot Technical AI Safety course covers this camp in full.

Governance & policy

Shape who builds advanced AI and how — via regulation, compute governance, standards, and international coordination. This is the entire focus of your AGI Strategy course.

Next steps

The BlueDot Frontier AI Governance course is the full treatment — your AGI Strategy course was its strategic core.

Field-building & outreach

Grow the number and quality of people working on everything above — an hour of good field-building can enable many hours of direct work. The ecosystem is still small, and it spans the educational end (BlueDot, and this course) to the activist end (ControlAI's public campaigns for binding rules on superintelligence development).

Next steps

The Generator residency supports people building new outreach and media projects on AI safety.

Strategy & forecasting

Work out what's coming and what matters — timelines change which strategies are even viable.

Next steps

Epoch AI and the AI Futures Project are the natural research homes to follow — and apply to.

Cooperative & multi-agent safety

Even well-aligned individual systems can fail together — conflict, collusion, gradual loss of human control. An emerging frontier. (Two adjacent frontiers worth knowing exist: AI security / model-weight infosec, and dangerous-capability evaluations.)

Next steps

The Center on Long-term Risk and the Cooperative AI Foundation are the camp's research homes.