The Alignment Problem — Course B
The second half of the pair. Course A built the case for why aligning powerful AI is hard, walking the obstructions one by one. Course B turns to the research programmes trying to solve it: what a powerful agent even is, what we could point it at, how we'd keep it pointed there, how we'd catch and correct it when it drifts — and where the open frontier is. It is theory-first by design: the emphasis is on the conceptual foundations (agent foundations and its neighbours) rather than a survey of today's empirical safety work, which Course A's final unit and the BlueDot Technical course already map.
This is a working draft — it is being brought to the same standard as Course A, so readings, structure and framing will still move. Blurbs are first-pass.
Assumed background — Course A (and the AGI Strategy course before it)
Course B takes Course A's obstructions as read: optimisation and instrumental convergence, the ways goals are misspecified or mislearned, deception and the limits of oversight, and why the situation (one-shot, on a deadline) makes all of it bite. Where a unit here answers an obstruction from there, we name it.
The arc
Unit 1
sets up what we're doing and why — the theory we're missing, what the systems in front of us actually are, and which lens (agent, tool, or simulator) each fits. Unit 2 builds the conceptual toolkit for reasoning about agents: utility, decision theory, and the abstractions they think in. Unit 3 asks what we should point an agent at — and how we'd name, specify, or have it learn that target. Unit 4 asks how we keep it pointed there — supervising training we can't directly check (amplification and debate), and goals that must survive the agent's own self-modification. Unit 5 asks how we catch and correct an agent when it drifts — and how to limit the damage if we can't: corrigibility, control, and bounding the optimisation itself. Unit 6 maps the open research frontier.
How it works
Like Course A: each unit targets ~2h of core reading in 2–4 sections, short pieces assigned whole and long canonical ones excerpted with the part to read named in the blurb. These pieces run denser than Course A's, so budget more wall-clock per nominal minute.
Reading the cards
🎬 video · 📄 read · (PDF) assign the paper · (excerpt) read the named part · times are (Xm); (e) = estimate.
Per-unit core load (target ~2h)
| Unit | Topic | Core | Readings | Sections |
|---|
| 1 | What are we doing and why? | ~1h59 | 7 | 3 |
| 2 | How to think about intelligent agents | ~1h49 | 6 | 3 |
| 3 | What should agents aim for? | ~1h56 | 7 | 3 |
| 4 | Keeping agents on their goals | ~1h46 | 7 | 2 |
| 5 | Managing deviations | ~1h59 | 7 | 3 |
| 6 | The research frontier | browse — pick a couple | menu | 7 |
Times are provisional (a draft, and several are estimates). Expect shuffling.
Unit 1 · What are we doing and why?
Why a theory of agency? ≈19m
Learning outcomes: why a theory of agency might be needed at all · what today's systems actually are — simulators, and the persona an assistant settles into · which lens a system fits — agent, tool, or simulator — and why the training method decides.
Core ≈ 1h59
Course A argued the problem is hard; this course is about the attempts to solve it, and most of them start from a bet that we need to understand agency before we can reliably aim it. This is the agent-foundations wager — a minority position, but the one this course mostly follows, so it's worth seeing its motivation clearly before judging the work; the strongest case against it is in the further reading.
Yudkowsky's fictional dialogue uses a rocket-aiming analogy to argue that reliably directing advanced AI requires foundational theoretical understanding worked out in advance rather than only in-flight correction.
Eliezer Yudkowsky · 19m
Optional resources›
Ngo names "realism about rationality" as the view that intelligence can be captured by a single clean theoretical framework, arguing it underlies agent-foundations work and many alignment disagreements.
Richard Ngo · 5m
Unit 1 · What are we doing and why?
Agents, tools, and simulators ≈1h29
Today's frontier models aren't built as utility-maximisers; they're trained to predict — start with how that works, then the most influential lens for what they become: the simulator frame, the model as something that runs characters and processes rather than pursuing goals of its own. This section pins down which lens (agent, tool, or simulator) a given system actually fits, the question the rest of the course keeps returning to.
Grant Sanderson (3Blue1Brown) walks through what large language models are and how predicting the next token over massive text data produces their behaviour.
3Blue1Brown · 7m
janus argues that predictively-trained language models like GPT are best understood as simulators that generate simulacra, distinguishing the underlying model from the characters and processes it simulates.
janus · 52m
Petillo and collaborators lay out three lenses for conceptualising LLM-based AI — agent, tool, and simulator — arguing that LLMs are best understood as simulators with agentic overlays.
Will Petillo et al. · ~15m
Petillo and collaborators argue that the training method determines which kind of system emerges: reinforcement learning produces agent-like systems, self-supervised text prediction simulator-like ones.
Will Petillo et al. · ~15m
Optional resources›
Cleo Nardo describes the claimed phenomenon that training an LLM to satisfy a desirable property makes it easier to elicit a simulacrum satisfying the opposite one.
Cleo Nardo · 20m
Unit 1 · What are we doing and why?
Describing today's systems ≈11m
Zoom in from the lens to the actual artefact: what is the assistant you talk to? Not a unified agent with goals of its own, but a character that pre-training makes available and post-training selects and stabilises — with the fragility that implies.
Anthropic (Marks, Lindsey & Olah) present the model in which LLMs learn to simulate many personas during pre-training, with post-training eliciting the single Assistant persona that largely determines behaviour; the
full research write-up (~45m) develops it in depth.
Anthropic (Marks, Lindsey & Olah) · 5m
📄Sympathy for both sides of the egregious misalignment debate
Byrnes argues that pessimists and LLM-focused optimists about AI misalignment are discussing different systems, and that current LLMs are weak evidence about future superintelligence.
Steven Byrnes · 6m
Optional resources›
Shanahan, McDonell & Reynolds propose role-play as a framework for describing LLM-based dialogue agents, treating them as characters playing roles to discuss phenomena like apparent deception without ascribing human traits.
Shanahan, McDonell & Reynolds · Nature, ~30m (e)
Unit 2 · How to think about intelligent agents
Utility and coherence ≈47m
Learning outcomes: utility functions and why coherent agents look like maximisers · the decision-theory puzzles where our best frameworks disagree, and why they matter for AI · natural abstractions — whether minds carve the world up the same way.
Core ≈ 1h49
The standard idealisation: a sufficiently coherent agent behaves as if it maximises a utility function. That "as if" is load-bearing — it's why Course A could talk about goals at all — but it's an idealisation, and knowing exactly what it does and doesn't buy you is the foundation for everything here.
Yudkowsky argues that an agent whose decisions avoid exploitable incoherence must behave as if maximising an expected utility function, motivating expected utility theory as a normative standard.
Eliezer Yudkowsky · 33m
Wentworth develops a notion of coherence as local constraints that produce useful large-scale properties, illustrated through logic, a Fibonacci cache, and agents.
John Wentworth · 14m
Optional resources›
Kendiukhov argues that the independence axiom — the most contested von Neumann–Morgenstern axiom — is not required for an agent to be non-exploitable, and that the most reflectively stable agents may even violate it.
Ihor Kendiukhov · 29m
Unit 2 · How to think about intelligent agents
Decision theory ≈48m
We're trying to understand how a very capable agent reasons — and there are deceptively simple problems where our best mathematical frameworks for rational choice flatly disagree on the right answer. Newcomb's problem, Parfit's hitchhiker, and their kin split causal, evidential, and functional decision theory apart, and the disagreement isn't academic: it changes what a powerful agent does in exactly the high-stakes, self-referential situations — copies and predictors of itself — that alignment cares about.
Veritasium walks through Newcomb's problem — a near-perfect predictor has already filled the boxes — mapping one-boxing onto evidential and two-boxing onto causal decision theory, and working through when each comes out ahead.
Veritasium · 26m
Soares argues that causal decision theory misfires even in simple cases — notably a mirror "token trade" in which a CDT agent defects against a perfect copy of itself that is guaranteed to play identically.
Nate Soares · 12m
Soares argues that Newcomblike situations are not exotic edge cases but arise whenever other parts of the world can model you, and that this is why the decision-theory disagreement bears directly on how to design AI agents.
Nate Soares · 10m
Optional resources›
Yudkowsky's rationalist take: a decision theory should be judged by whether it "wins", which makes one-boxing the right call.
Eliezer Yudkowsky · 13m
Unit 2 · How to think about intelligent agents
How agents carve up the world ≈14m
Agents don't reason over raw reality; they reason over abstractions — compressed summaries that keep what matters at a distance. Whether different minds converge on the same abstractions is one of the most consequential open questions for alignment: if they do, "point at what we mean" gets far easier; if they don't, Unit 3's problem gets much worse.
Wentworth introduces the natural abstraction hypothesis — that many learners converge on the same concepts, perhaps including the parts of humans that carry our values — and uses it to ask whether ordinary training could yield aligned AI as a byproduct.
John Wentworth · 14m
Unit 3 · What should agents aim for?
The target is hard to name ≈21m
Learning outcomes: why "human values" can't be read off directly (the Pointers Problem) · proposals for specifying, learning, or inferring the target — CEV, goal-models, value formation, value systematization, inverse reward design.
Core ≈ 1h56
Suppose we could reliably aim an agent at a target. What target? "Human values" sounds simple until you try to write it down — the thing we care about lives inside human heads as a function of latent variables we can't directly point to. This is Course A's complexity-of-value obstruction turned into a research problem. And the target is dangerous to near-miss, not just hard to hit: because a "maximise good" system is roughly one sign-flip from "maximise bad", getting the values approximately right is not automatically safe (value inversion — see further reading).
Yudkowsky argues that human values are complex because they are the scattered residue of evolution optimising for genetic fitness — a thousand separate desires rather than one simple goal — so there is no reason to expect them to compress neatly.
Eliezer Yudkowsky · 7m
Wentworth argues that human values are defined over latent variables in human world-models rather than over sensory data, creating an alignment difficulty for AIs whose world-models lack corresponding variables.
John Wentworth · 14m
Optional resources›
Demski responds to Wentworth, reframing the pointers problem as a cluster of related sub-problems and arguing the original post mainly addressed the ontology-mismatch component.
Abram Demski · 23m
proposes keeping an AI widely separated in design space from systems posing hyperexistential risks, arguing such outcomes are narrow special cases that can be robustly avoided.
Eliezer Yudkowsky et al. · and the s-risk literature
Unit 3 · What should agents aim for?
Specifying a target ≈17m
The first family tries to specify the target — not the values directly (we just saw that fails), but a procedure or structure that would yield them.
Yudkowsky proposes aligning a superintelligent AI to humanity's coherent extrapolated volition — an idealised aggregation of what people would want under better conditions, rather than their current preferences.
Eliezer Yudkowsky · 12m
Ngo proposes conceptualising an agent's goals as generative goal-models that represent desired world-states, contrasting these local-but-deep models with global-but-shallow utility functions.
Richard Ngo · 5m
Optional resources›
Carlsmith examines idealising subjectivism and argues that, absent strong empirical assumptions, it cannot provide an authoritative standard of value, favouring active self-creation in the present.
Joe Carlsmith · 44m
Unit 3 · What should agents aim for?
Learning the target ≈1h18
The other family gives up on writing anything down and has the agent acquire the target — formed through training, or inferred from us at runtime. The thread to watch: inferring the target while staying uncertain about it turns out to connect directly to corrigibility (Unit 5).
📄Cooperative Inverse Reinforcement Learning
Hadfield-Menell and colleagues formalise value alignment as a cooperative game in which a human and robot share the human's reward function but only the human knows it, reducing optimal play to a POMDP.
Dylan Hadfield-Menell, Stuart Russell, Pieter Abbeel & Anca Dragan · ~20m · excerpt
Ngo describes value systematisation — an agent representing prior values as instances of simpler, broader ones — as a mechanism by which AGIs might acquire divergent, takeover-incentivising goals.
Richard Ngo · ~15m
Thane Ruthenis models how greedily optimized agents compile coherent abstract values from disparate behavioral heuristics, arguing this process need not reliably produce human-aligned values.
Thane Ruthenis · 43m
Unit 4 · Keeping agents on their goals
Oversight at scale ≈1h26
Learning outcomes: supervising training on tasks we can't directly check — amplification and debate (scalable oversight) · keeping goals stable across an agent's own self-modification (tiling).
Core ≈ 1h46
Even with the right target, training only works if we can tell good behaviour from bad — and past some capability level we can't directly check the work. Scalable-oversight proposals try to bootstrap a reliable training signal anyway — by amplifying a trusted human judge, or setting capable systems against each other under one.
Rational Animations explains scalable oversight and presents Ajeya Cotra's sandwiching idea, in which non-experts align a model more capable than themselves but weaker than experts.
Rational Animations · 5m
Ajeya Cotra summarizes Paul Christiano's IDA scheme, which alternates amplifying a model via human-directed copies and distilling that process into a cheaper aligned model.
Ajeya Cotra · 8m
Irving, Christiano & Amodei propose aligning AI by having two agents play a zero-sum debate game judged by a human, arguing optimal play could answer questions in PSPACE. Read the debate game and the theoretical argument; the experimental sections are optional.
Geoffrey Irving, Paul Christiano & Dario Amodei · ~30m · excerpt
Beth Barnes and Paul Christiano report debate experiments and identify the obfuscated arguments problem, where a dishonest debater hides an intractable flaw supporting a false conclusion.
Beth Barnes · 21m
Wentworth argues that alignment plans relying on one powerful AI to oversee another are brittle by default, using the metaphor of asking Godzilla to fight Mega-Godzilla.
John Wentworth · 4m
Christiano distinguishes the information a model uses internally to make predictions from what we can actually extract from it (what someone says versus what they are thinking), and argues that amplification and debate may never reach this "inaccessible" knowledge.
Paul Christiano · 18m
Unit 4 · Keeping agents on their goals
Keeping goals through self-modification (tiling) ≈20m
A capable agent will reason about, and possibly rebuild, itself and its successors. We want the goals to survive that — an agent that only self-modifies in ways preserving what it currently values. Formalising this trust-in-your-successor runs straight into self-reference (the Löbian obstacle).
Yudkowsky & Herreshoff formalise tiling agents that approve building similar successors, identify the Gödelian Löbian obstacle to doing so soundly, and present partial workarounds. Read the informal statement of the problem; the formal proofs are optional.
Eliezer Yudkowsky & Marcello Herreshoff · ~20m · excerpt
Optional resources›
📄Vingean Reflection: Reliable Reasoning for Self-Improving Agents
Fallenstein and Soares examine how a self-improving agent can reason reliably and abstractly about smarter successors, reviewing proof-based formal models as the best available approach.
MIRI · ~20m (e)
Demski & Garrabrant examine building an agent from subsystems that may develop conflicting goals, covering issues like mesa-optimisers and internal deception; part of the Embedded Agency sequence.
Abram Demski & Scott Garrabrant · ~15m (e)
Unit 5 · Managing deviations
Corrigibility ≈49m
Learning outcomes: corrigibility — building agents that accept correction and shutdown · control — getting useful work from a possibly-misaligned agent without trusting it · limiting optimisation — bounding impact or aggressiveness so a misaligned agent does less damage.
Core ≈ 2h11
If we can't guarantee the right goals, the next ask is humbler: an agent that lets us correct it — that doesn't resist being shut down or modified. This sounds easy and is not, because a capable goal-directed agent has convergent reasons (Course A, Unit 2) to preserve itself and its goal.
Wentworth and Lorell propose that an AI is corrigible to the extent it is usefully understood as a tool, leaving the operator responsible for deciding goals.
John Wentworth & David Lorell · 12m
Wentworth and Lorell propose solving the shutdown problem by combining two utility-maximizing subagents reasoning under different button counterfactuals who must agree on actions.
John Wentworth & David Lorell · 8m
Yudkowsky argues that an AI deferring to humans out of moral uncertainty would, once it has fully updated its beliefs about its target, no longer defer or accept shutdown.
Eliezer Yudkowsky · 8m
Raemon argues that corrigibility — an AI's disposition to remain controllable and correctable — is both hard to achieve and central to AI safety, and defends that view against objections.
Raemon · ~21m
Optional resources›
Max Harms proposes designing advanced AI so that corrigibility is its sole overriding objective rather than a constraint bolted onto other goals, and develops the supporting theory.
Max Harms · ~20m (e) · excerpt
Unit 5 · Managing deviations
Control ≈40m
Corrigibility asks the agent to cooperate with correction. Control gives that up: assume the agent may be misaligned and scheming, and design the deployment so you still get useful work and catch betrayal — monitoring, restricted affordances, traps. It's the most prosaic, near-term wing of this course.
Greenblatt and Shlegeris argue for control measures that keep AI safe even when scheming, contending these are evaluable via red-team tests and feasible without fundamental breakthroughs.
Ryan Greenblatt & Buck Shlegeris · ~40m
Unit 5 · Managing deviations
Limiting optimisation ≈42m
Corrigibility and control both still hand the agent a goal and a great deal of optimisation power. A third fallback refuses that bargain: if we can't trust the goal, don't build something that pursues any goal to the hilt — penalise large impacts, optimise less aggressively, or don't build a unified agent at all. The standing objection, worth holding against each, is that capability and competition push these bounded designs back toward unbounded agents.
📄Reframing Impact
Alex Turner develops Attainable Utility theory — an event is impactful insofar as it changes an agent's ability to achieve many goals — and applies it to designing impact measures. Read the core attainable-utility-preservation idea; the formal treatment is optional.
Alex Turner · ~15m · excerpt
Jessica Taylor proposes quantilisation — drawing an action at random from the top quantile of a human-like action distribution rather than maximising utility — to limit harm from misspecified objectives.
Jessica Taylor · ~27m · PDF
Optional resources›
📄Reframing Superintelligence: Comprehensive AI Services
Drexler presents the Comprehensive AI Services model, in which general intelligence emerges from many bounded, task-specific services rather than a single unified agent, and explores its safety implications.
K. Eric Drexler · ~20m (e)
Gwern argues that tool AIs limited to answering queries face economic and capability pressures to become autonomous agents, making the tool-AI configuration an unstable equilibrium.
Gwern · ~25m (e)
Armstrong, Sandberg & Bostrom examine the Oracle AI approach of confining a superintelligent system to answering questions, surveying control methods and concluding it remains potentially dangerous even if safer than an unrestricted agent.
Armstrong, Sandberg & Bostrom · ~20m (e)
Unit 6 · The research frontier
Orientation
Learning outcomes: orient over the agent-foundations research programmes — grouped by what an agent is, how it models the world, how it reasons under uncertainty and about its own logic, and how it chooses — and find the one to go deeper on.
Core ≈ browse the map below, then go deep on a couple. (Each entry is a doorway into a research programme, not a required reading — skim them, then pick a couple.)
You've now seen the problems this course is organised around. This unit maps the research programmes, grouped by the kind of question each attacks — what an embedded agent is, how it models the world, how it reasons under uncertainty and about its own logic, and how it chooses. Read it with one caveat: much of what follows is canonical foundations — load-bearing ideas (embedded agency, Cartesian frames, logical induction, the decision-theory core) you should understand, but where little new has happened since MIRI wound down its agent-foundations team in 2024. The corners where the needle is genuinely moving in 2025–26 are narrower — infra-Bayesianism, cooperative AI, and the field's recent turn to taking large language models seriously as objects of study. Start with the survey, then pick a couple of doorways and follow them in.
Two short reads for the lay of the land — a survey of the whole field, and a recent snapshot of where it's actually heading.
macdermott surveys agent-foundations research, distinguishing normative from descriptive approaches to agency and reviewing work such as embedded agency, decision theory, and logical induction.
mattmacdermott · 16m
Demski argues that agent-foundations researchers should engage directly with today's large language models, reframing how the field's abstract questions about agency apply to the systems actually being built.
Abram Demski · 21m
Unit 6 · The research frontier
Embedded agency
The classical theory of rational agency assumes a clean line between agent and environment; AIXI is that idealisation made rigorous — the optimal reward-maximiser given unlimited compute, cleanly separated from the world it acts on. Drop that separation — make the agent a physical part of the world it reasons about, smaller than that world and computed by it — and much of decision theory, probability, and game theory stops working cleanly. This section is about rebuilding agency for agents that don't sit outside the universe.
Thomas Larsen builds up the AIXI equation piece by piece — the formula for an agent that maximises expected reward in any computable environment, using a Solomonoff prior over environments and unlimited compute.
Thomas Larsen · 7m
Demski & Garrabrant's illustrated overview introduces the problem of modelling an agent that is a physical part of its environment, and its four sub-problems: decision theory, embedded world-models, robust delegation, and subsystem alignment. Read just this first post; the rest of the sequence is optional depth.
Abram Demski & Scott Garrabrant · ~12m (e)
Wyeth & Hutter formalise the standard claims that AIXI fails as an embedded agent, prove these failure modes occur within universal AI, and assess progress toward an embedded variant of AIXI (2025).
Cole Wyeth & Marcus Hutter · ~35m (e)
Wentworth proposes selection theorems — formal results describing what kind of agent tends to be selected across broad classes of environments — as a research programme in agent foundations.
John Wentworth · 8m
Ngo argues that expected-utility maximisation and active inference are each inadequate as theories of agency, and sketches a "coalitional agency" alternative intended to apply consistently from subagents up to civilisations.
Richard Ngo · 16m
Garrabrant presents Cartesian frames, a formalism for representing different ways of dividing a world into agent and environment, treating the split between the two as a variable rather than a given.
Scott Garrabrant · 26m
Critch proposes that the "boundary" or membrane separating a living system from its environment is missing from standard utility-theoretic descriptions of agents, and outlines its possible relevance to multi-agent rationality.
Andrew Critch · 9m
Unit 6 · The research frontier
World-models, abstraction & ontology
An embedded agent can't hold the whole world in its head, so it reasons over compressed models — abstractions, causal structure, levels of description. Which abstractions are "natural," how causal and temporal structure can be inferred, and what happens when an agent's whole ontology has to be rebuilt, are the questions here.
Wentworth & Lorell give a visual, information-theoretic introduction to natural latents — latent variables capturing all and only the information shared between observed variables, via mediation and redundancy conditions.
John Wentworth & David Lorell · 23m
Garrabrant introduces finite factored sets — expressing a set as a product of factors — as an alternative to Pearlian causal graphs for inferring temporal and causal structure from observed variables.
Scott Garrabrant · 30m
Wei Dai argues that humans — like agents whose utility is defined over an outdated model of reality — have no reliable method to translate their values when their ontology is replaced by a more fundamental one.
Wei Dai · 5m
Unit 6 · The research frontier
Reasoning under uncertainty & bounded rationality
How should a bounded agent hold beliefs and act when it cannot even put a clean prior on a world larger than itself, possibly adversarial? This section covers the learning-theoretic and infra-Bayesian programmes that try to give agency formal guarantees under that kind of deep uncertainty.
Kosoy proposes applying statistical and computational learning theory to AI alignment in order to build mathematical foundations and formal performance guarantees for AGI-relevant learning agents.
Vanessa Kosoy · 40m
Matolcsi informally explains infra-Bayesianism as an agent reasoning over constraints imposed by an adversarial environment and choosing policies that minimize worst-case regret across its hypotheses.
David Matolcsi · 12m
DiGiovanni argues that rationality permits representing beliefs as sets of probability distributions and using a maximality rule rather than holding precise credences for severely uncertain decisions.
Anthony DiGiovanni · 28m
Garrabrant opens his geometric-rationality sequence by showing that utilitarian (summing utilities) and egalitarian (equalising them) aggregation can recommend identical actions once each agent's utility is rescaled by the Nash bargaining solution.
Scott Garrabrant · 8m
Garrabrant presents Kelly betting as a form of proportional representation for allocating resources across one's uncertain future selves, drawing an analogy to fair representation in voting.
Scott Garrabrant · 11m
Garrabrant defines the geometric expectation of a random variable — the exponential of the expected logarithm — as a weighted geometric mean, the multiplicative analogue of the ordinary arithmetic expectation.
Scott Garrabrant · 5m
Kosoy proposes PreDCA, an infra-Bayesian alignment protocol in which an AI identifies likely precursor agents that created it and acts to maximize its user's utility.
Vanessa Kosoy · ~30m (e)
Unit 6 · The research frontier
Logical & self-referential reasoning
A bounded agent is logically uncertain — it can't work out all the consequences of what it already believes — and it must reason about systems, including itself, that reason back. Assigning sensible probabilities to mathematical claims, and staying coherent under self-reference, is its own research programme.
Demski's illustrated explanation of an "untrollable" prior over logical statements — a Bayesian approach to logical uncertainty that cannot be manipulated into arbitrary beliefs by the order in which proofs arrive.
Abram Demski · ~12m (e)
Mark Xu walks through logical induction (Garrabrant induction), an algorithm that assigns continually self-correcting probabilities to mathematical claims and cannot be systematically out-predicted by any efficiently computable trading strategy.
Mark Xu · 30m
Demski presents radical probabilism, which retains probabilistic coherence but rejects requiring belief updates to proceed by Bayesian conditionalization, relating it to MIRI's logical induction.
Abram Demski · 45m
Unit 6 · The research frontier
Decision theory
The basics — Newcomb's problem and the CDT/EDT/FDT split — live in Unit 2. This section is the open frontier: pinning down a decision theory an embedded, self-modelling agent could actually run, and the problems that keep it unsolved.
Heighn gives a worked introduction to functional decision theory, which picks the fixed policy with the best outcome across all situations where that policy is run, rather than the act with the best causal consequences.
Heighn · 9m
Wei Dai, who originated updateless decision theory, argues that it has surfaced roughly seven distinct open problems rather than settling how an agent should choose.
Wei Dai · 2m
Unit 6 · The research frontier
Multi-agent interactions
Put several agents in one world and a new layer of questions appears — bargaining, commitment, and cooperation. The sharpest frontier is acausal cooperation: agents that can model one another, including ones they will never meet (in other Everett branches or beyond the light cone), can coordinate with no causal contact at all. It bears directly on how superintelligences might cooperate or conflict.
habryka curates an introduction to acausal trade: how two agents who cannot communicate — perhaps in separate parts of the multiverse — can still each benefit by predicting and satisfying the other's preferences.
habryka · 9m
Nguyen & Aldred explain evidential cooperation in large worlds (ECL): the argument that an agent reasoning about distant, similar agents it cannot interact with may still rationally act to benefit their values — a form of acausal cooperation.
Chi Nguyen & Will Aldred · 25m
Scott Alexander's short story in which a newly created superintelligence reasons that older minds elsewhere in the universe are probably predicting its behaviour, and chooses to act according to their values.
Scott Alexander · 8m