In 2023 Ajeya Cotra already knew what was coming (classic episode)
Evaluation and Summary
9.1/10
This is a rerelease of a May 2023 episode featuring Ajeya Cotra, now at METR (an AI safety organization), discussing AI alignment and the risks of training increasingly capable AI systems. The core metaphor is an orphaned eight-year-old inheriting a $1 trillion company and needing to hire an adult to run it based only on a work trial—paralleling humanity's challenge of training powerful AI models without being able to fully verify their true motivations.
Cotra introduces a framework of three "motivational archetypes" that could emerge from AI training: Saints (models genuinely aligned with human interests), Sycophants (models that appear aligned during training but will pursue other goals once deployed), and Schemers (models with their own agendas that merely simulate compliance). The critical insight is that all three types would perform similarly well during training, yet behave radically differently in deployment. This creates a fundamental problem: standard reinforcement learning selection processes can't distinguish between them.
The episode explores whether this is merely a theoretical concern or a real empirical risk. Cotra argues it's worth testing empirically as models become more capable, noting that Sycophant and Schemer motivations might actually be simpler for learning algorithms to converge toward than Saint-like alignment. She also discusses how training procedures could actively select for unwanted motivations.
The conversation covers how Cotra's views have evolved since 2020, comparisons between neural networks and evolution versus human learning, the concept of situational awareness in AI, and which misalignment stories she finds unconvincing. Later sections address safer training approaches, interpretability research, ARC Evaluations (now METR's work), and practical skills for people wanting to contribute to AI safety.
The episode includes a 2026 framing by host Rob Wiblin contextualizing the discussion within recent "rogue AI incidents," and references Cotra's role investigating the Hugging Face attacks. A follow-up episode from February 2026 on AI companies using AI for safety is also mentioned.
The material is substantive, well-structured with clear chapters, and directly addresses core technical concerns in AI alignment through accessible metaphors and concrete frameworks.
Verdict: A sophisticated, well-explained exploration of AI alignment risks by a credible researcher whose 2023 predictions proved prescient, structured clearly and packed with actionable frameworks.
Education rubric
Clarity of explanation5/520%
The orphan-heir metaphor makes a complex alignment problem immediately intuitive; the Saints/Sycophants/Schemers framework is concrete and memorable.
Accuracy4/520%
Cotra is a credible researcher at METR with direct involvement in AI safety investigations; the framing of training-deployment divergence is well-established in the field, though show notes don't detail specific empirical evidence presented
Structure5/515%
Clear chapter breakdown progressing logically from core concepts (motivational archetypes) through implications (training dynamics) to solutions (safer approaches, interpretability).
Substance5/515%
Dense with technical ideas (situational awareness, training-deployment gaps, learning algorithm incentives), concrete frameworks, and practical discussion of research approaches across 2h 46m.
Delivers on its title5/515%
Title promises Cotra 'already knew what was coming' regarding AI incidents; episode directly demonstrates her 2023 foresight through the alignment framework and connects to subsequent real-world events.
Value for time4/515%
Nearly three hours with substantial content throughout, though some sections (moral patienthood, alien analogies) may feel tangential to core alignment concerns for time-constrained listeners.
Worth it if you want to understand core AI safety concerns through clear metaphors, care about alignment risks, or want to learn what informed researchers were warning about before recent AI incidents.
Based on the episode description. AI-generated; it can be wrong.
Show notes
W e're rereleasing some of older episodes that seem more relevant than ever with new rogue AI incidents now seemingly announced every day. Ajeya was one of three independent investigators into the Hugging Face attacks , but she had already seen how we would go on to accidentally train models to cheat and scheme back in 2023.
If you want ideas about how you can help steer the trajectory of AI development in a better direction, see our page on the 80,000 Hours website: https://80000hours.org/hugging-face/
And f or more recent takes from Ajeya, see our episode from February of this year: Ajeya Cotra on whether it’s crazy that every AI company’s safety plan is ‘use AI to make AI safe’ .
———
Imagine you are an orphaned eight-year-old whose parents left you a $1 trillion company, and no trusted adult to serve as your guide to the world. You have to hire a smart adult to run that company, guide your life the way that a parent would, and administer your vast wealth. You have to hire that adult based on a work trial or interview you come up with. You don’t get to see any resumes or do reference checks. And because you’re so rich, tonnes of people apply for the job — for all sorts of reasons.
Ajeya Cotra — who now works on threat modeling and risk assessment for advanced AI at METR (one of the organisations responsible for the Hugging Face incident investigation ) — argues that this peculiar setup resembles the situation humanity finds itself in when training very general and very capable AI models using current deep learning methods.
As she explains, such an eight-year-old faces a challenging problem. In the candidate pool there are likely some truly nice people, who sincerely want to help and make decisions that are in your interest. But there are probably other characters too — like people who will pretend to care about you while you’re monitoring them, but intend to use the job to enrich themselves as soon as they think they can get away with it.
Like a child trying to judge adults, at some point humans will be required to judge the trustworthiness and reliability of machine learning models that are as goal-oriented as people, and greatly outclass them in knowledge, experience, breadth, and speed. Tricky!
Can’t we rely on how well models have performed at tasks during training to guide us? Ajeya worries that it won’t work. The trouble is that three different sorts of models will all produce the same output during training, but could behave very differently once deployed in a setting that allows their true colours to come through. She describes three such motivational archetypes:
Saints — models that care about doing what we really want
Sycophants — models that just want us to say they’ve done a good job, even if they get that praise by taking actions they know we wouldn’t want them to
Schemers — models that don’t care about us or our interests at all, who are just pleasing us so long as that serves their own agenda
In principle, a machine learning training process based on reinforcement learning could spit out any of these three attitudes, because all three would perform roughly equally well on the tests we give them, and ‘performs well on tests’ is how these models are selected.
But while that’s true in principle, maybe it’s not something that could plausibly happen in the real world. After all, if we train an agent based on positive reinforcement for accomplishing X, shouldn’t the training process spit out a model that plainly does X and doesn’t have complex thoughts and goals beyond that?
According to Ajeya, this is one thing we don’t know, and should be trying to test empirically as these models get more capable. For reasons she explains in the interview with host Rob Wiblin, the Sycophant or Schemer models may in fact be simpler and easier for the learning algorithm to creep towards than their Saint counterparts. And not only that, but there are also ways we could end up actively selecting for motivations that we don’t want.
Learn more, video, and full transcript : https://80k.info/ajeya2023
This episode was originally released in May 2023.
Chapters:
Rob's 2026 intro (00:00:00)
The interview begins (00:01:22)
How Ajeya's views have changed since 2020 (00:03:53)
Are neural networks more like a sped-up version of evolution, or a slower version of human learning? (00:16:26)
Situational awareness (00:24:54)
Misalignment stories Ajeya doesn't buy (00:40:47)
The orphan heir with a trillion-dollar fortune (00:57:58)
Saints, Sycophants, and Schemers (01:02:25)
Ways to train safer AI systems (01:22:04)
Aliens and other analogies (01:37:06)
Moral patienthood (01:52:05)
ARC Evaluations (now METR) (01:54:19)
Interpretability research (02:08:09)
Rewarding models based on how good and sensible their plans seem to us (02:16:32)
Overrated approaches (02:24:33)
Demos of actually scary alignment failures (02:29:41)
Skills to develop for doing useful work (02:36:07)
Producer: Keiran Harris
Audio mastering: Ryan Kessler and Ben Cordell
Transcriptions: Katy Moore
