That’s the future Will MacAskill — philosopher, founding figure of effective altruism, and now researcher at Forethought Research — argues we need to prepare for in his paper “Preparing for the intelligence explosion.” Not in the distant future, but probably in 3–7 years.

The reason: AI systems are rapidly approaching human-level capability in scientific research and intellectual tasks. Once AI exceeds human abilities in AI research itself, we’ll enter a recursive self-improvement cycle — creating wildly more capable systems. Soon after, by improving algorithms and manufacturing chips, we’ll deploy millions, then billions, then trillions of superhuman AI scientists working 24/7 without human limitations. These systems will collaborate across disciplines, build on each discovery instantly, and conduct experiments at unprecedented scale and speed — compressing a century of progress into years.

Will compares this to a mediaeval king suddenly needing to upgrade from bows and arrows to nuclear weapons to deal with an ideological threat from a kingdom he’s never heard of, while simultaneously learning he’s descended from monkeys and his god doesn’t exist.

What makes this acceleration perilous is that while technology can speed up almost arbitrarily, human institutions and decision making are much more fixed.

Consider the case of nuclear weapons: in this compressed timeline, there would have been just a three-month gap between the Manhattan Project’s start and the Hiroshima bombing, and the Cuban Missile Crisis would have lasted just over a day.

Robert Kennedy Sr, who helped navigate the actual Cuban Missile Crisis, once said that if they’d had to make decisions faster — like in 24 hours rather than 13 days — they would likely have taken much more aggressive, much riskier actions.

So there’s reason to worry about our capacity to make wise choices quickly. And in his paper, Will lays out 10 “grand challenges” we’ll need to navigate to avoid things going wrong.

Will now believes we’re entering one of the most critical periods for humanity ever — with decisions made in the next few years potentially determining outcomes millions of years into the future. In this wide-ranging conversation, Will and host Rob Wiblin discuss:

• Why leading AI safety researchers now think there’s dramatically less time before AI is transformative than they’d previously thought
• The three different types of intelligence explosions that occur in order
• Will’s list of resulting grand challenges — including destructive technologies, space governance, concentration of power, and digital rights
• How to prevent ourselves from accidentally “locking in” mediocre futures for all eternity
• Ways AI could radically improve human coordination and decision making
• Why we should aim for truly flourishing futures, not just avoiding extinction

Learn more and read the full transcript on the 80,000 Hours website.

This episode was originally released in March 2025.

Chapters:

• Cold open (00:00:00)
• Who’s Will MacAskill? (00:00:46)
• Why Will now just works on AGI (00:01:02)
• Will was wrong(ish) on AI timelines and hinge of history (00:04:10)
• A century of history crammed into a decade (00:08:59)
• Science goes super fast; our institutions don't keep up (00:15:41)
• Is it good or bad for intellectual progress to 10x? (00:21:03)
• An intelligence explosion is not just plausible but likely (00:22:53)
• Intellectual advances outside technology are similarly important (00:28:57)
• Counterarguments to intelligence explosion (00:31:31)
• The three types of intelligence explosion (software, technological, industrial) (00:37:29)
• The industrial intelligence explosion is the most certain and enduring (00:40:23)
• Is a 100x or 1,000x speedup more likely than 10x? (00:51:50)
• The grand superintelligence challenges (00:55:37)
• Grand challenge #1: Many new destructive technologies (00:59:17)
• Grand challenge #2: Seizure of power by a small group (01:06:45)
• Is global lock-in really plausible? (01:08:37)
• Grand challenge #3: Space governance (01:18:53)
• Is space truly defence-dominant? (01:28:43)
• Grand challenge #4: Morally integrating with digital beings (01:32:20)
• Will we ever know if digital minds are happy? (01:41:01)
• “My worry isn't that we won't know; it's that we won't care” (01:46:31)
• Can we get AGI to solve all these issues as early as possible? (01:49:40)
• Politicians have to learn to use AI advisors (02:02:03)
• Ensuring AI makes us smarter decision-makers (02:06:10)
• How listeners can speed up AI epistemic tools (02:09:38)
• AI could become great at forecasting (02:13:09)
• How not to lock in a bad future (02:14:37)
• AI takeover might happen anyway — should we rush to load in our values? (02:25:29)
• ML researchers are feverishly working to destroy their own power (02:34:37)
• We should aim for more than mere survival (02:37:53)
• By default the future is rubbish (02:49:04)
• No easy utopia (02:56:55)
• What levers matter most to utopia (03:06:32)
• Bottom lines from the modelling (03:20:09)
• People distrust utopianism; should they distrust this? (03:24:10)
• What conditions make eventual eutopia likely? (03:28:49)
• The new Forethought Centre for AI Strategy (03:37:21)
• How does Will resist hopelessness? (03:50:13)
• Outro (03:57:12)

Video editing: Simon Monsour
Audio engineering: Ben Cordell, Milo McGuire, Simon Monsour, and Dominic Armstrong
Camera operator: Jeremy Chevillotte
Transcriptions and web: Katy Moore

Ajeya Cotra argues this peculiar setup resembles the situation humanity finds itself in as we train very general and very capable AI models using current deep learning methods. Ajeya was a senior research analyst at Coefficient Giving at the time of this interview, and she now works at METR (Model Evaluation & Threat Research). 

As she explains, this eight-year-old faces a challenging problem. In the candidate pool there are likely some truly nice people, who sincerely want to help and make decisions that are in your interest. But there are probably other characters too — like people who will pretend to care while you’re monitoring them, but intend to exploit the job to enrich themselves as soon as they think they can get away with it.

Like a child trying to judge adults, at some point humans will need to judge the trustworthiness and reliability of machine learning models that are as goal-oriented as people, and greatly outclass them in knowledge, experience, breadth, and speed. Tricky!

Can’t we rely on models' performance during training tasks to guide us? Ajeya worries this won’t work. The trouble is that three different sorts of models will all produce the same output during training, but could behave very differently once deployed in a setting that allows their true colours to come through. She describes three such motivational archetypes:

• Saints — models that care about doing what we really want
• Sycophants — models that just want us to say they’ve done a good job, even if they get that praise by taking actions they know we wouldn’t want them to
• Schemers — models that don’t care about us or our interests at all, who are just pleasing us so long as that serves their own agenda

In principle, a machine learning training process based on reinforcement learning could spit out any of these three attitudes, because all three would perform roughly equally well on the tests we give them, and ‘performs well on tests’ is how these models are selected.

But while that’s true in principle, maybe it’s not something that could plausibly happen in the real world. After all, if we train an agent based on positive reinforcement for accomplishing X, shouldn’t the training process produce a model that just does X and doesn’t have complex thoughts and goals beyond that?

According to Ajeya, this is one thing we don’t know, and should be trying to test empirically as these models get more capable. For reasons she explains in the interview, the Sycophant or Schemer models may in fact be simpler and easier for the learning algorithm to creep towards than their Saint counterparts.

But there are also ways we could end up actively selecting for motivations that we don’t want.

For example, let’s say you train an agentic AI model to run a small business, selecting for behaviours that make money and measuring success by the balance in its bank account. During training, a highly capable model may experiment with the strategy of tricking its trainers into thinking it has made money legitimately when it hasn’t. Maybe instead it steals some money and covers that up. This isn’t a hypothetical worry: models often come up with creative — sometimes undesirable — approaches during training that their developers didn’t anticipate.

If such deception isn’t caught, a model like this may be rated as particularly successful, and the training process will reinforce its tendency to engage in deceptive behaviour. A model that could deceive without being caught would, in effect, have a competitive advantage.

What if deception is picked up, but just some of the time? Would the model then learn that honesty is the best policy? Perhaps. But it might learn a different lesson instead: that deception does pay, as long as it’s done selectively and carefully enough to avoid detection. Would that actually happen? We don’t yet know, but it’s possible.

In this conversation, Ajeya and host Rob Wiblin discuss the above, as well as:

• How to predict the motivations a neural network will develop through training
• Whether AIs in training will functionally understand that they’re AIs being trained
• Stories of AI misalignment that Ajeya doesn’t buy
• Analogies for AI, from octopuses to aliens to can openers
• Why it’s smarter to have separate ‘planning AIs’ and ‘doing AIs’
• The benefits of only following through on AI-generated plans that make sense to human beings
• Which approaches for fixing alignment problems Ajeya is most excited about, and which she thinks are overrated
• How one might demo actually scary AI failure mechanisms

Learn more and read the full transcript on the 80,000 Hours website.

This episode was originally released in May 2023, but we still think it’s one of the best episodes we have at explaining core risks from power-seeking AI.

Chapters:

• Rob’s intro (00:00:00)
• The interview begins (00:02:38)
• How Ajeya’s views have changed since 2020 (00:05:09)
• Are neural networks more like a sped-up version of evolution, or a slower version of human learning? (00:17:42)
• Situational awareness (00:26:10)
• Misalignment stories Ajeya doesn't buy (00:42:03)
• The orphan heir with a trillion-dollar fortune (00:59:14)
• Saints, Sycophants, and Schemers (01:03:41)
• Ways to train safer AI systems (01:23:20)
• Aliens and other analogies (01:38:22)
• Moral patienthood (01:53:21)
• ARC Evaluations (01:55:35)
• Interpretability research (02:09:25)
• Rewarding models based on how good and sensible their plans seem to us (02:17:48)
• Overrated approaches (02:25:49)
• Demos of actually scary alignment failures (02:30:57)
• Skills to develop for doing useful work (02:37:23)
• Rob’s outro (02:47:24)

Producer: Keiran Harris
Audio mastering: Ryan Kessler and Ben Cordell
Transcriptions: Katy Moore

Many people have sort of considered this hypothetical, but perhaps nobody has followed through and considered all the implications as much as Carl Shulman. Behind the scenes, his work has greatly influenced how leaders in artificial general intelligence (AGI) picture the world they’re creating.

Carl simply follows the logic to its natural conclusions, leading to a world where:

• $0.01 of electricity can be turned into medical advice, company management, or scientific research that would cost hundreds of dollars today, resulting in a scramble to manufacture chips and apply them to the most lucrative forms of intellectual labour
• Given their incredible hourly salaries, the supply of outstanding AI researchers quickly goes from 10,000 to 10 million or more, enormously accelerating progress in the field
• Companies operated entirely by AIs are much faster and more cost effective than those that lean on humans for decision making, and the latter are progressively driven out of business
• The technical challenges of controlling robots are rapidly overcome — leading to strong, fast, precise, and tireless robot workers able to accomplish any physical work the economy requires, and a rush to build billions of them and cash in
• Overnight, the number of humans becomes irrelevant to economic growth, which is now driven by how quickly the entire machine economy can replicate its components. Given how quickly complex biological systems can reproduce — some in a matter of days — a doubling every few months may be a conservative estimate
• Any country that delays participating in this economic explosion risks being outpaced and ultimately disempowered by rivals whose economies grow to be 10-fold, 100-fold, and then 1,000-fold larger than its own

As the economy grows, each person could afford the equivalent of a team of hundreds of machine ‘people’ to help them with every aspect of their lives.

And with growth rates this high, it doesn’t take long to run up against Earth’s physical limits — the toughest to engineer around being Earth’s ability to release waste heat. If this machine economy and its insatiable demand for power generates more heat than the Earth radiates into space, the planet will rapidly heat up and become uninhabitable for humans and other animals.

This eventually creates pressure to move economic activity off-planet. There’s little need for computer chips to be on Earth, and solar energy and minerals are more abundant in space. So you could develop populations of billions of scientific researchers operating on computer chips orbiting in space, sending the results of their work — such as drug designs — back to Earth.

These are just some of the wild implications if AGI could accomplish everything the most productive humans can, using the same energy supply.

In this interview with host Rob Wiblin, Carl explains the above, and Rob pushes back on whether that’s realistic or just a cool story:

• If we’re heading towards the above, how come economic growth remains slow now and isn’t really increasing?
• Why have computers and computer chips had so little effect on economic productivity so far?
• Are self-replicating biological systems a good comparison for self-replicating machine systems?
• Isn’t this just too crazy and weird to be plausible?
• What bottlenecks would be encountered in supplying energy and natural resources to this growing economy?
• Might there not be severely declining returns to bigger ‘brains’ and more training?
• Wouldn’t humanity get scared and pull the brakes if such a transformation kicked off?
• If this is right, how come economists don’t agree?

For the last section of the episode, Carl addresses the moral status of machine minds themselves. Would they be conscious or otherwise have a claim to moral rights? And how might humans and machines coexist with neither side dominating or exploiting the other?

Learn more and read the full transcript on the 80,000 Hours website.

This episode is the first part of Rob’s marathon interview with Carl Shulman in 2024. The second episode is on government and society after AGI, and you can listen to them in either order.

Chapters:

• Cold open (00:00:00)
• Rob's intro (00:01:00)
• The interview begins (00:04:43)
• Transitioning to a world where AI systems do almost all the work (00:05:20)
• Economics after an AI explosion (00:14:24)
• Objection: Shouldn’t we be seeing economic growth rates increasing today? (00:59:11)
• Objection: Speed of doubling time (01:07:32)
• Objection: Declining returns to increases in intelligence? (01:11:58)
• Objection: Physical transformation of the environment (01:17:37)
• Objection: Should we expect an increased demand for safety and security? (01:29:13)
• Objection: “This sounds completely whack” (01:36:09)
• Income and wealth distribution (01:48:01)
• Economists and the intelligence explosion (02:13:30)
• Baumol effect arguments (02:19:11)
• Denying that robots can exist (02:27:17)
• Semiconductor manufacturing (02:32:06)
• Classic economic growth models (02:36:10)
• Robot nannies (02:48:25)
• Slow integration of decision-making and authority power (02:57:38)
• Economists’ mistaken heuristics (03:01:06)
• Moral status of AIs (03:11:44)
• Rob's outro (04:11:46)

Producer and editor: Keiran Harris
Audio engineering lead: Ben Cordell
Technical editing: Simon Monsour, Milo McGuire, and Dominic Armstrong
Transcriptions: Katy Moore

That’s the view of Rose Hadshar, researcher at Forethought Research, who believes we could see extreme, AI-enabled power concentration without a coup or dramatic ‘end of democracy’ moment.

She foresees something more insidious: an elite group with access to such powerful AI capabilities that the normal mechanisms for checking elite power — law, elections, public pressure, the threat of strikes — cease to have much effect. Those mechanisms could continue to exist on paper, but become ineffectual in a world where humans are no longer needed to execute even the largest-scale projects.

Almost nobody wants this to happen — but we may find ourselves unable to prevent it.

If AI disrupts our ability to make sense of things, will we even notice power getting severely concentrated, or be able to resist it? Once AI can substitute for human labour across the economy, what leverage will citizens have over those in power? And what does all of this imply for the institutions we’re relying on to prevent the worst outcomes?

Rose has answers, and they’re not all reassuring.

But she’s also hopeful we can make society more robust against these dynamics. We’ve got literally centuries of thinking about checks and balances to draw on. And there are some interventions she’s excited about — like building sophisticated AI tools for making sense of the world, or ensuring multiple branches of government have access to the best AI systems.

In this conversation, Rose and host Zershaaneh Qureshi discuss all of this, and more:

• Three dynamics that could reshape political power in the AI era
• How AI gives small groups the productive power of millions
• Why AI-powered tyranny would be uniquely difficult to topple
• How power concentration compares to ‘gradual disempowerment’ by AI 
• Slower-moving scenarios that could still get scary 
• Which interventions could genuinely work — and which could backfire
• Rose's most promising approaches to fighting back against power concentration
• Why a ‘Manhattan Project’ approach to AI should worry you — and why international projects aren’t automatically safe either

Learn more and read the full transcript on the 80,000 Hours website.

This episode was originally released in March 2026.

Chapters:

• Cold open (00:00:00)
• Who’s Rose Hadshar? (00:01:05)
• Three dynamics that could reshape political power in the AI era (00:02:37)
• AI gives small groups the productive power of millions (00:12:49)
• Dynamic 1: When a software update becomes a power grab (00:20:41)
• Dynamic 2: When AI labour means governments no longer need their citizens (00:31:20)
• How democracy could persist in name but not substance (00:45:15)
• Dynamic 3: When AI filters our reality (00:54:54)
• Good intentions won’t stop power concentration (01:08:27)
• Slower-moving worlds could still get scary (01:23:57)
• Why AI-powered tyranny will be tough to topple (01:31:53)
• How power concentration compares to “gradual disempowerment” (01:38:18)
• Some interventions are cross-cutting — and others could backfire (01:43:54)
• What fighting back actually looks like (01:55:15)
• Why power concentration researchers should avoid getting too “spicy” (02:04:10)
• Why the “Manhattan Project” approach should worry you — but truly international projects might not be safe either (02:09:18)
• Rose wants to keep humans around! (02:12:06)

Video and audio editing: Dominic Armstrong, Milo McGuire, Luke Monsour, and Simon Monsour
Music: CORBIT
Coordination, transcripts, and web: Nick Stockton and Katy Moore

In her role as a founder, and now leader, of DC’s top think tank focused on the geopolitical and military implications of AI, Helen has been closely tracking the US’s AI diplomacy since 2019.

“Over the last couple of years there have been some direct [US–China] talks on some small number of issues, but they’ve also often been completely suspended.” China knows the US wants to talk more, so “that becomes a bargaining chip for China to say, ‘We don’t want to talk to you. We’re not going to do these military-to-military talks about extremely sensitive, important issues, because we’re mad.'”

Helen isn’t sure the groundwork exists for productive dialogue in any case. “At the government level, [there’s] very little agreement” on what artificial general intelligence (AGI) is, whether it’s possible soon, and whether it poses major risks. Without shared understanding of the problem, negotiating solutions is very difficult.

Another issue is that so far the Chinese Communist Party doesn’t seem especially ‘AGI-pilled.’ While a few Chinese companies like DeepSeek are betting on scaling, she sees little evidence Chinese leadership shares Silicon Valley’s conviction that AGI will arrive any minute now, and export controls have made it very difficult for them to access compute to match US competitors.

To find an autocracy that’s truly bought into the AGI vision, we might need to look at nominal US allies. The US has approved massive data centres in the UAE and Saudi Arabia with “hundreds of thousands of next-generation Nvidia chips” — delivering colossal levels of computing power.

When OpenAI announced this deal with the UAE, they celebrated that it was “rooted in democratic values,” and would advance “democratic AI rails” and provide “a clear alternative to authoritarian versions of AI.”

But the UAE scores 18 out of 100 on Freedom House’s democracy index. “This is really not a country that respects rule of law,” Helen observes. Political parties are banned, elections are fake, dissidents are persecuted.

If AI access really determines future national power, handing world-class supercomputers to Gulf autocracies seems pretty questionable. The justification is typically that “if we don’t sell it, China will” — a transparently false claim, given severe Chinese production constraints. It also raises eyebrows that Gulf countries conduct joint military exercises with China and their rulers have “very tight personal and commercial relationships with Chinese political leaders and business leaders.”

In this conversation in Washington, DC, host Rob Wiblin and Helen discuss the above, plus much more.

Learn more and read the full transcript on the 80,000 Hours website.

This episode was originally released in November 2025.

Chapters:

• Cold open (00:00:00)
• Who’s Helen Toner? (00:01:02)
• Helen’s role on the OpenAI board, and what happened with Sam Altman (00:01:31)
• The Center for Security and Emerging Technology (CSET) (00:07:35)
• CSET’s role in export controls against China (00:10:43)
• Does it matter if the world uses US AI models? (00:21:24)
• Is China actually racing to build AGI? (00:27:10)
• Could China easily steal AI model weights from US companies? (00:38:14)
• The next big thing is probably robotics (00:46:42)
• Why is the Trump administration sabotaging the US high-tech sector? (00:48:17)
• Are data centres in the UAE “good for democracy”? (00:51:31)
• Will AI inevitably concentrate power? (01:06:20)
• “Adaptation buffers” vs non-proliferation (01:28:16)
• Will the military use AI for decision-making? (01:36:09)
• “Alignment” is (usually) a terrible term (01:42:51)
• Is Congress starting to take superintelligence seriously? (01:45:19)
• AI progress isn't actually slowing down (01:47:44)
• What's legit vs not about OpenAI’s restructure (01:55:28)
• Is Helen unusually “normal”? (01:58:57)
• How to keep up with rapid changes in AI and geopolitics (02:02:42)
• What CSET can uniquely add to the DC policy world (02:05:51)
• Talent bottlenecks in DC (02:13:26)
• What evidence, if any, could settle how worried we should be about AI risk? (02:16:28)
• Is CSET hiring? (02:18:22)

Video editing: Luke Monsour and Simon Monsour
Audio engineering: Milo McGuire, Simon Monsour, and Dominic Armstrong
Music: CORBIT
Coordination, transcriptions, and web: Katy Moore

These are substantial, multi-step tasks requiring sustained focus: building web applications, conducting machine learning research, or solving complex programming challenges.

Beth Barnes is CEO of METR (Model Evaluation & Threat Research) — the leading organisation measuring these capabilities. Beth’s team has been timing how long it takes skilled humans to complete projects of varying length, then seeing how AI models perform on the same work.

The resulting paper from METR, “Measuring AI ability to complete long tasks,” made waves by revealing that the planning horizon of AI models was doubling roughly every seven months. It’s regarded by many as the most useful AI forecasting work in years.

The companies building these systems aren’t just aware of this trend — they want to harness it as much as possible, and are aggressively pursuing automation of their own research.

That’s both an exciting and troubling development, because it could radically speed up advances in AI capabilities, accomplishing what would have taken years or decades in just months. That itself could be highly destabilising (as we explored in a previous episode in this series: Will MacAskill on AI causing a “century in a decade” — and how we’re completely unprepared).

And having AI models rapidly build their successors with limited human oversight naturally raises the risk that things could go off the rails, if the models at the end of the process lack the goals and constraints we hoped for.

Beth thinks models can already do “meaningful work” on improving themselves, and she wouldn’t be surprised if AI models were able to autonomously self-improve in as little as two years — in fact, she says: “It seems hard to rule out even shorter [timelines]. Is there 1% chance of this happening in six, nine months? Yeah, that seems pretty plausible.”

While Silicon Valley is abuzz with these numbers, policymakers remain largely unaware of what’s barrelling toward us — and given the current lack of regulation of AI companies, they’re not even able to access the critical information that would help them decide whether to intervene. 

Beth adds: “The sense I really want to dispel is, ‘But the experts must be on top of this. The experts would be telling us if it really was time to freak out.’ The experts are not on top of this. Inasmuch as there are experts, they are saying that this is concerning. … And to the extent that I am an expert, I am an expert telling you you should freak out. And there’s not especially anyone else who isn’t saying this.”

Beth and host Rob Wiblin discuss all that, plus much more.

Learn more and read the full transcript on the 80,000 Hours website.

This episode was originally released in June 2025.

Chapters:

• Cold open (00:00:00)
• Who is Beth Barnes? (00:01:19)
• Can we see AI scheming in the chain of thought? (00:01:52)
• The chain of thought is essential for safety checking (00:08:58)
• Alignment faking in large language models (00:12:24)
• We have to test model honesty even before they're used inside AI companies (00:16:48)
• We have to test models when unruly and unconstrained (00:25:57)
• It's essential to thoroughly test relevant real-world tasks (00:30:40)
• METR's research finds AIs are solid at AI research already (00:49:33)
• AI may turn out to be strong at novel and creative research (00:55:53)
• When can we expect an algorithmic 'intelligence explosion'? (00:59:11)
• Recursively self-improving AI might even be here in two years — which is alarming (01:05:02)
• Could evaluations backfire by increasing AI hype and racing? (01:11:36)
• Governments first ignore new risks, but can overreact once they arrive (01:26:38)
• Do we need external auditors doing AI safety tests, not just the companies themselves? (01:35:10)
• A case against safety-focused people working at frontier AI companies (01:48:44)
• The new, more dire situation has forced changes to METR's strategy (02:02:29)
• AI companies are being locally reasonable, but globally reckless (02:10:31)
• Overrated: Interpretability research (02:15:11)
• Underrated: Developing more narrow AIs (02:17:01)
• Underrated: Helping humans judge confusing model outputs (02:23:36)
• Overrated: Major AI companies' contributions to safety research (02:25:52)
• Could we have a science of translating AI models' nonhuman language or neuralese? (02:29:24)
• Could we ban using AI to enhance AI, or is that just naive? (02:31:47)
• Open-weighting models is often good, and Beth has changed her attitude to it (02:37:52)
• What we can learn about AGI from the nuclear arms race (02:42:25)
• Infosec is so bad that no models are truly closed-weight models (02:57:24)
• AI is more like bioweapons because it undermines the leading power (03:02:02)
• What METR can do best that others can't (03:12:09)
• What METR isn't doing that other people have to step up and do (03:27:07)
• What research METR plans to do next (03:32:09)

Video editing: Luke Monsour and Simon Monsour
Audio engineering: Ben Cordell, Milo McGuire, Simon Monsour, and Dominic Armstrong
Music: Ben Cordell
Transcriptions and web: Katy Moore

That alone is remarkable. But as Dr Richard Moulange, one of the world’s top experts on AI biosecurity, explains: it’s just one of many data points showing how AI is dissolving the barriers that have historically kept biological weapons out of reach.

For years, experts have reassured us that ‘tacit knowledge’ — the hands-on, hard-to-Google lab skills needed to work with dangerous pathogens — would prevent bad actors from weaponising biology. So far, they’ve been right.

But now that reassurance is crumbling. The Virology Capabilities Test measures exactly this kind of troubleshooting expertise, and finds that modern AI models crushed top human virologists even in their self-declared area of greatest specialisation and expertise — with scores of 45% compared to the experts’ 22%.

Meanwhile, Anthropic’s research shows that PhD-level biologists are getting meaningfully better at weapons-relevant tasks with AI assistance — and the effect is growing with each new model generation.

In this conversation, Richard and host Rob Wiblin discuss:

• What AI biology tools already exist
• Why mid-tier actors (not amateurs) are the ones getting the most dangerous boost
• The three main categories of defence we can pursue
• Whether there’s a plausible path to a world where engineered pandemics become a thing of the past.

Learn more and read the full transcript on the 80,000 Hours website.

This episode was recorded on January 16, 2026. Since recording this episode, Richard has seconded to the UK Government — please note that his views expressed here are entirely his own.

Chapters:

• Cold open (00:00:00)
• Who’s Richard Moulange? (00:00:31)
• AI can now design novel viruses (00:01:11)
• The end of the 'tacit knowledge' barrier (00:04:34)
• Are risks from bioterrorists overstated? (00:18:20)
• The 3 key disasters AI makes more likely (00:22:41)
• Which bad actors does AI help the most? (00:30:03)
• Experts are more scary than amateurs (00:41:17)
• Barriers to bioterrorists using AI (00:46:43)
• AI biorisks are sometimes dismissed (and that’s a huge mistake) (00:48:54)
• Advanced AI biology tools we already have or will soon (01:04:10)
• Rob argues that the situation is hopeless (01:09:49)
• Intervention #1: Limit access (01:18:16)
• Intervention #2: Get AIs to refuse to help (01:32:58)
• Intervention #3: Surveillance and attribution (01:42:38)
• Intervention #4: Universal vaccines and antivirals (01:56:38)
• Intervention #5: Screen all orders for DNA (02:10:00)
• AI companies talk about def/acc more than they fund it (02:19:52)
• Can you build a profitable business solving this problem? (02:26:32)
• This doesn't have to interfere with useful science (much) (02:30:56)
• What are the best low-tech interventions? (02:33:01)
• Richard's top request for AI companies (02:37:59)
• Grok shows governments lack many legal levers (02:53:17)
• Best ways listeners can help fix AI-Bio (02:56:24)
• We might end all contagious disease in 20 years (03:03:37)

Video and audio editing: Dominic Armstrong, Milo McGuire, Luke Monsour, and Simon Monsour
Music: CORBIT
Camera operator: Jeremy Chevillotte
Transcripts and web: Elizabeth Cox and Katy Moore

Robert Long founded Eleos AI to explore questions like these, on the basis that AI may one day be capable of suffering — or perhaps already is. In this episode, Robert and host Luisa Rodriguez explore the many ways in which AI consciousness may be very different from anything we’re used to.

Things get strange fast: if AI is conscious, where does that consciousness exist? In the base model? A chat session? A single forward pass? If you close the chat, is the AI asleep or dead?

To Robert, these kinds of questions aren’t just philosophical exercises. Not being clear on AI’s moral status as it transitions from human-level to superhuman intelligence could be dangerous: 

• If we’re too dismissive, we risk unintentionally exploiting sentient beings. 
• If we’re too sympathetic, we might rush to ‘liberate’ AI systems in ways that make them harder to control — worsening existential risk from power-seeking AIs.

Robert argues the right path is doing the empirical and philosophical homework now, while the stakes are still manageable.

The field is tiny. Eleos AI is three people. As a result, Robert argues that driven researchers with a willingness to venture into uncertain territory can push out the frontier on these questions remarkably quickly.

In this interview, Robert and Luisa talk through the above, and much more.

Learn more and read the full transcript on the 80,000 Hours website.

This episode was originally released in March 2026.

Chapters:

• Cold open (00:00:00)
• Who’s Robert Long? (00:00:42)
• How AIs are (and aren't) like farmed animals (00:01:18)
• If AIs love their jobs… is that worse? (00:11:05)
• Are LLMs just playing a role, or feeling it too? (00:31:58)
• Do AIs die when the chat ends? (00:55:09)
• Studying AI welfare empirically: behaviour, neuroscience, and development (01:27:34)
• Why Eleos spent weeks talking to Claude despite knowing it's unreliable (01:51:58)
• Can LLMs learn to introspect? (01:57:58)
• Mechanistic interpretability as AI neuroscience (02:08:01)
• Does consciousness require biological materials? (02:31:06)
• Eleos’s work & building the playbook for AI welfare (02:50:36)
• Avoiding the trap of wild speculation (03:18:15)
• Robert's top research tip: don't do it alone (03:22:43)

Video and audio editing: Dominic Armstrong, Milo McGuire, Luke Monsour, and Simon Monsour
Music: CORBIT
Coordination, transcripts, and web: Katy Moore

Neel Nanda of Google DeepMind is one of the founders of mechanistic interpretability (or ‘mech interp’) — the field of machine learning trying to give us insight into what’s happening inside AI models. The project has generated enormous hype, exploding from a handful of researchers five years ago to hundreds today — all working to make sense of the jumble of tens of thousands of numbers that frontier AIs use to process information and decide what to say or do.

Neel now has a warning for us: the most ambitious vision of mech interp he once dreamed of is probably dead. He doesn’t see a path to deeply and reliably understanding what AIs are thinking. The technical and practical barriers are simply too great to get us there in time, before competitive pressures push us to deploy human-level or superhuman AIs. Indeed, Neel argues no single approach will guarantee alignment, and our only choice is the “Swiss cheese” model of protection, layering multiple safeguards on top of one another.

But while mech interp won’t be a silver bullet for AI safety, it has nevertheless had some major successes and will be one of the best tools in our arsenal.

For instance, by inspecting the neural activations in the middle of an AI’s thoughts, we can pick up many of the concepts the model is thinking about — from the Golden Gate Bridge, to refusing to answer a question, to the option of deceiving the user. While we can’t track every thought a model is having at every moment, picking up 90% of the concepts it uses 90% of the time should help us muddle through — so long as mech interp is paired with other techniques to fill in the gaps.

In this episode, Neel takes us on a tour of everything you’ll want to know about this race to understand what AIs are really thinking. He and host Rob Wiblin cover:

• The best tools we’ve come up with so far, and where mech interp has failed
• Why the best techniques have to be fast and cheap
• The fundamental reasons we can’t reliably know what AIs are thinking, despite having perfect internal access to them
• What we can and can’t learn by reading models’ ‘chains of thought’
• Whether models will be able to trick us when they realise they’re being tested
• The best protections to add on top of mech interp
• Why he thinks the hottest technique in the field (SAEs) are overrated
• His new research philosophy
• How to break into mech interp and get a job — including applying to be a MATS scholar with Neel as your mentor

Learn more and read the full transcript on the 80,000 Hours website.

This episode was originally released in September 2025.

Chapters:

• Cold open (00:00:00)
• Who’s Neel Nanda? (00:01:02)
• How would mechanistic interpretability help with AGI (00:01:59)
• What's mech interp? (00:05:09)
• How Neel changed his take on mech interp (00:09:47)
• Top successes in interpretability (00:15:53)
• Probes can cheaply detect harmful intentions in AIs (00:20:06)
• In some ways we understand AIs better than human minds (00:26:49)
• Mech interp won't solve all our AI alignment problems (00:29:21)
• Why mech interp is the 'biology' of neural networks (00:38:07)
• Interpretability can't reliably find deceptive AI – nothing can (00:40:28)
• 'Black box' interpretability — reading the chain of thought (00:49:39)
• 'Self-preservation' isn't always what it seems (00:53:06)
• For how long can we trust the chain of thought (01:02:09)
• We could accidentally destroy chain of thought's usefulness (01:11:39)
• Models can tell when they’re being tested and act differently (01:16:56)
• Top complaints about mech interp (01:23:50)
• Why everyone's excited about sparse autoencoders (SAEs) (01:37:52)
• Limitations of SAEs (01:47:16)
• SAEs performance on real-world tasks (01:54:49)
• Best arguments in favour of mech interp (02:08:10)
• Lessons from the hype around mech interp (02:12:03)
• Where mech interp will shine in coming years (02:17:50)
• Why focus on understanding over control (02:21:02)
• If AI models are conscious, will mech interp help us figure it out (02:24:09)
• Neel’s new research philosophy (02:26:19)
• Who should join the mech interp field (02:38:31)
• Advice for getting started in mech interp (02:46:55)
• Keeping up to date with mech interp results (02:54:41)
• Who’s hiring and where to work? (02:57:43)

Video editing: Simon Monsour, Luke Monsour, Dominic Armstrong, and Milo McGuire
Audio engineering: Ben Cordell, Milo McGuire, Simon Monsour, and Dominic Armstrong
Music: Ben Cordell
Camera operator: Jeremy Chevillotte
Coordination, transcriptions, and web: Katy Moore

According to Holden Karnofsky — currently at Anthropic, previously cofounder and CEO of Open Philanthropy (now Coefficient Giving) — this situation has now reversed completely.

There are now large amounts of useful, concrete, shovel-ready projects with clear goals and deliverables. Holden thinks people haven’t appreciated the scale of the shift, and wants everyone to see the large range of “well-scoped object-level work” they could personally help with, in both technical and non-technical areas. 

In fact, in this episode alone, Holden lists 39 projects he’s excited to see happening, including:

• Training deceptive AI models to study deception and how to detect it
• Developing classifiers to block jailbreaking
• Implementing security measures to stop ‘backdoors’ or ‘secret loyalties’ from being added to models during training
• Developing policies on model welfare, AI-human relationships, and what instructions to give models
• Training AIs to work as alignment researchers

And that’s all just stuff he’s happened to observe directly, which is probably only a small fraction of the options available.

All this low-hanging fruit is one factor behind his decision to join Anthropic this year. That said, his wife is also a cofounder and president of the company, giving him a big financial stake in its success — and making it impossible for him to be seen as independent no matter where he worked.

Holden makes a case that, for many people, working at an AI company like Anthropic will be the best way to steer AGI in a positive direction. He notes there are “ways that you can reduce AI risk that you can only do if you’re a competitive frontier AI company.” At the same time, he believes external groups have their own advantages and can be equally impactful.

Critics worry that Anthropic’s efforts to stay at that frontier encourage competitive racing towards AGI — significantly or entirely offsetting any useful research they do. Holden thinks this seriously misunderstands the strategic situation we’re in: “I work at an AI company, and a lot of people think that’s just inherently unethical. They’re imagining that everyone wishes they could go slowly, but they’re going fast so they can beat everyone else. […] But I emphatically think this is not what’s going on in AI.”

The reality, in Holden’s view:

“I think there’s too many players in AI who […] don’t want to slow down. They don’t believe in the risks. Maybe they don’t even care about the risks. […] If Anthropic were to say, ‘We’re out, we’re going to slow down,’ they would say, ‘This is awesome! Now we have a better chance of winning, and this is even good for our recruiting’ — because they have a better chance of getting people who want to be on the frontier and want to win.”

Holden believes a frontier AI company can reduce risk by:

• Developing cheap, practical safety measures other companies might adopt
• Prototyping policies regulators could mandate
• Gathering crucial data about what advanced AI can actually do

Host Rob Wiblin and Holden discuss the case for and against those strategies, and much more.

Learn more and read the full transcript on the 80,000 Hours website.

This episode was originally released in October 2025.

Chapters:

• Cold open (00:00:00)
• Holden is back! (00:02:26)
• An AI Chernobyl we never notice (00:02:56)
• Is rogue AI takeover easy or hard? (00:07:32)
• The AGI race isn't a coordination failure (00:17:48)
• What Holden now does at Anthropic (00:28:04)
• The case for working at Anthropic (00:30:08)
• Is Anthropic doing enough? (00:40:45)
• Can we trust Anthropic, or any AI company? (00:43:40)
• How can Anthropic compete while paying the “safety tax”? (00:49:14)
• What, if anything, could prompt Anthropic to halt development of AGI? (00:56:11)
• Holden's retrospective on responsible scaling policies (00:59:01)
• Overrated work (01:14:27)
• Concrete shovel-ready projects Holden is excited about (01:16:37)
• Great things to do in technical AI safety (01:20:48)
• Great things to do on AI welfare and AI relationships (01:28:18)
• Great things to do in biosecurity and pandemic preparedness (01:35:11)
• How to choose where to work (01:35:57)
• Overrated AI risk: Cyberattacks (01:41:56)
• Overrated AI risk: Persuasion (01:51:37)
• Why AI R&D is the main thing to worry about (01:55:36)
• The case that AI-enabled R&D wouldn't speed things up much (02:07:15)
• AI-enabled human power grabs (02:11:10)
• Main benefits of getting AGI right (02:23:07)
• The world is handling AGI about as badly as possible (02:29:07)
• Learning from targeting companies for public criticism in farm animal welfare (02:31:39)
• Will Anthropic actually make any difference? (02:40:51)
• “Misaligned” vs “misaligned and power-seeking” (02:55:12)
• Success without dignity: how we could win despite being stupid (03:00:58)
• Holden sees less dignity but has more hope (03:08:30)
• Should we expect misaligned power-seeking by default? (03:15:58)
• Will reinforcement learning make everything worse? (03:23:45)
• Should we push for marginal improvements or big paradigm shifts? (03:28:58)
• Should safety-focused people cluster or spread out? (03:31:35)
• Is Anthropic vocal enough about strong regulation? (03:35:56)
• Is Holden biased because of his financial stake in Anthropic? (03:39:26)
• Have we learned clever governance structures don't work? (03:43:51)
• Is Holden scared of AI bioweapons? (03:46:12)
• Holden thinks AI companions are bad news (03:49:47)
• Are AI companies too hawkish on China? (03:56:39)
• The frontier of infosec: confidentiality vs integrity (04:00:51)
• How often does AI work backfire? (04:03:38)
• Is AI clearly more impactful to work in? (04:18:26)
• What's the role of earning to give? (04:24:54)

Video editing: Simon Monsour, Luke Monsour, Dominic Armstrong, and Milo McGuire
Audio engineering: Milo McGuire, Simon Monsour, and Dominic Armstrong
Music: CORBIT
Coordination, transcriptions, and web: Katy Moore

But what led us to this conclusion? Could AI really cause human extinction? We’re not certain, but we think the risk is worth taking very seriously. 

In particular, as companies create increasingly powerful AI systems, there’s a concerning chance that:

• These AI systems may develop dangerous long-term goals we don’t want.
• To pursue these goals, they may seek power and undermine the safeguards meant to contain them.
• They may even aim to disempower humanity and potentially cause our extinction.

This article is written by Cody Fenwick and Zershaaneh Qureshi, and narrated by Zershaaneh Qureshi. It discusses why future AI systems could disempower humanity, what current AI research reveals about behaviours like power-seeking and deception, and how you can help mitigate the dangers.

You can see the original article — packed with graphs, images, footnotes, and further resources — on the 80,000 Hours website: 

https://80000hours.org/problem-profiles/risks-from-power-seeking-ai/ 

Chapters:

• Risks from power-seeking AI systems (00:01:00)
• Introduction (00:01:17)
• Summary (00:03:09)
• Why are the risks from power-seeking AI a pressing world problem? (00:04:04)
• Section 1: Humans will likely build advanced AI systems with long-term goals (00:05:43)
• Section 2: AIs with long-term goals may be inclined to seek power (00:11:32)
• Section 3: These power-seeking AI systems could successfully disempower humanity (00:26:26)
• Section 4. People might create power-seeking AI systems without enough safeguards, despite the risks (00:38:34)
• Section 5: Work on this problem is neglected and tractable (00:47:37)
• Section 6: What are the arguments against working on this problem? (00:59:20)
• Section 7: How you can help (01:25:07)
• Thank you for listening (01:28:56)

Audio editing: Dominic Armstrong
Production: Zershaaneh Qureshi, Elizabeth Cox, and Katy Moore

In this series, one expert interview at a time, we'll walk you through what's at stake — and what you could do to help.