p(doom)

What Is p(doom)?

p(doom) is a subjective estimate of the probability that advanced artificial intelligence causes human extinction or a permanent loss of human control over the future. The “p” means probability, and “doom” is shorthand for an existentially catastrophic outcome, though people do not always define it the same way.

The term emerged in AI safety discussions as shorthand for a view about long-term risk. It is not a measured fact, model forecast, or scientific consensus. It is a personal judgment that combines beliefs about technological progress, institutions, malicious actors, and whether powerful systems can remain under meaningful control. Someone can see real AI risks while assigning a low p(doom), or favor urgent action while remaining uncertain about the number.

In this context, “AI doom” does not mean today’s common software tools are inherently dangerous. The discussion usually concerns hypothetical future systems with far greater capability, autonomy, and resource access than current systems. An AI glossary can help distinguish present-day applications from claims about advanced systems.

What Counts as “Doom” in AI Risk Discussions?

“Doom” needs a definition before any probability is useful. The strictest meaning is human extinction. A broader meaning includes permanent disempowerment, where humans survive but can no longer shape the long-term future because an AI system or AI-enabled power structure has lasting control.

Some speakers also include severe but non-existential outcomes, such as global conflict, biological misuse, critical infrastructure disruption, or irreversible authoritarian concentration of power. These are grave harms, but they differ analytically from extinction. An estimate is incomplete if it does not specify the outcome it covers.

Two people may state the same p(doom) while meaning different things. One may estimate extinction before a deadline, while another includes permanent loss of political agency over a longer period. The outcome definition and time horizon matter as much as the percentage.

How a p(doom) Estimate Is Formed

A careful estimate can be broken into conditional questions rather than treated as one intuition. This does not create certainty, but it makes assumptions easier to examine.

1. Define the outcome clearly, such as extinction, permanent disempowerment, or a specified global catastrophe.
2. Set a time horizon, because a claim about the next decade differs from one about the next century.
3. Estimate whether highly capable AI is likely to be developed within that period, including uncertainty about progress and deployment incentives.
4. Assess pathways from capability to harm, including loss of control, deliberate misuse, cyber compromise, dangerous scientific assistance, or destabilizing competition.
5. Assess safeguards, such as technical evaluations, access controls, monitoring, incident response, international coordination, and enforceable governance.
6. Combine judgments cautiously, noting that factors may interact rather than acting as independent measurements.

This framework separates “Will powerful AI be built?” from “If it is built, will it cause catastrophe?” A low probability of development can coexist with high conditional safety concern, while rapid development does not automatically imply catastrophe.

Key Assumptions Behind p(doom)

Every p(doom) estimate rests on assumptions. These include whether AI systems will gain broad reasoning, planning, scientific, or operational abilities, and whether they can pursue multi-step goals with limited human supervision.

Other assumptions concern alignment, meaning whether an AI system reliably follows intended goals and constraints. They also concern access. A capable system without permissions, computing resources, financial channels, physical tools, or human collaborators has fewer routes to cause harm than one with broad real-world access.

Governance assumptions are equally important: whether organizations test systems before deployment, governments can monitor dangerous capabilities, competitors coordinate on safety, and malicious users can obtain powerful models. An AI app builder is a tool category for bounded development tasks, not evidence of independent strategic control.

p(doom) vs. Other AI Risk Measures

p(doom) captures one narrow class of long-term concern. Other measures may be more useful for operational decisions because they address harms that are easier to observe, prevent, and manage.

Why People Use p(doom)

The term is useful when it starts a disciplined conversation rather than ends one. Its main value is communicating under substantial uncertainty.

• It provides shorthand for discussing severe AI risks without assuming certainty.
• It separates the likelihood of harm from emotionally vivid worst-case rhetoric.
• It can expose assumptions about timelines, technical control, misuse, and governance.
• It helps researchers, funders, and policymakers compare priorities when evidence is incomplete.
• It allows views to change as evaluations, incidents, and governance conditions change.

Practical Limits of p(doom)

p(doom) is an uncertain judgment, not a precise instrument. There is no direct historical record of societies developing and governing systems with the hypothesized capabilities.

• There is no reliable reference class for calculating a frequency of AI-caused existential catastrophe.
• Key factors are dependent, so capability growth, deployment pressure, and weak governance may reinforce one another.
• Definitions of “doom” vary, making similar estimates hard to compare.
• Time horizons can dominate the result because longer periods allow more opportunities for both harm and prevention.
• A stated number can be mistaken for a prediction, consensus, or scientific precision.

Common Pitfalls When Discussing p(doom)

Good risk communication makes uncertainty and scope explicit. Poor communication turns a conditional judgment into headline-sized certainty.

• Using decimal-level precision when assumptions are broad and disputed.
• Anchoring on a famous researcher’s or executive’s estimate instead of their reasoning.
• Treating disagreement as ignorance rather than differences in definitions, timelines, or assumptions.
• Mixing an unconditional estimate with risk conditional on highly capable AI being developed.
• Ignoring near-term harms, benefits, and practical safety work because long-term catastrophe gets more attention.
• Using one number instead of concrete evaluations, security practices, emergency plans, and governance measures.

Where p(doom) Is Used

p(doom) appears most often in long-term AI safety discussions, but it can inform decisions when used with more concrete evidence.

• AI safety research uses it to clarify which failure pathways seem most urgent.
• Organizational risk governance uses it alongside model evaluations, red-team findings, and deployment controls.
• Public policy debates use it to discuss appropriate precaution for advanced systems.
• Philanthropic organizations may use it when weighing investments in technical safety, governance, resilience, or other global priorities.
• Readers use it to interpret expert claims while recognizing that a personal estimate is not proof.

A More Useful Way to Read Any p(doom) Claim

When someone states a p(doom), ask what outcome they include and by when. Ask whether the number is conditional on advanced AI being developed, which harm mechanisms they consider plausible, and which safeguards they expect to succeed or fail. A strong claim should identify evidence that would change the estimate, such as better alignment evaluations, demonstrated dangerous capabilities, stronger access controls, or effective international agreements.

Finally, ask what action the speaker believes follows from the estimate. A useful risk judgment should lead to proportionate steps, not fatalism. p(doom) is a decision-relevant expression of uncertainty about severe AI outcomes. It can sharpen thinking, but cannot replace scenario analysis, technical testing, accountable governance, or preparation for harms already possible.