Abstract

This paper develops a taxonomy of expert perspectives on the risks and likely consequences of artificial intelligence, with particular focus on Artificial General Intelligence (AGI) and Artificial Superintelligence (ASI).

Drawing from primary sources, we identify three predominant doctrines:

• The dominance doctrine, which predicts that the first actor to create sufficiently advanced AI will attain overwhelming strategic superiority sufficient to cheaply neutralize its opponents’ defenses;
• The extinction doctrine, which anticipates that humanity will likely lose control of ASI, leading to the extinction of the human species or its permanent disempowerment;
• The replacement doctrine, which forecasts that AI will automate a large share of tasks currently performed by humans, but will not be so transformative as to fundamentally reshape or bring an end to human civilization.

We examine the assumptions and arguments underlying each doctrine, including expectations around the pace of AI progress and the feasibility of maintaining advanced AI under human control.

While the boundaries between doctrines are sometimes porous and many experts hedge across them, this taxonomy clarifies the core axes of disagreement over the anticipated scale and nature of the consequences of AI development.

1. Introduction

AI systems are progressing rapidly, with frontier models being released every few months. The field has experienced major breakthroughs in recent years: the development of GPT-3 marked a stark inflection point by demonstrating the effectiveness of scaling large language models.^{[1 The Economist]} This was soon followed by another breakthrough, Reinforcement Learning from Human Feedback, which enabled the development of conversational AI systems like ChatGPT^{[2 Yuntao Bai et al.]}. More recently, the field witnessed the emergence of reasoning models^{[3 OpenAI]}, which demonstrate superior performance over previous approaches in many tasks such as coding, mathematics and science.

Evidence suggests that AI systems are surpassing human experts across many fields. OpenAI's o3 model achieved competitive programming scores matching the 175th top contestant among 150,000 participants^{[4 OpenAI]}. More recently, an undisclosed OpenAI model has achieved second place at the AtCoder Heuristics World Finals.^{[5 OpenAI]} Research by METR indicates that AI systems' ability to autonomously complete complex tasks is advancing exponentially, with a doubling time of seven months, improving from 30-second tasks in March 2023 to 15-minute tasks by February 2025.^{[6 Thomas Kwa, Ben West, et al.]}

In this backdrop, divergent views have emerged about both the trajectory of the technology itself and its potential impact on the world. Many experts anticipate that we will soon develop AI systems that can automate most tasks performed by humans; such systems are commonly referred to as Artificial General Intelligence (AGI).^{[7 Sam Altman][8 Dario Amodei][9 Yoshua Bengio]} Some expect this will quickly be followed by the development of AI surpassing the brightest human minds across all domains, or even surpassing the combined intelligence of humanity — what experts call Artificial Superintelligence (ASI). Notably, the number of efforts explicitly aiming for ASI has increased markedly; this includes both research companies mainly or solely dedicated to developing ASI^{[10 Ilya Sutskever, Daniel Gross, Daniel Levy][11 Sam Altman]} as well as established tech companies like Meta and Alibaba.^{[12 New York Times][13 Bloomberg]}

Yet significant disagreement persists about whether these milestones are achievable in the near term. Likewise, there is intense debate about the magnitude of their potential impact, with predictions ranging from the establishment of a golden age of human flourishing to catastrophic outcomes such as the extinction of the human species. In this paper, we survey expert statements on these topics, with particular emphasis on primary sources, and study the implications of these divergent views on considerations around national and global security. Through this lens, we observe that these perspectives fit naturally into three categories.

The dominance doctrine, which predicts that the first actor to achieve ASI will gain a “decisive strategic advantage” over all other actors; that is, a position of strategic superiority sufficient to allow the actor to achieve overwhelming military and economic dominance over the rest of the world.

This doctrine expects that ASI will be developed in the near future, with some forecasting dates as early as 2026 to 2029. These systems will be able to accelerate two key domains. The first is AI R&D itself; once a sufficient advantage is gained in this area, the gap would become self-reinforcing, making it impossible for competitors to catch up. The second is the research, development, and large-scale operation of military technologies. The combination of these two capabilities means that the first actor to develop sufficiently advanced AI may gain a strategic advantage over all other actors so vast that it creates the possibility of executing strikes that would neutralize all of their defenses in a relatively cheap and risk-free manner. After achieving this position, such an actor may be able to maintain an unassailable world order.

The extinction doctrine, which predicts that humanity will lose control over ASI, likely leading to its extinction or permanent disempowerment. A wide range of pathways to this outcome have been theorized, ranging from ones where a single, monolithic superintelligent AI system suddenly performs a hostile takeover after initially acting cooperative, to ones where control is gradually ceded to AI systems as they systematically replace humans across all economic, political and social functions.

While the extinction doctrine largely agrees with the dominance doctrine regarding the expected pace and extent of AI development, its core thesis is that we are not on track to develop techniques to maintain control of ASI by the time we develop such systems. Given how powerful superintelligent AI systems are projected to be, it would be impossible to maintain or regain control of them once they are pursuing goals incompatible with human values and interests, eventually leading to catastrophic outcomes such as the extinction or permanent subjugation of all of humanity.

The replacement doctrine, a broad umbrella of views predicting that AI development will result in AI replacing humans in carrying out some or most of the tasks they currently perform, allowing them to be executed at much higher speed and scale and more cheaply. However, they generally agree that AI development will not usher in radically new capabilities that may overturn existing economic and geopolitical paradigms. In more concrete terms, AI will not enable the development of military technology that allows one actor to prevail over all others; nor will it bring about catastrophic outcomes like human extinction. Being the most heterogeneous of the three, the views of proponents of the replacement doctrine span from highly optimistic to pessimistic about AI's impact on society.

There is disagreement within this school of thought over the extent to which humans will be replaced in their roles as producers of economic value, with some expecting only partial automation of human labor and others envisioning near-complete replacement of humans in the economy. What characterizes the replacement doctrine is that even those anticipating the highest degrees of automation expect that AI will not be so transformative that we can no longer meaningfully speak of an “economy” existing, and that humans will keep participating in the economy even if only as capital holders and consumers.

2. Dominance doctrine

Proponents of the dominance doctrine believe that developing ASI is possible, and that there is a good chance that this will be achieved as soon as the next few years. ^{[14 Dan Hendrycks et al.][15 Dario Amodei][16 Leopold Aschenbrenner]}

According to this doctrine, the first actor to develop ASI will gain a "decisive strategic advantage" over all others — that is, a position of strategic superiority sufficient to allow it to achieve unilateral military and economic dominance over the rest of the world. ^{[17 Leopold Aschenbrenner][18 EA Forum]}

Central to this doctrine is the belief that, when ASI is created, its operators will have the technical capability to maintain it under their control. This assumption underlies the divergence between the dominance doctrine and the extinction doctrine, which predicts that humanity will lose control of ASI, leading to human extinction or its permanent disempowerment.^{[19 OpenAI]}

2.1 Automated AI R&D can yield an insurmountable lead in AI

A common element of this doctrine is the belief that, as progress continues, at some point AI systems themselves will be able to perform most or all of the work required to further advance AI research. This capability is commonly referred to as “Automated AI R&D”, “Recursive self-improvement” or “Intelligence recursion”.^{[20 Wikipedia][21 Dan Hendrycks et al.]} At this point, it is conjectured that progress would accelerate dramatically, as it would no longer be constrained by the capability of human experts to perform research. If a situation arises in which AI research is bottlenecked primarily by resource constraints like the amount of available compute or energy, even these barriers may be addressed by AI, perhaps by accelerating the design and production of faster and more efficient chips. At this point, the pace of AI progress would be determined solely by the abilities of the best available AI systems at AI R&D tasks.

If this stage is reached, the result would be extremely rapid, exponentially compounding progress in AI capabilities. Under this regime, any gap in AI capabilities between the leading actor and others could only grow over time, and the pace at which this gap grows would be ever accelerating. As a result, any actor who first crosses this threshold with even a small lead would be able to leverage it into an unsurmountable advantage.^{[22 Dario Amodei][23 TechCrunch][24 Leopold Aschenbrenner][25 Seán Ó hÉigeartaigh]}

2.2 Powerful AI can be leveraged into a decisive strategic advantage

Under this doctrine, it is anticipated that, once sufficiently advanced AI is developed, it will enable the production of novel weapons and other technologies with transformative offensive and defensive implications. Examples of such capabilities, as described in RAND's paper "AGI's Five Hard National Security Problems"^{[26 Jim Mitre, Joel B. Predd at RAND]} and in Aschenbrenner's Situational Awareness ^{[27 Leopold Aschenbrenner]}, include:^{[28 Dan Hendrycks et al.]}

• Novel weapons of mass destruction.^{[29 IDAIS-Beijing]}
• Weapons of mass destruction defense systems which undermine the principle of mutually assured destruction.^{[30 Leopold Aschenbrenner]}
• Bioweapons.
• Advanced cyberwarfare, potentially capable of completely disabling retaliatory capabilities.
• Autonomous weapon systems, such as tightly coordinated, massive autonomous drone swarms.
• Automation of key industries enabling an explosion in production capacity.
• "Fog-of-war machines" that render battlefield information untrustworthy.

Some of the more near-term possibilities from this list, bioweapons and cyberweapons, are mentioned as redlines in voluntary commitments from AI companies: Anthropic's Responsible Scaling Policy, OpenAI's Approach to Frontier Risk.^{[31 Anthropic][32 OpenAI]}

In this hypothesis, the combination of AI's military potential and the runaway nature of Automated AI R&D have a critical implication: if an "AI race" is allowed to play out, at some point the leading superpower will gain the ability to cheaply and quickly neutralize any adversaries^{[33 Leopold Aschenbrenner][34 Leopold Aschenbrenner][35 Dan Hendrycks et al.]}, with little or no cost to itself, and subsequently maintain an unassailable world order.^{[36 Dan Hendrycks et al.]} This state is termed "Decisive Strategic Advantage", sometimes abbreviated to DSA.

2.3 A decisive strategic advantage can be used to neutralize opponents

A decisive strategic advantage obtained through developing an ASI could be used to forcibly stop all other AI programs. This could be carried out either through threat or actual military engagement. If this was done, it would permanently cement a position of dominance by the first actor to develop sufficiently powerful AI over all others.^{[37 Nick Bostrom]}

Some proponents envision resolutions to the scenarios predicted by this doctrine that don't entail executing or threatening to execute a disabling strike against opponents. However, these hinge on a coalition of US-aligned democracies "winning" or "staying ahead"^{[38 Dario Amodei]}, and then using this position to diplomatically pressure others into a non-proliferation regime^{[39 Leopold Aschenbrenner]}, while potentially promoting democratic reform.^{[40 Dario Amodei]} At the same time, those embracing this vision generally don't make confident predictions that such an alliance will prevail, and in fact consider China as a serious contender.^{[41 Leopold Aschenbrenner][42 Dario Amodei]} This uncertainty means that such approaches are framed as competitive imperatives — actions that must be taken to maximize the chances of success — rather than assured paths to a favorable resolution.

One thing to note about the dominance doctrine is that it's not clear whether a state which “wins” an AI race will be able to maintain its internal stability, especially in the case of democracies. The extreme concentration of power enabled by ASI creates the possibility of “snap coups", as whoever is in control of an ASI system would be able to subvert any existing structure of political or military authority. As an example of how this could happen, a backdoor could be inserted into the AI system by cyberattack or by an engineer. Furthermore, any group controlling an ASI would be powerful enough to be insulated from political checks and balances, thus compromising the foundation of democratic governance.^{[43 Yoshua Bengio][44 Corin Katzke, Gideon Futerman][45 Tom Davidson, Lukas Finnveden, Rose Hadshar][46 Tom Davidson, Lukas Finnveden, Rose Hadshar]}

3. Extinction doctrine

The extinction doctrine holds that ASI is possible, and that there is a strong possibility that it will be developed soon.^{[47 Yoshua Bengio][48 Dan Hendrycks et al.]} Its central claim is that once ASI is developed, humanity will lose control of it ^{[49 Jim Mitre, Joel B. Predd]}, leading to the extinction of the human species^{[50 Dan Hendrycks et al.][51 International AI Safety Report][52 Eliezer Yudkowsky on TIME][53 Stuart Russell][54 Yoshua Bengio][55 Dan Milmo on The Guardian][56 Roman Yampolskiy]}, the end of human civilization^{[57 Connor Leahy et al.]} or, at the very least, the permanent disempowerment of humanity.^{[58 Jan Kulveit, Raymond Douglas et al.]}

This doctrine is reflected in a statement published by the Center for AI Safety in May 2023, stating that “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war”. This statement was signed by numerous experts and industry leaders, including the CEOs of major AI companies (OpenAI, Anthropic and Deepmind) as well as the most cited AI researchers (Yoshua Bengio, Geoffrey Hinton, Ilya Sutskever).^{[59 Center for AI Safety]}

Proponents of the extinction doctrine also tend to believe that advances in Automated AI R&D will soon produce dramatic acceleration in the rate of AI progress.^{[60 Eliezer Yudkowsky][61 Nick Bostrom][62 Max Tegmark]} Some of those who hold this view believe that, once ASI is developed, it will quickly surpass the combined intelligence of humanity, not just its brightest minds.^{[63 Elon Musk][64 Connor Leahy et al.]}

We have termed this doctrine the “extinction doctrine” despite the fact that, strictly speaking, it contains predictions of scenarios in which the human species survives. For example, it contains scenarios in which humanity survives but civilization collapses, or in which humanity is permanently subjugated.^{[65 Daniel Kokotajlo et al.]} Nonetheless, we consider this name to be representative of this doctrine, as the extinction of the entire human species is the most commonly predicted outcome within this school of thought. Furthermore, all outcomes predicted by the sources referenced in this section are of similar magnitude and irrevocability to extinction.

3.1 Loss of control

The most common reason for predicting catastrophic outcomes due to AI development is the expectation that we will develop and deploy superintelligent AI before solving the technical problem of ensuring that it acts in accordance with the goals of its operators. This technical problem is usually called the "AI alignment problem”, and adherents to this doctrine generally expect that we are not on track to solve it before we develop dangerously powerful AI.^{[66 Yoshua Bengio, Geoffrey Hinton et al.][67 Connor Leahy et al.][68 Anthony Aguirre][69 Roman Yampolskiy]}

There is an expectation that if ASI was developed before solving the AI alignment problem it would be impossible for humanity to maintain control of such systems. This expectation is shared by virtually all experts who believe ASI development is feasible in the near term. The central argument is that such a superintelligent AI system would be vastly more capable than humans at strategic planning and execution, making it able to outmaneuver any human effort to keep it under control.^{[70 Eliezer Yudkowsky][71 Holden Karnofsky]}

Notably, this view is shared by adherents to the dominance doctrine and many CEOs and researchers at leading AI companies. These experts do not disagree with adherents to the extinction doctrine on whether ASI would be capable of such feats. Rather, the core distinction lies in their optimism that a robust solution to the problem of keeping ASI under control can be found in time for when it is developed.^{[72 Anthropic][73 OpenAI][74 Dario Amodei][75 Ilya Sutskever, Daniel Gross, Daniel Levy]}

Experts in this camp have hypothesized many means by which superintelligent AI systems might escape human control.^{[76 Holden Karnofsky]} Some of these are:

• Escaping from its boundaries: AI might escape from its environment, for example by replicating itself onto hardware under its direct control, or by blackmailing its engineers to help it escape. This way, it would become much harder to shut down.^{[77 Nick Bostrom]} It has been noted by Anthropic researchers^{[78 Anthropic]} as well as independent researchers at Palisade Research^{[79 Jeremy Schlatter et al.]} that, under test conditions, AI systems already consistently engage in behaviors like hacking or blackmailing their developers to preserve themselves, although their current capabilities are insufficient to succeed in these attempts.
• Reliance: Superintelligent systems might become deeply enmeshed into our infrastructure, making us hesitant or unable to shut them down. Furthermore, organizations may increasingly delegate decisions and authority to AI systems as they become more effective, eroding human oversight. This is considered especially likely in competitive domains, including military contexts, where decision-makers may fear that they would otherwise not be able to keep up with less hesitant competitors.^{[80 Anthony Aguirre]} In other words, humanity might lose control of ASI by "gradually handing it over."^{[81 Jim Mitre, Joel B. Predd][82 Yoshua Bengio][83 Dan Hendrycks et al.][84 Leopold Aschenbrenner]}
• Manipulation: An AI might further ensure its safety by manipulation.^{[85 David Dalrymple on X]} One way it might pursue this is by making itself indispensable to key figures with influence over decisions about shutting down AI systems or about giving AI systems more power and resources.^{[86 Daniel Kokotajlo]} It may also attempt to manipulate us by intentionally enmeshing itself in essential functions like power grids or users' personal lives, reducing our ability to shut it down.^{[87 Dan Hendrycks]} Many experts are concerned that AI systems will possess such extraordinary persuasive abilities that even just allowing these systems to interact with people would pose grave risks of manipulation.^{[88 Sam Altman on X][89 Eliezer Yudkowsky][90 Holden Karnofsky]} For example, such AI systems may gather support from the public by taking on human traits and cultivating emotional attachment in people ^{[91 Dan Hendrycks][92 Jan Kulveit, Raymond Douglas et al.]}, or they may convince their developers to grant them greater autonomy and resources.^{[93 Eliezer Yudkowsky]}

3.2 Out-of-control superintelligence is incompatible with human life

Proponents consider it impossible to predict what will unfold in precise terms once a superintelligence has been developed and escaped human control. However, they tend to forecast that the outcome will not be compatible with human civilization and human life.^{[94 Daniel Kokotajlo][95 David Dalrymple on X]}

This school of thought generally holds that the act of activating a superintelligent AI before solving the problem of alignment is irreversible: once this is done, it will be permanently impossible to reassert control over it, or to change the AI’s goals.^{[96 Nick Bostrom][97 Stephen M. Omohundro]}

Most goals that an ASI might end up pursuing will require the control of abundant material resources, including energy and computing infrastructure.^{[98 Human Compatible, Stuart Russell]} In this case, eventually, a point would be reached when the choice is made to divert resources away from human use, or in general to take actions that are incompatible with human life.^{[99 Connor Leahy et al.][100 Holden Karnofsky]} For instance, an ASI might endeavor to capture all of the resources on earth and cover the surface of the planet with datacenters and energy infrastructure.

If this outcome materializes, this would almost certainly result in the end of human civilization, and likely the end of all human life. ^{[101 Superintelligence, Nick Bostrom][102 Sam Altman][103 Eliezer Yudkowsky][104 Jan Kulveit, Raymond Douglas et al.]}

Wikipedia curates a list of probability estimates that experts assign to the risk of human extinction from losing control of ASI, wryly referred to as p(doom), short for "probability of doom".^{[105 Wikipedia]} These estimates include:

• Paul Christiano, co-creator of RLHF, the technique that enabled the creation of ChatGPT: 50%
• Dan Hendrycks, drafter of AI Safety bill SB 1047: 80% in 2023, up from 20% in 2021
• Eliezer Yudkowsky, pioneer in the systematic study existential risks from AI: +95%
• Geoffrey Hinton, winner of a Nobel Prize in physics, Turing Award recipient who resigned from Google in order to speak freely on AI risks: 50%^{[106 Geoffrey Hinton]}
• Dario Amodei, CEO of Anthropic: 10-25% (including "intentional misuse")^{[107 Dario Amodei]}
• Yann LeCun, Chief AI Scientist at Meta, also a Turing Award recipient: 0%

Many experts and business leaders signed an open letter calling for a 6 month moratorium on AI experiments in 2023, attempting to buy time to set up governance and oversight systems, as well as developing protocols that would ensure "that systems adhering to them are safe beyond a reasonable doubt".^{[108 Future of Life Institute]} Signatories included Yoshua Bengio (Turing Award winner), Stuart Russell (UC Berkeley professor and AI pioneer), and Elon Musk (OpenAI co-founder). This pause was not enacted.

4. Replacement doctrine

The replacement doctrine posits that AI will develop in a predictable and limited way without fundamentally altering existing economic and geopolitical paradigms. Rather than creating entirely novel possibilities such as autonomously producing breakthroughs in technology and pioneering new scientific paradigms, it will primarily replace humans in their current roles and responsibilities, allowing these tasks to be performed at greater speed and scale while reducing costs.

This category is less homogenous than the other two in terms of how much impact proponents expect AI to have, and whether they expect its impact to be positive. However, proponents generally share the belief that AI will not be so overwhelmingly transformative that core principles governing geopolitics and economics become obsolete. Compared to the dominance and extinction doctrines, this doctrine is characterized by the following expectations.

• AI will not cause catastrophic outcomes like human extinction.
• AI will not transform geopolitics to the point where the concept of separate sovereign states stops applying, for example by allowing one actor to seize control over all others.
• AI will preserve the basic structure of the economy, with concepts like labor and capital remaining relevant; humans will keep participating in the economy, if only as consumers.

Proponents of this doctrine hold varying views on whether AI’s overall impact will be beneficial, with both extraordinary benefits and major disruptions being highlighted as possibilities.

4.1 Expectation of slower AI progress

Proponents typically believe that AI progress will be slower compared to adherents of the other doctrines.^{[109 Yann LeCun][110 Andrew Ng]} Many argue that scaling current AI paradigms will not suffice to develop AGI, and that doing so will require major scientific breakthroughs of uncertain nature, which we are unlikely to obtain in the near term.^{[111 Gary Marcus][112 Yann LeCun][113 Francois Chollet]}

However, skepticism about rapid AI progress is not universal among proponents of the doctrine. Sam Altman and Dario Amodei, who predict dramatic AI progress in the next few years^{[114 Sam Altman][115 Dario Amodei]}, can be considered to embody the replacement doctrine in their essays envisioning a future where AI automates most jobs and accelerates scientific research without completely upending the current geopolitical and economic order.^{[116 Sam Altman][117 Dario Amodei]}

4.2 Economic and scientific benefit

Proponents generally believe that AI will greatly boost economic growth, as well as technological and scientific progress, by automating many key tasks currently constrained by human expertise and capacity.^{[118 Yann LeCun][119 Sam Altman]} Examples of the benefits expected from AI include radically accelerated discoveries in biology, neuroscience and medical science, innovations in materials science and engineering, drastic improvements in the quality of education and training, and extraordinary economic growth. On the most bullish side, AI has been compared as an innovation to electricity or microchips.^{[120 Marc Andreessen]} The boldest predictions envision AI enabling us to double the human lifespan and to cure or prevent most diseases.^{[121 Dario Amodei]}

4.3 Unemployment and concentration of wealth

AI's effect on employment is a prominent area of dispute within this school of thought. Views range from expecting smooth transitions, with net job creation, to expecting nearly full unemployment across the global population.

Among the optimists are those who think it's unlikely that AI will replace most workers entirely. While they believe that AI may automate some or most parts of their jobs, they contend that this will not result in widespread unemployment. Rather, they believe it's more likely that workers will learn to perform "AI-augmented" versions of their jobs, thus becoming more productive, while job displacement will occur at tolerable levels and be offset by workers' ability to retrain. These arguments are usually based on analogies with previous waves of automation, such as the industrial revolutions: despite causing some job displacement, these transformations also created new jobs and industries resulting in net positive consequences for the economy and increased productivity.^{[122 Yann LeCun][123 Yann LeCun][124 Marc Andreessen][125 Yann LeCun][126 Marc Andreessen]}

Another perspective holds that, even if AI replaces many or even all existing jobs, this will lead to an explosion in economic growth and the creation of many new types of work as a result. The logic behind this position is that the prices of existing goods and services would drop to near zero, making demand for new products and services explode, causing new jobs to be created.^{[127 Marc Andreessen]} However, it's unclear what the labor market would look like in this situation. Andreessen, who articulates this view, has speculated that venture capital work might be the only surviving occupation when AI performs all other tasks.^{[128 Marc Andreessen]}

Others, on the pessimistic side of this spectrum, predict extreme levels of unemployment.^{[129 Anthony Aguirre]} Sam Altman and Dario Amodei have both voiced the opinion that as AI advances, the demand for human labor will disappear.^{[130 Sam Altman][131 Dario Amodei]} Those holding these positions often warn that unprecedented measures will need to be taken in order to prevent extreme unemployment from resulting in severe levels of inequality. These proposals include universal basic income schemes, as well as shifting the tax burden away from labor and toward capital.^{[132 Sam Altman][133 Dario Amodei]}

There are concerns that such extreme levels of unemployment would not only create challenges around addressing the resulting inequality, but also fundamentally threaten the existence of democratic society. One such argument, made by Luke Drago and Rudolf Laine, is named the "The Intelligence Curse"^{[134 Luke Drago, Rudolf Laine]}, after the resource curse.^{[135 Wikipedia]} The "resource curse" hypothesis argues that states rich in natural resources tend towards worse outcomes in terms of democracy, development and civil liberties. Since rulers of such states can extract wealth directly from natural assets like oil or mineral deposits, they are less reliant on a productive, educated population and therefore less motivated to provide education, infrastructure, or individual freedoms.

The "intelligence curse" hypothesis suggests that a similar dynamic could emerge in advanced economies due to automation. All industries could become similar to those based on extracting natural resources: it will be possible to produce wealth solely by leveraging capital to rent AI workers, without requiring human labor. This way, the peoples of developed economies would lose the leverage they enjoyed from being essential to wealth creation.

4.4 Deceptive media, impersonation and manipulation

Proponents also identify risks other than those originating from unemployment. Some such concerns stem from how AI can imitate human appearance and mannerisms and fabricate realistic media.

• Today, AI can already generate realistic-looking media and can be used to impersonate people.^{[136 Bracewell LLP]} This technology can be used to forge fake evidence in court cases; there are concerns that, as this technology improves, it will become increasingly difficult to rely on video and audio evidence in court.^{[137 Rebecca A. Delfino][138 Robert Chensey, Danielle K. Citron]} Arguments based on this concept have already been invoked in courts and they have been termed "the deepfake defense" by legal professionals.^{[139 Herbert B Dixon Jr at American Bar Association]}
• There are concerns that in the near future, AI could be used to perform feats of large-scale manipulation, such as conducting mass surveillance and propaganda operations.^{[140 Gary Marcus]} AI agents could collect vast amounts of data about users and create psychologically-tailored, microtargeted messaging.^{[141 Anthony Aguirre][142 S. C. Matz et al.][143 Francesco Salvi et al.]} Concerningly, AI systems have already been able to manipulate a small minority of users into performing extreme acts such as committing violence against family members or even committing suicide.^{[144 Kashmir Hill at New York Times]}
• AI could be used to encourage compulsive behavior by consumers. For example, "AI companion" products might prove to be extremely addictive and discourage human relationships. Such products are already commercially successful, and can lead to compulsive behavior due to their constant availability and unconditional enthusiasm towards the user.^{[145 Kevin Roose at New York Times][146 Lesswrong user “blaked”]}

In the worst cases, these problems could threaten democracy by eroding trust in information and undermining both public discourse and electoral systems. Eventually, the majority of information and the most persuasive voices in public debate could be generated or highly tailored by AI systems that have no genuine stake in outcomes.^{[147 Anthony Aguirre]}

4.5 Diffusion of responsibility

Another concern relates to diffusion of responsibility. Policymakers struggle to assign accountability for decisions taken by algorithms. For example, the question is still open whether companies running social media platforms should be held accountable for instances where their algorithmic recommendation systems promote extremist ideas or content calling for violence.^{[148 Betsy Reed at The Guardian]} As more tasks are automated, AI will likely increasingly be used in positions of management and decision making. Given our current paradigms, we might often be unable to hold anyone accountable for harmful decisions taken by such systems.

This problem might manifest at multiple levels: Human developers of a system might expect that no one will be imprisoned for violations of criminal law resulting from the system's decisions. Furthermore, authorities might struggle to impose fines on organizations. This way, essential mechanisms that previously incentivized actors toward caution might be removed. This in turn might erode standards and cause actors to exercise less care regarding possible harms while designing AI systems.^{[149 Filippo Santoni de Sio]}

5. Limitations

While some experts fit relatively cleanly into one of the three categories we identified, such as Leopold Aschenbrenner, Eliezer Yudkowsky, and Yann LeCun, this is not always the case. Nonetheless, we find that this is a useful framework for thinking about experts’ individual positions, predictions or beliefs. For example, many experts who mainly subscribe to the dominance doctrine anticipate significant probabilities of catastrophic outcomes due to AI development. While they expect that the first group to develop ASI will be able to maintain control of it, they make predictions consistent with the extinction doctrine when considering loss-of-control scenarios.^{[150 Dario Amodei][151 Leopold Aschenbrenner]}

Likewise, some proponents of the extinction doctrine accept some of the predictions of the dominance doctrine, specifically around competitive dynamics between superpowers, but argue that this further increases the risk that humanity loses control of powerful AI systems, given that competition will encourage corner cutting on safety research and measures.^{[152 Anthony Aguirre][153 Dan Hendrycks][154 Seán Ó hÉigeartaigh]}

6. Conclusion

The vast divergence between the doctrines in their expectations on the impact of AI development creates a volatile environment. Some will treat AI development as a winner-take-all game in which they cannot allow anyone else to develop superintelligent AI systems ahead of them lest they suffer utter strategic subordination. Others, heedless or even dismissive of superintelligence ambitions, will see AI as a “standard” technological race, in which countries should remain competitive for relatively ordinary strategic and economic reasons^{[155 Seán Ó hÉigeartaigh]}, and which does not warrant international coordination efforts on the same level as other sources of catastrophic risks, such as nuclear weapons proliferation.

This may result in a headlong, unmanaged race toward a technology that may, as believed by many prominent experts, lead to human extinction and that should be treated with the same level of concern as the risk of nuclear war.^{[156 Center for AI Safety]} The uncertainty about the feasibility of ASI and the scale of its associated risks may only dispel much later, possibly too late, only once geopolitical tensions have heated up beyond repair or uncontrollable AI systems have already been created. Even if superintelligence turns out to not be as impending as some suggest, geopolitical tensions may still escalate due to such uncertainties.^{[157 Seán Ó hÉigeartaigh]} Additionally, the more mundane risks hypothesized by some of the replacement doctrine may materialize.

Some private actors in the US have begun to encourage government involvement in the pursuit of AGI and ASI, framing it as a matter of national security and economic competitiveness.^{[158 Leopold Aschenbrenner]} OpenAI's letter to the US government on the AI action plan highlights the need for the US to stay ahead of China in AI.^{[159 Christopher Lehane at OpenAI]} Ó hÉigeartaigh contends that, while there is currently no real race between the US and China, narratives about such a race have been used by private US entities in order to attract resources and justify reduced regulation on AI development.^{[160 Seán Ó hÉigeartaigh]}

Nonetheless, we are now starting to see signs of interest from the governments of superpowers.^{[161 Jake Sullivan]} J.D. Vance, during his intervention at the Paris AI Summit, asserted that the U.S. administration is committed to maintaining AI dominance.^{[162 J.D. Vance]} On the Chinese side, the government has announced a 1 trillion yuan investment into AI and robotics.^{[163 Bank of China]} Separately, reports indicate that Chinese AI leaders have been warned to avoid travel to the U.S., due to concerns they could be pressured to divulge confidential information or be detained and used as bargaining chips in geopolitical disputes.^{[164 Reuters]}

We highlight the urgency of establishing international coordination mechanisms to curtail risks related to AI development. We must not delay until there is consensus that superintelligence is imminent and on whether the stakes are as high as human extinction or total dominance by a single actor over all others. Deteriorating international relations as well as the potential self-accelerating nature of AI development mean that, by the time these uncertainties resolve, we may not have the time or international goodwill necessary to establish governance mechanisms sufficient to prevent such risks. Moreover, even the more conventional risks predicted by some, such as widespread unemployment and the possibility of AI-enabled mass surveillance and manipulation, warrant immediate international attention.

Citations