AI existential risk probabilities are (still) too unreliable to inform policy

Key points

• Over two years ago we wrote a detailed deconstruction of p(doom) and argued that its primary effect is to launder vague intuitions and fears through a facade of quantification.1
• As the essay makes clear, the infeasibility of building useful quantitative models is specific to forecasts of AI x-risk, not forecasting in general.1
• That said, we think the whole p(doom) culture is emblematic of a particular view of AI safety that centers existential risk and superintelligence in a way that is actively counterproductive to a broader conception of safety.1
• AI existential risk probabilities are too unreliable to inform policy1

Sources (1)

• [1]AI existential risk probabilities are (still) too unreliable to inform policy
  AI Snake Oil · Sep 28, 11:13 AM

  Unsurprisingly, today’s probability estimates of AI existential risk are no more rigorous than the ones from 2024.

  Over two years ago we wrote a detailed deconstruction of p(doom) and argued that its primary effect is to launder vague intuitions and fears through a facade of quantification.

Extractive summary: sentences quoted from the sources.