The intention of this topic is to add some knowledge for the OpenAI model users. If you think it has some scientific or technical value for your research or work, please use it with discretion.
DESCRIPTION SECTION:
Preventing an AI-related catastrophe
AI might bring huge benefits — if we avoid the risks
By Benjamin Hilton · Published August 2022 · Last updated March 28th, 2023
The complete article and its audio version can be found here:
Preventing an AI-related catastrophe - Problem profile
Why do we think that reducing risks from AI is one of the most pressing issues of our time? There are technical safety issues that we believe could, in the worst case, lead to an existential threat to humanity.
Est. reading time: 93 minutes
Maybe it is necessary to subscribe to 80000 hours.
INTRODUCTION SECTION:
by Luisa Rodriguez (80,000 Hours: 80000hours.org)
Hi everyone,
The idea this week: how quickly could AI transform our entire world?
Artificial intelligence is moving fast. When we published our problem profile on preventing an AI-related catastrophe last August, we argued the recent AI progress was already rapid. Less than a year later, there are now even more powerful models, such as GPT-4, and a growing marketplace of useful AI products.
There’s no reason to think progress will stop tomorrow. So how fast could it really get? To try to answer this question, I spoke to Tom Davidson, a senior research analyst at Open Philanthropy, 1 for the latest episode of The 80,000 Hours Podcast.
I wanted to know why he thinks the world could change really fast as AI gets increasingly powerful and why the future may be much more alien than people expect.
And I wanted to better understand how friendly chatbots and tools for creating beautiful images could be the precursors of technology that many think is an existential risk.
What he had to say blew me away.
“Today there are maybe tens of millions of people whose job it is to discover new and better technologies, working in science and research and development,” he explained. “But like we’ve been talking about, shortly after [we have artificial general intelligence], there’s going to be billions of top human researcher equivalents … working on trying to advance technology and come up with new ideas. Then you have now 10 or 100 times the effort that’s going into that activity. And these AIs are also able to think maybe 10 or 100 times as quickly as humans can think.”
He continued: “It might be possible to focus them much more effectively on the most important types of R&D, whereas humans maybe are more inclined to follow their interests, even when it’s not the most useful thing to be researching. All of those things together just mean that we’ll be generating 100 times as many new good ideas and innovations each year compared with today, and then that would drive the development of technologies to be at least 10 times faster than today.”
That means that we might see something like 50 years of technological change in the span of five years — or even less. We’d be entering a dramatically different world, possibly with few guardrails.
To be honest, this all sounded crazy to me. And Tom’s not sure when it will happen — if it happens at all.
This kind of research is especially difficult and uncertain, because our models are most likely to be wrong when we’re forecasting events with no real precursor. So we should treat Tom’s conclusions with scepticism.
But I think the possibility that AI-driven changes happen much faster than some expect is very real, and having technology this powerful could pose a lot of serious risks. As it stands, we think it’s likely we’ll be able to build powerful AI systems at some point in the coming decades, but we’re not yet prepared for it — politically, economically, socially, or emotionally.
If something has the capacity to supercharge the world economy like never before, we should be sure we know what we’re doing when we create it.
That’s another reason this issue really matters, and why we encourage people to consider thinking about how they could use their careers to help tackle these problems.
If you want to understand these issues better, I encourage you to listen to the full episode of the podcast with Tom. I learned a lot, and I hope you will too.
You can also listen on Spotify or Apple Podcasts.
ARTICLE SECTION: Summary only
Preventing an AI-related catastrophe - Summary
By Benjamin Hilton
I expect that there will be substantial progress in AI in the next few decades, potentially even to the point where machines come to outperform humans in many, if not all, tasks. This could have enormous benefits, helping to solve currently intractable global problems, but could also pose severe risks. These risks could arise accidentally (for example, if we don’t find technical solutions to concerns about the safety of AI systems), or deliberately (for example, if AI systems worsen geopolitical conflict). I think more work needs to be done to reduce these risks.
Some of these risks from advanced AI could be existential — meaning they could cause human extinction, or an equally permanent and severe disempowerment of humanity. There have not yet been any satisfying answers to concerns — discussed below — about how this rapidly approaching, transformative technology can be safely developed and integrated into our society. Finding answers to these concerns is very neglected, and may well be tractable. I estimate that there are around 400 people worldwide working directly on this. As a result, the possibility of AI-related catastrophe may be the world’s most pressing problem — and the best thing to work on for those who are well-placed to contribute.
Promising options for working on this problem include technical research on how to create safe AI systems, strategy research into the particular risks AI might pose, and policy research into ways in which companies and governments could mitigate these risks. If worthwhile policies are developed, we’ll need people to put them in place and implement them. There are also many opportunities to have a big impact in a variety of complementary roles, such as operations management, journalism, earning to give, and more — some of which we list below.
Our overall view
Recommended - highest priority
We think this is among the most pressing problems in the world.
Scale
AI will have a variety of impacts and has the potential to do a huge amount of good. But we’re particularly concerned about the possibility of extremely bad outcomes, especially an existential catastrophe. Some experts on AI risk think that the odds of this are as low as 0.5%, some think that it’s higher than 50%. We’re open to either being right — and you can see further discussion of this here. My overall guess is that the risk of an existential catastrophe caused by artificial intelligence within the next 100 years is around 1%. This puts me on the less worried end of 80,000 Hours staff, whose views on our last staff survey ranged from 1–55%, with a median of 15%.
Neglectedness
Around $50 million was spent on reducing catastrophic risks from AI in 2020 — while billions were spent advancing AI capabilities. While we are seeing increasing concern from AI experts, I estimate there are still only around 400 people working directly on reducing the chances of an AI-related existential catastrophe (with a 90% confidence interval ranging between 200 and 1,000). Of these, it seems like about three quarters are working on technical AI safety research, with the rest split between strategy (and other governance) research and advocacy.
Solvability
Making progress on preventing an AI-related catastrophe seems hard, but there are a lot of avenues for more research and the field is very young. So I think it’s moderately tractable, though I’m highly uncertain — again, assessments of the tractability of making AI safe vary enormously.
Profile depth
In-depth