Anthropic CEO and co-founder Dario Amodei has posted a new in-depth 'We Must Pace the Frontier' article that argues that AI companies need to slow the pace of AI model development to address the increasingly serious risks posed by current advances. At a high level, the main risks are losing control of AI systems, misusing AI for cyberattacks and terrorism, and AI's potential to cause "serious economic disruption."
With the accelerated pace of AI, which Dario calls a "race to the bottom, spurred by commercial incentives," the call to action is less about pumping the brakes on AI model capabilities and more about using that time wisely to mitigate the risks. The article makes it clear he's not calling for a halt to model training or optimization, but for building AI at a "balanced rate that aims to ensure its safety while still achieving its benefits and grappling with important geopolitical dilemmas."
Since posting the article, both Elon Musk (xAI) and Sam Altman (OpenAI) have come out in support of the plan. Over on X, Elon Musk simply wrote "Dario is right," with Sam Altman expanding on his agreement by adding, "This has been a primary topic of discussions we've had at OpenAI in recent weeks." As for the three-step plan he lays out, Dario calls it a useful framework for achieving what "needs to be accomplished."
Here are the top-level details, covering safety and coordination between companies, countries, and governments.
- Embedded Evaluators. Each frontier AI company commits to giving ongoing, employee-like access to a team of embedded third-party evaluators (such as METR), whose role is to verify adherence to safety practices and commitments, report incidents, and help assess the alignment of not just completed AI models but training pipelines and processes. This is the key step for verifiability of any pacing commitments, and has precedent in the banking industry, which sometimes involves regulatory "supervisors" embedded along with employees. Anthropic is unilaterally committing to this step now. We intend this to be part of a broader push to redouble efforts on our safety and alignment work.
- Democratic Coordination. Frontier AI companies within democratic countries coordinate to establish common safety standards as well as limits on the rate of unchecked AI progress. Some forms of coordination that would be impactful for pacing are legally challenging, and will require government support.
- Global Coordination. The US and other democratic governments attempt to coordinate with authoritarian governments, to the extent this is possible, while taking seriously the challenges of verifying compliance.
The full article goes into greater detail on all three points, and when it comes to collaborating with democratic governments, Dario does respond to the question of China. For those who have been following China's AI development, you'll no doubt be aware of the US government's hardware restrictions put in place to curb that progress. Dario agrees with this point and this measure, writing bluntly: "Do not sell powerful AI chips or semiconductor manufacturing equipment to China, and crack down on chip smuggling operations and remote access to data centers outside China."
Adding that "chips will be the main determinant of China's AI strength," it's a key part of the plan to ensure that over the next three to five years America's lead expands. However, Dario believes that any plan to control the pace of AI development will still require cooperation with China.
"We must not be naïve here," Dario writes. "The geopolitical stakes are so high that there will likely be stark limits on what can be achieved, especially at first. If we greatly restrain our AI capabilities in the belief that China will do the same, and then China defects, AI could be so powerful that such a defection could lead to their geopolitical dominance. Therefore any agreement must either have ironclad verifiability, or must be limited enough that defection would not be militarily existential."
Frequently Asked Questions
Open a question for an answer from TweakTown's coverage of this news, or ask your own below.
How would Anthropic’s embedded third-party evaluators work in practice?
What kinds of AI safety incidents would these evaluators be expected to report?
Would Dario Amodei’s proposal apply only to frontier AI companies or to the whole AI industry?
What government involvement does the article suggest would be needed for democratic coordination on AI pace limits?
Have a question that isn't listed here? Ask below, and TweakBot will answer it.
With the current focus on building new data centers and expanding chip manufacturing to accommodate the growing AI industry as the next big global infrastructure rollout, the call for "third-party evaluators," regulation, and cooperation between companies and governments certainly feels timely.
