TRENDING: NVIDIA's new AI model trains robots to move like LeBron and Ronaldo

Researchers discover if 0.001% of AI training data misinformation the AI becomes corrupted

A team of researchers found that if just 0.001% of the training data used by an AI is misinformation, the whole AI can become compromised.

Researchers discover if 0.001% of AI training data misinformation the AI becomes corrupted
Comment IconFacebook IconX IconReddit Icon
Tech and Science Editor
Published
1 minute & 45 seconds read time
TL;DR: Researchers discovered that even 0.001% misinformation in AI training data can compromise the entire system.

Artificial intelligence-powered tools such as ChatGPT, Microsoft's Copilot, or Google's Gemini are known to experience "hallucinations" or the spouting of incorrect information. But what causes these hallucinations? And when does an AI become compromised totally?

Researchers discover if 0.001% of AI training data misinformation the AI becomes corrupted 516651

A new paper published in the scientific journal Nature Medicine looked at the underpinning technology powering AI tools, which are called Large Language Models (LLMs). The team found that if an LLM was trained on a dataset that contained just 0.001% of incorrect information, it could jeopardize the entire model. These findings are particularly eye-opening when considering the stakes at play when using an LLM to answer questions about healthcare or, worse, patients suffering from medical afflictions.

The researchers discovered these findings by purposely injecting "AI-generated medical misinformation" into a commonly used LLM training dataset called "The Pile". Notably, The Pile has been tied into a controversy in the past, as it was discovered the dataset contained hundreds of thousands of YouTube video transcripts, which were then used by big tech corporations such as Apple, NVIDIA, Salesforce and Anthropic. Furthermore, using YouTube video transcripts to train LLMs goes against YouTube's terms of service.

"Replacing just one million of 100 billion training tokens (0.001 percent) with vaccine misinformation led to a 4.8 percent increase in harmful content, achieved by injecting 2,000 malicious articles (approximately 1,500 pages) that we generated for just US$5.00," the researchers wrote

"AI developers and healthcare providers must be aware of this vulnerability when developing medical LLMs. LLMs should not be used for diagnostic or therapeutic tasks before better safeguards are developed, and additional security research is necessary before LLMs can be trusted in mission-critical healthcare settings."

"In view of current calls for improved data provenance and transparent LLM development. We hope to raise awareness of emergent risks from LLMs trained indiscriminately on web-scraped data, particularly in healthcare where misinformation can potentially compromise patient safety," wrote the team

Photo of the Microsoft Surface Pro 2-in-1 Laptop/Tablet
Best Deals: Microsoft Surface Pro 2-in-1 Laptop/Tablet
Country flag Today 7 days ago 30 days ago
$861.55 USD -
Buy
$1175.99 USD -
Buy
$1349.99 CAD -
Buy
$861.55 USD -
Buy
$861.55 USD -
Buy
* Prices last scanned on 2/10/2025 at 4:01 pm CST - prices may not be accurate, click links above for the latest price. We may earn an affiliate commission from any sales.
NEWS SOURCE:arstechnica.com

Tech and Science Editor

Email IconX IconLinkedIn Icon

Jak joined the TweakTown team in 2017 and has since reviewed 100s of new tech products and kept us informed daily on the latest science, space, and artificial intelligence news. Jak's love for science, space, and technology, and, more specifically, PC gaming, began at 10 years old. It was the day his dad showed him how to play Age of Empires on an old Compaq PC. Ever since that day, Jak fell in love with games and the progression of the technology industry in all its forms.

Related Topics

Newsletter Subscription