Cohere CEO says LLMs training each other is like a 'human centipede effect', Elon Musk agrees

Elon Musk's AI startup is training its next-generation Grok 3 LLM model, which is being trained on a monster 100,000 x NVIDIA H100 AI GPUs.

In a new post on X, the SpaceX and Tesla founder talked about his xAI start-up, replying to @BasedBeffJezos on X who was replying to @BasedBeffJezos who posted a video from Aidan Gomez, the CEO and co-founder of Cohere, who said "models training on each other's data is like a human centipede effect".

This is where Musk replied, saying: "Sadly quite true. It takes a lot of work to purge LLMs from the internet training data. Grok 2, which comes out in August, will be a giant improvement in this regard". This is where @BasedBeffJezos said "Grok 2 in August will be Based AGI". Musk replied again, saying: "Grok 3 end of year after training on 100K H100s should be really something special".

You can see that video below:

Earlier this year, Tesla said it would be sending billions of dollars buying NVIDIA AI GPUs and AMD AI GPUs, so these numbers will radically change throughout the year as Tesla scoops up more AI silicon from NVIDIA. The recent $500 million investment into the Dojo Supercomputer is "only equivalent" to 10,000 x NVIDIA H100 AI GPUs, said Musk in January 2024, adding, "Tesla will spend more than that on NVIDIA hardware this year. The table stakes for being competitive in AI are at least several billion dollars per year at this point".