AI-powered NVIDIA ACE Digital Human added to this multiplayer PC game

At Computex 2024, we went hands-on with one of NVIDIA's impressive digital humans as part of the Covert Protocol tech and game demo (developed in collaboration with Inworld AI and powered by NVIDIA ACE technologies). How it works is relatively straightforward: In-game characters are all AI-generated with personalities, knowledge, and relationships, so using text or voice, the goal is to find specific information in a detective-style role.

Check out my full report on how it works and how I extracted critical information from one of the characters by skipping a few puzzle-like steps. It's a fascinating look at the future of game development, with the demonstration running locally on a high-powered GeForce RTX 40 Series GPU.

At Gamescom 2024, NVIDIA is presenting an actual in-game version of the technology running in Mecha BREAK, a multiplayer mech game from Amazing Seasun Games currently in development. The game is powered by NVIDIA's new Nemotron-4 4B NIM small language model for game characters - a breakthrough NVIDIA says will help bring Digital Humans to games sooner rather than later.

3

"Mecha BREAK's Digital Human also leverages Audio2Face-3D NIM for dynamic facial animation running on-device," NVIDIA adds. "Third-party models from Whisper provide speech recognition running on device, and Elevenlabs provides realistic voices running in the cloud​."

So yeah, it's a mix of speech recognition, tapping into the generative AI, and then using AI to create the facial animation and voice you hear.

It's cutting-edge stuff, and in Mecha BREAK, players can interact with mechanic Martel to talk about upcoming missions, mech load-outs, and advice. Martel can act on voice input, make suggestions, and change mech configurations for players - making it an excellent tool for new players to use the game's in-depth customization tools.

The Digital Human Mecha BREAK demo will be on the Gamescom show floor. NVIDIA notes that AI requires powerful hardware to run locally, so a powerful GPU is required to run this in real-time. However, there will be cloud components to offload processing. With Digital Humans running in a game like Mecha BREAK, the impact on frame rate is around 5%, so it's not a significant hit to performance, like changing texture quality from low to high. That said, the demo is running on a GeForce RTX 4090-powered rig, so like Path Tracing, this technology sits on the high end of PC gaming.

For more on Mecha BREAK check out thegame's Steam page.