Ubisoft's Project NEO NPC game demo uses NVIDIA's Audio2Face and Inworld generative AI

Yesterday, we examined AI's use to create digital human characters (NPCs) as part of a new Covert Protocol tech demo from Inworld and NVIDIA. Using the Inworld AI engine with NVIDIA ACE, which includes NVIDIA Riva Automatic Speech Recognition (ASR) and NVIDIA Audio2Face (A2F) for speech and animation, the demo gave a glimpse at a potential AI-powered future for games.

3

At GDC 2024, Ubisoft and its developers at Ubisoft Paris presented their demo, Project NEO NPC, in partnership with NVIDIA and Inworld. Using Inworld's extensive Character Engine with LLM technology, the team created NPCs with complete backgrounds, knowledge, and distinct conversational styles Ubisoft leveraging NVIDIA Audio2Face.

"Characters are programmed to react to player prompts in certain ways, or not react, depending on their personalities," Ubisoft writes in its blog post. Narrative Director Virginie Mosser at Ubisoft Paris adds, "For the first time in my life, I can have a conversation with a character I've created. I've dreamed of that since I was a kid."

Based on the screenshots, Project NEO NPC's story involves a heist with players' given objectives and goals.

Ubisoft adds that Project NEO NPC is a prototype, and its veteran designers had to iterate to ensure that responses were not "robotic" or "boring." Getting its AI-driven character to react like real people or fictional characters with specific traits and motivations was a collaborative process between creators, engineers, and AI specialists.

3

"It is really important to us that it behaves like the character Virginie created," said Data Scientist Mélanie Lopez Malet. "So, while we're talking to it, we ask ourselves: 'Is this Lisa? Would Lisa say this?' and if the answer is no, we need to go back and find out what happened within the model to make it stray from the vision Virginie had."

"It's garbage in, garbage out," Malet adds when players input 'nonsensical prompts.' "But it's also magic in, magic out. And when the player creates their own scene and it all clicks? That's an incredible experience for them."

The model also has measures to account for toxicity or players insulting characters, so if you actively try to break the fourth wall, the system won't work as intended - which is to be a collaborative experience for players. Ubisoft believes generative AI and storytelling "can live side by side," but there's "still a way to go before it can be implemented in a game."