NVIDIA Audio2Face uses AI to generate lip synching and facial animation, showcased in two games

Localization is a big thing in gaming, where a game's dialogue and presentation are recorded in various languages to reach a global audience. Like how Netflix dubs all of its original programming into multiple languages, localization for games is a complex process that requires accurately translating dialogue and text, maintaining dramatic or comedic tones, and then re-recording dialogue with voice actors from different countries.

2

NVIDIA's Audio2Face technology for lip-synching and facial animation in the Unreal Engine 5-powered Unawake.

Where games differ from movies or television drama is that all performances are digital and completely malleable, which is why you've got a situation where Sony's PlayStation-exclusive Ghost of Tsushima (which is coming soon to PC) offers both English and native Japanese language options with full lip-syncing and correct facial animation.

Although it's possible to do this for dozens of languages, Ghost of Tsushima limits full lip-synching to two language options due to the task's complexity and the animation involved. This is where generative AI, specifically NVIDIA's Audio2Face technology, will step in.

Another example is a sprawling RPG or online title where lip-synching is basic and straightforward, allowing for the implementation of new dialogue or alternative dialogue. NVIDIA Audio2Face is a powerful generative AI tool that can create accurate and realistic lip-synching and facial animation based on audio input and character traits. Developers are already using it.

Facial animation involves communicating a character's emotion, which makes the tool invaluable because it can pair words with emotion to present a fully animated dramatic facial performance.

The first game featuring the tech is World of Jade Dynasty, a martial arts MMO from the Chinese studio Perfect World Games. With NVIDIA Audio2Face, the studio can generate accurate lip-synching and animation in Chinese and English. As an online title, NVIDIA Audio2Face will allow developers to add new voiced content with the same realistic animation.

The second game showcasing the tech is RealityArts Studio and Toplitz Productions' Unreal Engine 5-powered Unawake, a new action-adventure game launching later this year with DLSS 3 support. This video presents a dynamic look at NVIDIA Audio2Face in action, showcasing the various sliders covering various emotions.

• Read more: NVIDIA's Covert Protocol tech demo has you become a detective investigating AI Digital Humans
• Read more: NVIDIA's next-gen Blackwell AI GPU: multi-chip GPU die, 208 billion transistors, 192GB HBM3E
• Read more: NVIDIA GB200 Grace Blackwell Superchip: 864GB HBM3E memory, 16TB/sec memory bandwidth
• Read more: NVIDIA creates Earth-2 digital twin: generative AI to simulate, visualize weather and climate
• Read more: NVIDIA's new Project GROOT: fully humanoid robot to compete against Tesla Optimus AI robot