NVIDIA Audio2Face uses AI to generate lip synching and facial animation, showcased in two games

Generative AI is slowly but surely making its way into gaming, and NVIDIA's AI-powered Audio2Face is game changer for facial animation and localization.

Published
Updated
2 minutes & 41 seconds read time

Localization is a big thing in gaming, where a game's dialogue and presentation are recorded in various languages to reach a global audience. Like how Netflix dubs all of its original programming into multiple languages, localization for games is a complex process that requires accurately translating dialogue and text, maintaining dramatic or comedic tones, and then re-recording dialogue with voice actors from different countries.

NVIDIA's Audio2Face technology for lip-synching and facial animation in the Unreal Engine 5-powered Unawake.

NVIDIA's Audio2Face technology for lip-synching and facial animation in the Unreal Engine 5-powered Unawake.

Where games differ from movies or television drama is that all performances are digital and completely malleable, which is why you've got a situation where Sony's PlayStation-exclusive Ghost of Tsushima (which is coming soon to PC) offers both English and native Japanese language options with full lip-syncing and correct facial animation.

Although it's possible to do this for dozens of languages, Ghost of Tsushima limits full lip-synching to two language options due to the task's complexity and the animation involved. This is where generative AI, specifically NVIDIA's Audio2Face technology, will step in.

Another example is a sprawling RPG or online title where lip-synching is basic and straightforward, allowing for the implementation of new dialogue or alternative dialogue. NVIDIA Audio2Face is a powerful generative AI tool that can create accurate and realistic lip-synching and facial animation based on audio input and character traits. Developers are already using it.

Facial animation involves communicating a character's emotion, which makes the tool invaluable because it can pair words with emotion to present a fully animated dramatic facial performance.

The first game featuring the tech is World of Jade Dynasty, a martial arts MMO from the Chinese studio Perfect World Games. With NVIDIA Audio2Face, the studio can generate accurate lip-synching and animation in Chinese and English. As an online title, NVIDIA Audio2Face will allow developers to add new voiced content with the same realistic animation.

The second game showcasing the tech is RealityArts Studio and Toplitz Productions' Unreal Engine 5-powered Unawake, a new action-adventure game launching later this year with DLSS 3 support. This video presents a dynamic look at NVIDIA Audio2Face in action, showcasing the various sliders covering various emotions.

Buy at Amazon

ASUS TUF Gaming GeForce RTX 4090 Graphics Card TUF-RTX4090-O24G-OG-GAMING

TodayYesterday7 days ago30 days ago
Buy at Newegg
$1799.99$1799.99$2129.99
$1799.99$1849.99$2079.99
* Prices last scanned on 4/24/2024 at 4:00 pm CDT - prices may not be accurate, click links above for the latest price. We may earn an affiliate commission.
NEWS SOURCE:nvidia.com

Kosta is a veteran gaming journalist that cut his teeth on well-respected Aussie publications like PC PowerPlay and HYPER back when articles were printed on paper. A lifelong gamer since the 8-bit Nintendo era, it was the CD-ROM-powered 90s that cemented his love for all things games and technology. From point-and-click adventure games to RTS games with full-motion video cut-scenes and FPS titles referred to as Doom clones. Genres he still loves to this day. Kosta is also a musician, releasing dreamy electronic jams under the name Kbit.

Newsletter Subscription

Related Tags