Insomniac may be using Sony AI machine learning for Spider-Man on PS5

Insomniac Games is using machine learning to enhance in-game visuals on the PlayStation 5, and the studio could be working alongside Sony's AI division on experimental new technology.

3

Insomniac Games recently turned heads by confirming the PlayStation 5 can support machine learning. The team is using it in creative ways, starting with a new PS5 update for Spider-Man Miles Morales that significantly changes muscular appearance in the game. The new technique creates more realistic "muscle deformation," a term used in animation to describe how models are transformed and manipulated in 3D rigs.

According to Lead Character Technical Director Josh DiCarlo, the technique is using machine learning inference, which means Insomniac is feeding the algorithm data in real-time that's running on the PlayStation 5. DiCarlo says this technique doesn't come at a graphical hit.

"Not sure how specific I can get with specs for now, but you are correct in the assumption that all final deformations are resulting via ML inference at runtime on the PS5 hardware. There are no blendshapes, skin decomp, or traditional tricks of the trade..."

"...High-level, we have a highly detailed & anatomically accurate skeleton, muscle, and fat/skin system fitted inside Miles, along with costumes that have varying material properties (cloth, rubber, etc.), which are all simulated together to produce final per-frame deformations."

3

It's also worth noting that ML inference is the second phase of this process.

The first is AI training. It's possible that Insomniac has been working closely with Sony's internal AI division to "train" the game to perform the new deformation techniques.

"We have brilliant engineers working with brilliant engineers. We're running as fast as the non-physically based version and did not require any adjustments to the graphical settings,"DiCarlo said on Twitter.

3

Sony AI has been experimenting with machine learning for a while, and the division's website says gaming is a core focus of this new technology.

"All Sony AI research projects will be pursued in close collaboration with the relevant Sony Group business units, including those in gaming, imaging, and sensing," the site reads. "Sony AI currently is pursuing three flagship projects in Gaming, Imaging and Sensing, and Gastronomy."

DiCarlo says that Insomniac Games plans to utilize the tech in new ways--and this should be true for all of Sony's first-party studios including Naughty Dog and Sony Santa Monica.

"...We're just getting warmed up," DiCarlo said.