Switch 4K: Nintendo R&D has been spending on deep learning since 2019

The new Switch OLED disappointed many, but Nintendo R&D is experimenting with deep learning. Does that mean a DLSS-powered Switch 4K will happen after all?

8

Note: This is a sizable article filled with both facts and speculation. The facts are regarding Nintendo's R&D spending habits, with breakdowns of DLSS and NVIDIA's mobile Tegra chips. The speculation is which chip the theoretical upgraded Switch could use and why.

While scouting Nintendo's recent 2021 Annual Report, I noticed something interesting: The company specifically mentions deep learning in its R&D activities. References to deep learning are also found in the 2020 report, confirming Nintendo has been investigating and experimenting with deep learning since as early as April 2019. Deep learning is not mentioned in any other annual report from Nintendo, and the company spend more on R&D in FY2021 than it has in any other period in its history.

"With respect to hardware, we continuously investigate and undertake research on fundamental technologies spanning data storage technology such as semiconductor memory, display technology such as liquid crystal displays, and electronic components, while we also carry out research and development activities to examine the applicability of various technologies including interfaces such as [...] deep learning and big data analysis to the field of home entertainment," the 2021 report reads.

8

Deep learning is the process that makes NVIDIA's Deep Learning Super-Sampling (DLSS) tech possible, a kind of digital wizardry that can upscale a 1080p image to 4K resolution (as well as improve in-game frame rates).

The best part about DLSS is that it reduces rendering costs. Devs can now use lower resolution textures in their games and rely on DLSS to up-sample pixel quality. Developers no longer have to rely solely on processing power and can shave off a lot of VRAM, GPU, and CPU performance requirements for their games.

DLSS reduces the pixel workload required by hardware, including graphics buffering, lighting, and other intensive effects. "The more pixel-bound you are, the more performance benefit you'll get."

This sounds like a perfect solution for a weaker, mobile SoC-powered handheld like the Switch. So far, NVIDIA has only used DLSS in its dedicated graphics cards that feature

A special type of AI network, called a convolutional autoencoder, takes the low resolution current frame, and the high resolution previous frame, to determine on a pixel-by-pixel basis how to generate a higher quality current frame.

During the training process, the output image is compared to an offline rendered, ultra-high quality 16K reference image, and the difference is communicated back into the network so that it can continue to learn and improve its results. This process is repeated tens of thousands of times on the supercomputer until the network reliably outputs high quality, high resolution images.

DLSS 1.0 required each game to be trained individually.

It was heavily rumored the upgraded Switch Pro (aka Switch 4K) would use DLSS to allow 4K gaming at acceptable frame rates. This would require Tensor Cores, a special kind of dedicated AI accelerator built into newer NVIDIA graphics cards like the RTX 20 and 30 series. The Tegra X1 wouldn't work for deep learning; the newer Switch would need an upgraded SoC, the very kind of chip that was mentioned in Bloomberg's original report.

DLSS is enabled by Tensor Cores which are found in three architectures:

• First generation Tensor Cores - (Volta)
• Second generation Tensor Cores (Turing) + First generation dedicated Raytracing (RT) Cores - GeForce GTX 16 series, RTX 20 series
• Third generation Tensor Cores (Ampere) + Second generation dedicated Raytracing (RT) Cores - GeForce RTX 30 series

There are two different iterations of DLSS, both of which require Tensor Cores:

• DLSS 1.0 - AI needs to be trained specifically for each game, very time consuming
• DLSS 2.0 - Trains using non-game-specific content, delivering a generalized network that works across games

NVIDIA is also working on new Ultra Quality modes in DLSS 2.2.9.

Investigations have been quite complicated and no one knows for sure which exact chip the Switch Pro will use.

There's a possibility Nintendo and NVIDIA will adjust the Xavier SoC, which is outfitted with a Volta GPU with first-generation Tensor cores, in a smaller form factor to meet the Switch's needs (Xavier's die size is a substantial 350mm squared compared to the Tegra X1's smaller 118mm squared--the X1 is a lot better for a handheld).

8

Volta wasn't used in commercial hardware outside of the $3,000 Titan V, and was mostly used in HPC applications like the Tesla V100 datacenter accelerator as well as the automotive Xavier SoC.

It's this automotive chip that may be used in the Switch Pro, the same way the Tegra X1 was used in the base 2017 Switch (and the 2017 revision with the 16nm Tegra X1+).

However there's also a possibility Nintendo would want to embrace a more future-proof chip and opt for a Turing or an Ampere-based SoC...or something beyond.

Right now NVIDIA doesn't make a Turing-based Tegra, but it's possible they could create one specifically for Nintendo. The Switch has proven to be a sales monster with 84.59 million units sold worldwide and it's safe to say that the platform is a good bet for NVIDIA.

8

NVIDIA's new Ampere-based Orin line has a 15W camera chip that could be a good fit for the Switch Pro.

NVIDIA has an upcoming Ampere-based SoC called Orin that's in development, and it could theoretically tap third-generation Tensor Cores for boosted DLSS performance.

However, reliable Twitter tecchie kopite7kimi indicates the Switch Pro could be powered by NVIDIA's new Ada Lovelace GPU built on Ampere Next.

Theoretically, the Switch Pro could use a special customized Orin SoC, possibly built on the low-powered 15W model for cameras for significantly lowered TDP.

Kopite also says the new 4K Switch could use a custom T239 Orin Tegra SoC with the Black Knight codename

No one knows what Nintendo will do next, but the deep learning R&D is pretty interesting.

8