Switch 4K: Nintendo R&D has been spending on deep learning since 2019

Nintendo has been spending R&D on deep learning since 2019, possibly hinting a new DLSS-powered Switch Pro is coming after all.

5 minutes & 47 seconds read time

The new Switch OLED disappointed many, but Nintendo R&D is experimenting with deep learning. Does that mean a DLSS-powered Switch 4K will happen after all?

Switch 4K: Nintendo R&D has been spending on deep learning since 2019 1

Note: This is a sizable article filled with both facts and speculation. The facts are regarding Nintendo's R&D spending habits, with breakdowns of DLSS and NVIDIA's mobile Tegra chips. The speculation is which chip the theoretical upgraded Switch could use and why.

While scouting Nintendo's recent 2021 Annual Report, I noticed something interesting: The company specifically mentions deep learning in its R&D activities. References to deep learning are also found in the 2020 report, confirming Nintendo has been investigating and experimenting with deep learning since as early as April 2019. Deep learning is not mentioned in any other annual report from Nintendo, and the company spend more on R&D in FY2021 than it has in any other period in its history.

"With respect to hardware, we continuously investigate and undertake research on fundamental technologies spanning data storage technology such as semiconductor memory, display technology such as liquid crystal displays, and electronic components, while we also carry out research and development activities to examine the applicability of various technologies including interfaces such as [...] deep learning and big data analysis to the field of home entertainment," the 2021 report reads.

Switch 4K: Nintendo R&D has been spending on deep learning since 2019 26

Deep learning is the process that makes NVIDIA's Deep Learning Super-Sampling (DLSS) tech possible, a kind of digital wizardry that can upscale a 1080p image to 4K resolution (as well as improve in-game frame rates).

The best part about DLSS is that it reduces rendering costs. Devs can now use lower resolution textures in their games and rely on DLSS to up-sample pixel quality. Developers no longer have to rely solely on processing power and can shave off a lot of VRAM, GPU, and CPU performance requirements for their games.

DLSS reduces the pixel workload required by hardware, including graphics buffering, lighting, and other intensive effects. "The more pixel-bound you are, the more performance benefit you'll get."

This sounds like a perfect solution for a weaker, mobile SoC-powered handheld like the Switch. So far, NVIDIA has only used DLSS in its dedicated graphics cards that feature

A special type of AI network, called a convolutional autoencoder, takes the low resolution current frame, and the high resolution previous frame, to determine on a pixel-by-pixel basis how to generate a higher quality current frame.

During the training process, the output image is compared to an offline rendered, ultra-high quality 16K reference image, and the difference is communicated back into the network so that it can continue to learn and improve its results. This process is repeated tens of thousands of times on the supercomputer until the network reliably outputs high quality, high resolution images.

Switch 4K: Nintendo R&D has been spending on deep learning since 2019 25Switch 4K: Nintendo R&D has been spending on deep learning since 2019 27

DLSS 1.0 required each game to be trained individually.

It was heavily rumored the upgraded Switch Pro (aka Switch 4K) would use DLSS to allow 4K gaming at acceptable frame rates. This would require Tensor Cores, a special kind of dedicated AI accelerator built into newer NVIDIA graphics cards like the RTX 20 and 30 series. The Tegra X1 wouldn't work for deep learning; the newer Switch would need an upgraded SoC, the very kind of chip that was mentioned in Bloomberg's original report.

DLSS is enabled by Tensor Cores which are found in three architectures:

  • First generation Tensor Cores - (Volta)
  • Second generation Tensor Cores (Turing) + First generation dedicated Raytracing (RT) Cores - GeForce GTX 16 series, RTX 20 series
  • Third generation Tensor Cores (Ampere) + Second generation dedicated Raytracing (RT) Cores - GeForce RTX 30 series

There are two different iterations of DLSS, both of which require Tensor Cores:

  • DLSS 1.0 - AI needs to be trained specifically for each game, very time consuming
  • DLSS 2.0 - Trains using non-game-specific content, delivering a generalized network that works across games

NVIDIA is also working on new Ultra Quality modes in DLSS 2.2.9.

Investigations have been quite complicated and no one knows for sure which exact chip the Switch Pro will use.

There's a possibility Nintendo and NVIDIA will adjust the Xavier SoC, which is outfitted with a Volta GPU with first-generation Tensor cores, in a smaller form factor to meet the Switch's needs (Xavier's die size is a substantial 350mm squared compared to the Tegra X1's smaller 118mm squared--the X1 is a lot better for a handheld).

Switch 4K: Nintendo R&D has been spending on deep learning since 2019 23

Volta wasn't used in commercial hardware outside of the $3,000 Titan V, and was mostly used in HPC applications like the Tesla V100 datacenter accelerator as well as the automotive Xavier SoC.

It's this automotive chip that may be used in the Switch Pro, the same way the Tegra X1 was used in the base 2017 Switch (and the 2017 revision with the 16nm Tegra X1+).

However there's also a possibility Nintendo would want to embrace a more future-proof chip and opt for a Turing or an Ampere-based SoC...or something beyond.

Right now NVIDIA doesn't make a Turing-based Tegra, but it's possible they could create one specifically for Nintendo. The Switch has proven to be a sales monster with 84.59 million units sold worldwide and it's safe to say that the platform is a good bet for NVIDIA.

Switch 4K: Nintendo R&D has been spending on deep learning since 2019 24

NVIDIA's new Ampere-based Orin line has a 15W camera chip that could be a good fit for the Switch Pro.

NVIDIA has an upcoming Ampere-based SoC called Orin that's in development, and it could theoretically tap third-generation Tensor Cores for boosted DLSS performance.

However, reliable Twitter tecchie kopite7kimi indicates the Switch Pro could be powered by NVIDIA's new Ada Lovelace GPU built on Ampere Next.

Theoretically, the Switch Pro could use a special customized Orin SoC, possibly built on the low-powered 15W model for cameras for significantly lowered TDP.

Kopite also says the new 4K Switch could use a custom T239 Orin Tegra SoC with the Black Knight codename

No one knows what Nintendo will do next, but the deep learning R&D is pretty interesting.

Switch 4K: Nintendo R&D has been spending on deep learning since 2019 2
Buy at Amazon

Nintendo Switch with Neon Blue and Neon Red JoyCon

TodayYesterday7 days ago30 days ago
CAD $387.40CAD $314.99CAD $389.99
* Prices last scanned on 7/14/2024 at 3:45 pm CDT - prices may not be accurate, click links above for the latest price. We may earn an affiliate commission.
NEWS SOURCE:nintendo.co.jp

Derek joined the TweakTown team in 2015 and has since reviewed and played 1000s of hours of new games. Derek is absorbed with the intersection of technology and gaming, and is always looking forward to new advancements. With over six years in games journalism under his belt, Derek aims to further engage the gaming sector while taking a peek under the tech that powers it. He hopes to one day explore the stars in No Man's Sky with the magic of VR.

Newsletter Subscription

Related Tags