Intel's new Gaudi 3 AI accelerator launched: cheaper than NVIDIA H100 AI GPU, but also slower

Intel has officially launched its Gaudi 3 AI accelerator, with the new AI chip coming in slower than NVIDIA's dominant H100 and its new HBM3E-fueled H200 AI GPUs, meaning Intel is aiming its Gaudi 3 by pushing that it's cheaper, and has a lower total cost of ownership (TCO).

4

Inside, the new Intel Gaudi 3 AI accelerator features two chiplets with 64 tensor processor cores (TPCs, 256x256 MAC structure with FP32 accumulators), eight matrix multiplication engines (MMEs, 256-bit wide vector processor), and 96MB of on-die SRAM cache with a 19.2TB/s bandwidth.

Gaudi 3 also features 24 x 200GbE networking interfaces and 14 media engines, with the media engines capable of handling H.265, H.264, and VP9 to support vision processing. Intel's new Gaudi 3 AI accelerator features 128GB of HBM2E memory with up to 3.67TB/sec of memory bandwidth.

Intel says that its new Gaudi 3 AI accelerator offers up to 1856 BF16/FP8 matrix TFLOPS as well as up to 28.7 BF16 vector TFLOPS at around 600W TDP, which when compared to the NVIDIA H100 AI GPU, the Gaudi 3 is slightly slower with BF16 matrix performance (1856 vs 1979 TFLOPS), 2x slower FP8 matrix performance (1856 vs 3958 TFLOPS) and massively, massively slower BF16 vector performance (28.7 vs 1979 TFLOPS), yikes.

• Read more: Intel discounting Gaudi 3 AI accelerator: $16K against NVIDIA H100 AI GPU for $30K+
• Read more: Intel Gaudi 3 AI accelerator: 128GB HBM2e at up to 3.7TB/sec, up to 900W power

Justin Hotard, Intel executive vice president and general manager of the Data Center and Artificial Intelligence Group, said: "Demand for AI is leading to a massive transformation in the datacenter, and the industry is asking for choice in hardware, software, and developer tools. With our launch of Xeon 6 with P-cores and Gaudi 3 AI accelerators, Intel is enabling an open ecosystem that allows our customers to implement all of their workloads with greater performance, efficiency, and security".

4

Intel highlights its Gaudi 3 AI accelerator: Specifically optimized for large-scale generative AI, Gaudi 3 boasts 64 Tensor processor cores (TPCs) and eight matrix multiplication engines (MMEs) to accelerate deep neural network computations. It includes 128GB of HBM2e memory for training and inference, and 24 x 200Gb Ethernet ports for scalable networking. Gaudi 3 also offers seamless compatibility with the PyTorch framework and advanced Hugging Face transformer and diffuser models. Intel recently announced a collaboration with IBM to deploy Intel Gaudi 3 AI accelerators as a service on IBM Cloud. Through this collaboration, Intel and IBM aim to lower the total cost of ownership to leverage and scale AI, while enhancing performance.