About the NVIDIA H100 SXM5 GPU

The NVIDIA H100 is based on NVIDIA's latest GPU architecture, hopper. The H100 is engineered to excel in tasks involving deep learning, data analytics, and scientific computing.

The H100 is fast. Its 528 fourth-generation Tensor Cores and 16896 CUDA Cores enable it to provide breakthrough performance in AI model training and inference.

Apart from pure performance, its massive 80 GB of VRAM and 3.35 TB/s of memory bandwidth make it ideal for large-scale LLM, data analytics, and scientific computing tasks that require large amounts of fast memory.

The H100 also boasts advanced features like Multi-Instance GPU (MIG), enabling it to efficiently handle multiple workloads simultaneously — up to 7 10GiB VRAM instances per physical GPU. Overall, the NVIDIA H100 represents a significant leap forward in GPU technology, offering unmatched performance and efficiency for the most demanding computing tasks.

See full data sheet.

19,694 tok/sec LLaMa 7B

A token is a word or subword, so the H100 can generate 19,694 words per second on the LLaMa 7B model.

16,896 CUDA Cores

CUDA cores are the basic processing units of NVIDIA GPUs. The more CUDA cores, the better.

528 Tensor Cores

Tensor cores are specialized processing units that are designed to efficiently execute matrix operations, used in deep learning.

3,958 Teraflops FP8

With sparsity.

80 GB VRAM @ 3.35 TB/s

The more VRAM, the more data a GPU can store at once.