NVIDIA H100 GPU FAQ — Everything You Need to Know
Complete H100 FAQ: specs, performance, pricing, use cases, alternatives. All your H100 questions answered.
What are the NVIDIA H100 GPU specifications?
H100 specs: Architecture: Hopper. Memory: 80GB HBM3. Bandwidth: 3.35 TB/s. FP8: 3,958 TFLOPS. FP16: 1,979 TFLOPS. FP32: 67 TFLOPS. TDP: 700W. Interconnect: NVLink 4.0 (900 GB/s). Form factor: SXM5 or PCIe 5.0.
How fast is the H100 compared to A100?
H100 is 3-4x faster than A100 for training and up to 30x faster for inference (with FP8). The Transformer Engine in H100 automatically switches between FP8 and FP16 for optimal LLM performance. For a 175B parameter model: A100 takes ~30 days to train, H100 takes ~8 days.
What is H100 FP8 and why does it matter?
FP8 (8-bit floating point) is a new precision format in H100 that delivers 2x speedup over FP16 with minimal accuracy loss. For LLM inference, FP8 enables 30x faster throughput vs FP32. The Transformer Engine automatically selects the best precision per layer.
What workloads is the H100 best for?
H100 excels at: (1) Large language model training (GPT, LLaMA, Claude), (2) LLM inference and serving, (3) Computer vision (stable diffusion, DALL-E), (4) Recommendation systems, (5) Scientific computing (CFD, molecular dynamics), (6) High-frequency trading analytics.
How much power does an H100 GPU consume?
H100 SXM5 has a TDP of 700W; H100 PCIe has a TDP of 350W. A server with 8x H100 SXM5 consumes ~5.6kW for GPUs alone (plus ~1kW for CPU, memory, networking). This requires liquid cooling in most datacenter environments.
What cooling does the H100 require?
H100 SXM5 (700W) requires liquid cooling — either direct-to-chip cold plates or immersion cooling. H100 PCIe (350W) can use advanced air cooling but liquid is recommended for density. Harch Corp uses direct-to-chip liquid cooling for all H100 deployments, supporting up to 50kW/rack.
Can I use H100 for inference, not just training?
Yes. H100 is excellent for inference, especially for large models. With FP8 and TensorRT-LLM, H100 can serve 70B parameter models at 2,000+ tokens/second. For smaller models (7B-13B), A100 or even L4 may be more cost-effective for inference.
What is NVLink and why is it important for H100?
NVLink 4.0 is NVIDIA's high-speed GPU interconnect, providing 900 GB/s bidirectional bandwidth between H100 GPUs (vs 64 GB/s for PCIe 5.0). NVLink enables: (1) Faster distributed training, (2) Larger models that span multiple GPUs, (3) Efficient tensor parallelism for LLMs.
How many H100 GPUs do I need to train a 70B parameter model?
To train a 70B parameter model (like LLaMA 2 70B): (1) Fine-tuning: 8x H100 (80GB) for ~10 hours, (2) Full training from scratch: 64-256x H100 for 2-4 weeks. Cost estimate: 8x H100 fine-tuning = ~$200; 256x H100 full training = ~$200K-500K.
What are H100 alternatives?
H100 alternatives: (1) H200 — 141GB memory, better for very large models, (2) A100 — cheaper, good for inference on <30B models, (3) B200 — next-gen Blackwell, 192GB, 4x faster, (4) AMD MI300X — competitive for some workloads, (5) Intel Gaudi 3 — cost-effective alternative.