How to Set Up a GPU Kubernetes Cluster on Harch Corp
Step-by-step guide to setting up a GPU Kubernetes cluster on Harch Corp for AI/ML workloads.
Provision GPU instances
Create Harch Corp GPU instances (H100 or A100) with Ubuntu 22.04. Minimum 3 nodes for HA. Choose instance sizes based on your workload (e.g., 8x H100 per node for large model training).
Install Kubernetes
Use kubeadm to initialize the control plane on the master node. Join worker nodes to the cluster. Install Calico or Cilium for networking. Install MetalLB for load balancing (bare metal).
Install NVIDIA GPU Operator
The GPU Operator automates GPU management on Kubernetes: `kubectl apply -f https://raw.githubusercontent.com/NVIDIA/gpu-operator/main/deployments/gpu-operator/nvidia-gpu-operator.yml`. This installs GPU drivers, container runtime, device plugin, and DCGM monitoring.
Verify GPU access
Run a test pod: `kubectl run gpu-test --image=nvidia/cuda:12.0-base --command -- nvidia-smi`. Verify the pod can see GPUs. Check node capacity: `kubectl describe node | grep -A 10 Capacity` should show nvidia.com/gpu.
Set up storage
Install a storage provider (Longhorn, Rook/Ceph, or NFS). Create PersistentVolumes for model weights, datasets, and checkpoints. Use NVMe storage for high-performance I/O (training data).
Configure autoscaling
Install Cluster Autoscaler to add/remove nodes based on demand. Install NVIDIA GPU Autoscaler for GPU-specific scaling. Set up Horizontal Pod Autoscaler for workload scaling. Configure scale-to-zero for dev environments.
Deploy your AI workload
Create a Kubernetes Deployment with GPU resource requests: `resources: limits: nvidia.com/gpu: 8`. Use ConfigMaps for configuration, Secrets for API keys. Set up Services for networking, Ingress for external access.
Set up monitoring
Install Prometheus + Grafana for cluster monitoring. Install NVIDIA DCGM-Exporter for GPU metrics. Set up alerting for: GPU utilization, memory usage, node health, pod failures. Use Loki for log aggregation.