Kubernetes for AI Workloads: GPU Scheduling, Model Serving, and Auto-Scaling Inference Pods

Kubernetes wasn't designed for AI workloads, but with the right configurations it becomes a powerful platform for serving models at scale.

1. GPU-Aware Pod Scheduling

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-inference
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: vllm
        image: vllm/vllm-openai:latest
        resources:
          limits:
            nvidia.com/gpu: 1
          requests:
            memory: "32Gi"
            cpu: "8"
        env:
        - name: MODEL
          value: "meta-llama/Llama-3.3-70B"
      nodeSelector:
        gpu-type: "a100"
      tolerations:
      - key: "nvidia.com/gpu"
        operator: "Exists"
        effect: "NoSchedule"

2. GPU-Metrics Auto-Scaling

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: llm-inference-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: llm-inference
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Pods
    pods:
      metric:
        name: gpu_utilization
      target:
        type: AverageValue
        averageValue: "75"

Kubernetes provides the orchestration backbone for serving AI models with enterprise-grade reliability, auto-scaling, and zero-downtime deployments.