Self-Hosting DeepSeek and Llama on Proxmox and Kubernetes: Bare-Metal GPU Clustering Guide

While public cloud providers (AWS, GCP, Azure) offer elastic GPU instances, their hourly pricing can quickly drain operational budgets. For teams generating millions of tokens daily, running on rented H100s at $3.50/hour per GPU equates to over $25,000 monthly for a standard 8-GPU node.

Building an On-Premises Bare-Metal GPU Cluster pays for itself in less than 6 months.

In this comprehensive guide, we walk through constructing a private enterprise AI cluster running on Proxmox Virtual Environment (PVE) and Kubernetes (k3s), configured for high-throughput vLLM serving.

Hardware Architecture: Consumer vs Enterprise Silicon

For on-premises inference, two primary hardware routes exist:

1. The Enterprise Route: 4x or 8x NVIDIA RTX 6000 Ada (48GB VRAM each) or refurbished NVIDIA A100 80GB PCIe cards.
2. The High-Density Prosumer Route: 4x NVIDIA GeForce RTX 4090 (24GB VRAM each) using PCIe bifurcation and water cooling, yielding 96GB total VRAM capable of serving 70B models in FP8 precision for under $10,000 total build cost.

Cluster Physical Topology:
┌─────────────────────────────────────────────────────────────┐
│              Supermicro / ASUS 4U Rack Server               │
│  Dual AMD EPYC 9354 (64 Cores) • 512GB DDR5 ECC RAM         │
│                                                             │
│  PCIe Gen 5 Slot 1: [NVIDIA RTX 4090 / 6000 Ada - 16x]      │
│  PCIe Gen 5 Slot 2: [NVIDIA RTX 4090 / 6000 Ada - 16x]      │
│  PCIe Gen 5 Slot 3: [NVIDIA RTX 4090 / 6000 Ada - 16x]      │
│  PCIe Gen 5 Slot 4: [NVIDIA RTX 4090 / 6000 Ada - 16x]      │
│                                                             │
│  Networking: Dual 100GbE Mellanox ConnectX-6 NICs           │
└──────────────────────────────┬──────────────────────────────┘
                               │
                               ▼
               [Proxmox VE 8.3 Hypervisor]
                               │
               ┌───────────────┴───────────────┐
               ▼                               ▼
       [VM 101: Kubernetes Node]      [VM 102: Storage Node]
       (4x GPU Passthrough via IOMMU) (Ceph / TrueNAS NVMe Pool)

Proxmox IOMMU Configuration for GPU Passthrough

To pass multiple GPUs directly into a virtualized Kubernetes worker with zero hypervisor overhead:

# 1. Enable IOMMU in GRUB (/etc/default/grub)
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream,multifunction"
update-grub

# 2. Blacklist host Nouveau and NVIDIA drivers (/etc/modprobe.d/pve-blacklist.conf)
blacklist nouveau
blacklist nvidia
blacklist nvidia*

# 3. Bind GPUs to VFIO driver (/etc/modprobe.d/vfio.conf)
# Find GPU PCI IDs with 'lspci -nn | grep -i nvidia'
options vfio-pci ids=10de:2684,10de:22ba

Kubernetes vLLM Deployment Manifest

Once passed through into the Linux VM, deploy vLLM with NVIDIA Container Toolkit:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-deepseek-qwen-32b
  namespace: ai-serving
spec:
  replicas: 1
  selector:
    matchLabels:
      app: deepseek-32b
  template:
    metadata:
      labels:
        app: deepseek-32b
    spec:
      containers:
      - name: vllm
        image: vllm/vllm-openai:v0.7.2
        args:
          - "--model"
          - "deepseek-ai/DeepSeek-R1-Distill-Qwen-32B"
          - "--tensor-parallel-size"
          - "2"
          - "--max-model-len"
          - "16384"
          - "--gpu-memory-utilization"
          - "0.94"
        resources:
          limits:
            nvidia.com/gpu: 2
        ports:
        - containerPort: 8000
        volumeMounts:
        - mountPath: /root/.cache/huggingface
          name: model-cache
      volumes:
      - name: model-cache
        persistentVolumeClaim:
          claimName: fast-nvme-models-pvc

Financial & Operational Payoff

• Monthly Cloud Cost (AWS g6e.12xlarge): ~$4,800/mo.
• On-Prem Server Build (amortized over 24 months): ~$650/mo including electricity (1,200W sustained load at $0.12/kWh).
• Net Annual Savings: Over $49,000 USD while maintaining 100% internal data sovereignty.