--- pattern_id: PAT-010 title: "PVE host OOM-kill freezes guest VM as zombie (RSS collapse signature)" category: infrastructure severity: high status: active first_observed: 2026-09 last_updated: 2026-09-30 related_systems: [proxmox-cluster, rke2-kubernetes] related_solution_docs: [docs/solutions/bug-fixes/2026-09-30-proxmox6-oom-frozen-vm102-worker05.md] related_skills: [rke2-cluster-administration, systematic-debugging] --- # PAT-010: PVE host OOM-kill freezes guest VM as zombie ## Symptom - K8s node NotReady, kubelet heartbeat stops abruptly - PVE shows VM `running`, but: no ping, no SSH, qemu-guest-agent dead, tap-interface TX counters static - **Signature:** `qm status --verbose` → kvm RSS collapses to a tiny fraction (<5%) of assigned RAM — guest kernel no longer touches its memory - Downstream alerts (DS misscheduled, workload CrashLoops) fire, but NOTHING alarms on the actual OOM ## Root Cause Host RAM oversubscription (allocations >> physical RAM). OOM-killer picks the largest anon-RSS process = biggest kvm. After TWO consecutive kills of the same guest (HA auto-restarts in between), the revived QEMU comes up but the guest kernel stays inert → zombie VM. ## Mitigation 1. Relieve host pressure FIRST (move movable tenants away) — otherwise the unfreeze re-boots the guest into the same thrash. 2. Hard cycle the VM: `qm stop` + `qm start` (respect HA guards). 3. Re-query `ha-manager status` afterwards — HA may relocate the VM. 4. Uncordon the K8s node; orphan DS pods self-reconcile. ## Prevention - Allocation ceiling per PVE host (≤ ~70% of RAM) enforced in placement/ IaC; swap is a buffer, not capacity. - PSI/OOM alerting per PVE node in Prometheus — OOM kills are currently invisible to alerting (noticed only via downstream K8s symptoms). ## Evidence - 2026-09-30: proxmox6 (16 GB, ~47.8 GB allocated): OOM-killed rke2-worker-05 kvm twice (18:43:38, 20:44:12 UTC on 29.09.), second revival = zombie (RSS 239 MB / 12 GB). Fixed via Laya-CT migration + hard recycle; HA relocated VM to ms-a2-2. Full RCA in related doc.