GPU passthrough into LXC, without the pain

Every guide to passing GPUs into a Proxmox LXC container ends at the same place: the container boots, nvidia-smi prints a table with your cards in it, and the guide congratulates you. That part is genuinely easy, and there are a dozen good writeups of it.

This post is about everything after that table prints. Specifically: four RTX 5060 Ti 16GB cards in one container, and the PCIe lane topology that turned out to decide which card is allowed to do which job. Not as a benchmark footnote - as a hard scheduling constraint that reshaped the training layout twice and cost me a production outage when I stopped paying attention to the hardware underneath it.

If you’re running one GPU, none of this applies to you and you should enjoy that. If you’re running four on a consumer board, this is the thing the guides don’t tell you.

The box

The host is the machine from the homelab build post on this site: a Ryzen 9 9950X with 128GB of DDR5 and four RTX 5060 Ti 16GB cards on an open frame, 64GB of VRAM in total. It runs Proxmox, and all four GPUs are handed to a single research container which gets 24 CPUs and 80GB of RAM. That container does model training, backtesting, and it serves live inference. Everything else on the host is CPU-only.

One container holding all four cards is a deliberate choice, and it’s the right one for a homelab. GPUs do not divide cleanly across containers - the driver, the device nodes, and the CUDA runtime all want to agree with each other, and every boundary you draw is another place for that agreement to break. Put the cards where the work is, and schedule inside the container instead of across the hypervisor.

Verification, from the host, is the only part of the setup worth memorising:

# Do all four cards enumerate inside the container?
pct exec 110 -- nvidia-smi -L

# How much VRAM is each one holding?
pct exec 110 -- nvidia-smi --query-gpu=memory.used,memory.total --format=csv

Run the first one after every host kernel change. I will come back to why.

The lane map is the whole story

Do not read your lane widths off the board’s spec sheet. The number that matters is the width each card actually negotiated, and the only place that number lives is the host:

# Per-device PCIe detail, including the negotiated link width
lspci -vv

That dumps every device on the bus, so filter it down to the LnkSta line under each card’s address. LnkSta is the link’s status - what the card and the slot settled on at boot, not what the slot is rated for. Here is what came back, and it ended up governing everything:

GPU	PCIe	Role
0	x8	Training
1	x8	Training
2	x1	Training, solo only
3	x1	Reserved: inference + testing

Four identical cards. Four very different jobs. Nothing about the GPUs themselves explains that table - identical silicon, wildly non-identical plumbing.

The naive reading of this table is “GPUs 2 and 3 will load models more slowly.” That reading is wrong in the way that matters, and it’s wrong because of DDP.

Distributed Data Parallel synchronises gradients between GPUs on every backward pass. Each rank computes its own gradients on its own shard of the batch, and then all ranks perform an all-reduce so every copy of the model gets the same averaged update before the next step. That all-reduce is not a startup cost you pay once. It happens every step, for the full size of the gradient tensor, for the entire run.

Put that traffic on a x1 link and the arithmetic stops being subtle. GPUs 0 and 1, both at x8, are a symmetric DDP pair: the same width on both ranks, so the all-reduce costs each side the same. GPUs 2 and 3 must never be paired in DDP - the bus bottlenecks the all-reduce, and both cards spend the run waiting on each other rather than computing. The correct use of a x1 card is a job that never needs to talk to another card: single-GPU training, or inference.

That is the sentence I’d have paid money for before I built this thing. A lane map isn’t a performance table. It’s a list of which pairings are legal.

The layout that came out of it

The first layout that respected the lane map was the obvious one: put the two x8 cards in a 2-GPU DDP pair, run a second independent job solo on GPU 2, keep GPU 3 for inference. Two training jobs in flight.

The layout that actually runs today doesn’t use DDP at all:

GPU 0 ── Slot 0: solo training (CUDA_VISIBLE_DEVICES=0, PCIe x8)
GPU 1 ── Slot 1: solo training (CUDA_VISIBLE_DEVICES=1, PCIe x8)
GPU 2 ── Slot 2: solo training (CUDA_VISIBLE_DEVICES=2, PCIe x1)
GPU 3 ── Reserved: TFT inference service + ad-hoc testing

Three independent single-GPU training jobs, each pinned to one card with CUDA_VISIBLE_DEVICES, each chewing through its own walk-forward fold. Three folds per round instead of the two the DDP layout managed, from the same four cards, by removing the distributed training.

The reason is a chain of constraints, and it’s worth following because the same chain probably applies to you.

It starts with system RAM. The PyTorch DataLoader forks worker processes, and with the dataset sizes involved here, num_workers > 0 reliably OOM’d the container’s system RAM rather than its VRAM. So num_workers=0 - the training process loads its own batches, synchronously, on the main process.

That single setting is what kills DDP. Normally the gradient all-reduce partly hides behind data loading: while the collective is in flight, worker processes are already preparing the next batch, so the sync cost is overlapped rather than added. With num_workers=0 there is no data-loading parallelism left to hide behind, so the all-reduce becomes pure serial overhead on the critical path. A 2-GPU DDP pair plus one solo GPU loses to three solo GPUs, and it isn’t close.

The RAM budget works out comfortably: three training slots at roughly 7GB of system RAM each is about 21GB, against the container’s 80GB, leaving plenty of headroom for the inference service and everything else. The card reserved for inference holds a forecast service in about 1.3GB of VRAM, so “reserved” costs almost nothing in capacity and buys the thing you actually want, which is that a training job crashing never takes production inference down with it.

That reservation is not a nicety. Keep reading.

VRAM is a cliff, not a slope

The other constraint the lane map interacts with is VRAM, and it’s the one that punishes attention models specifically. Transformer attention is O(n²) in sequence length, so VRAM usage for the temporal fusion transformers here scales with encoder length times batch size - and encoder length is set by how much history the strategy needs, which is a modelling decision, not a tuning knob.

This is the record from the DDP era, before the three-solo layout above replaced it: the configurations that fit on 16GB cards at each timeframe, and how many cards each job was spread across to get there without an OOM. Encoder length is in bars:

Timeframe	Encoder	Batch	GPUs
1d	7	256	4
4h	42	256	4
1h	168	97	4
30m	192	85	2
15m	384	42	2

Read the last two rows. The sub-hourly timeframes are the ones that need the longest encoders in bar terms, and they’re the rows where the GPU count drops to two. Running 30m or 15m across all four cards with DDP OOM-killed repeatedly. The fix was two changes at once: cut the context window to roughly four days of history - 192 bars at 30m, 384 at 15m - and stop trying to spread the job over four cards. Sub-hourly strategies hold positions for one to eight hours; they were being fed seven days of context they had no use for.

The general shape: a bigger cluster does not rescue you from an O(n²) memory curve. It gives you more places for the same OOM to happen, and it adds all-reduce traffic to a bus that in this box is x1 on half the cards.

And then three of the four fans stopped

Everything above reads like a system under control. Here is the part where it wasn’t.

During an ablation campaign, a routine status check on three in-flight training runs found this:

• GPU 3, the reserved inference card, had dropped off the PCIe bus. nvidia-smi reported Unable to determine the device handle for gpu 0000:13:00.0: Unknown Error. Production forecasts had been stale for hours.
• GPU 2 had died mid-run with torch.AcceleratorError: CUDA error: unspecified launch failure. Its handle was gone too.
• GPU 1 was a zombie. The process sat in Sl state at 0% CPU while nvidia-smi cheerfully reported 99% utilisation and a commanded 32% fan speed. The log hadn’t been written to in over twelve hours. The fan was not physically spinning.
• GPU 0 was fine, and its fan was the only one moving.

dmesg was flooded with NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f.

That RPC failure is the GSP - the GPU System Processor, the on-card microcontroller the modern NVIDIA driver delegates most GPU management to, including fan curves. When a card’s GSP firmware faults, the fan PWM is left at whatever value was last written, which during idle is low. If that card then takes load, the fans never ramp, the die heats, and the fault propagates to neighbouring cards through shared driver state.

Every observation fits: fans stuck off only on the faulted cards, the RPC flood, a cascade that began under load rather than at boot. Each card has its own PSU cable and the fans are PCB-powered, which rules out a wiring fault as a single explanation.

The fans were the symptom. The lie was that nvidia-smi was still cheerfully reporting a commanded fan speed for a fan that was not moving. Every dashboard I had was reading the driver’s intent, and the driver’s intent was fine.

The recovery, and the trap inside it

Nothing gentle worked. nvidia-smi --gpu-reset returned No devices were found, because the handles were already gone and the driver had nothing to address. A driver reload was blocked by nvidia-persistenced sitting in Rs state inside a wedged kernel ioctl, ignoring SIGKILL through systemd’s full 90-second stop timeout.

So: reboot the host. And this is where a Proxmox default stretched the recovery across four reboots and several hours.

kexec-pve.service pre-loads the newest installed kernel into memory, so systemctl reboot performs a warm kexec jump rather than a real hardware reboot. It’s enabled by default. On a normal host it’s a nice latency win. On a GPU host it means your reboots never POST, which means firmware state - including GSP state - is never reset. You are rebooting the operating system on top of the exact hardware fault you are trying to clear.

It also means GRUB is irrelevant. grub-reboot, edits to /etc/default/grub, update-grub, efibootmgr - all of it does nothing on the kexec path. Four consecutive reboots came back on the same kernel while I edited boot configuration that was never being read.

Worse, the kernel it kept landing on had no NVIDIA driver at all. The driver had been installed months earlier with NVIDIA’s .run installer, which builds modules for exactly one kernel version. Every kernel upgrade since had silently shipped no nvidia.ko for the new kernel. The machine had been fine for 37 days purely because it hadn’t rebooted into a newer kernel in that window. The thermal incident is what finally made it reboot.

What actually worked:

systemctl disable --now kexec-pve.service
kexec -u                 # unload any pre-loaded kernel
systemctl reboot         # now this is a real POST

The host finally did a hardware-level POST, GRUB selected the known-good kernel, the driver loaded, all four cards enumerated, and the fans came up to a commanded 40-70% PWM at 25-26°C idle - confirmed spinning by looking at them, which after that day felt like the only trustworthy instrument in the building.

The runs that were relaunched afterwards completed cleanly, about 32 hours of wall time, within 1% of the pre-crash compute rate, with no CUDA errors or RPC failures across roughly 40 GPU-hours. Whether the thermal excursion silently corrupted any training math before the crash is, honestly, unfalsifiable with the data I have: a seeding change landed between the pre- and post-crash batches, so the two aren’t bit-reproducible against each other even at identical seeds. I’d rather say that than imply a comparison I can’t make.

What I’d tell you to change today

Five things, in the order they’d have saved me time:

Disable kexec-pve.service on any GPU host. The cost is a slower reboot. The benefit is that systemctl reboot actually resets firmware state. On a box where GSP can wedge, that trade is not close. One wrinkle if you go as far as masking it: the unit file isn’t owned by any dpkg package here, and systemctl mask refuses to overwrite a real file, so the file has to be backed up and removed before the mask symlink can be created. The upside of it not being package-owned is that no upgrade postinst can quietly re-enable it.

Use a DKMS-packaged driver, not the .run installer. DKMS rebuilds the module for every installed kernel automatically. The .run installer builds for one kernel and then quietly stops being true. If you’re stuck on .run for now, treat “re-run the installer against the new kernel before rebooting” as a mandatory step of every kernel upgrade, and pin a known-good kernel until you’ve migrated.

Alert on the physical state, not the driver’s intent. Three cheap checks that would have paged in minutes instead of the twelve-plus hours a fanless zombie ran:

# Fan stopped while hot
nvidia-smi --query-gpu=fan.speed,temperature.gpu --format=csv,noheader
#   alert if any GPU shows fan.speed=0 while temperature.gpu > 55°C for >60s

# A card fell off the bus
nvidia-smi -L                      # alert if fewer than 4 GPUs enumerate

# GSP is unhappy
dmesg | grep 'rpcSendMessage failed'   # alert on any hit

Reserve a card for inference, but know what that reservation does and doesn’t buy. It buys you blast radius on the scheduling side: a training job that OOMs or crashes cannot take the forecast service with it. It buys you nothing against a GSP cascade, which propagates through shared driver state regardless of what you scheduled - the reserved card was the first one to fall off the bus. What the reservation did give me here was a degraded failure rather than a hard one: the forecast service went stale for about two and a half hours, and the downstream consumer kept running on stale input rather than stopping. Whether that’s better than failing loudly depends entirely on the consumer, and it’s worth deciding deliberately rather than discovering during an incident.

Consider turning GSP off. NVreg_EnableGpuFirmware=0 forces software-mode GPU management and avoids the GSP path entirely at some performance cost. I haven’t committed to it here because I want the benchmark against current training times first, and I’d rather tell you it’s an open question than pretend I’ve measured it.

The thing underneath all of it

Both halves of this post are the same mistake in two costumes.

The lane map is invisible from inside the container. nvidia-smi shows you four identical cards with identical VRAM and gives you no reason to think GPU 2 and GPU 3 are different animals from GPUs 0 and 1. The fan state is invisible from inside the driver - it reports the PWM it commanded, not the PWM the fan obeyed. In both cases the software layer presents a clean abstraction, and the abstraction is a summary of intent rather than a report of the world.

Passing GPUs into a container is easy because that’s the layer where everything is designed to look uniform. Running four of them for real means knowing precisely where that uniformity is a fiction: which links are x1, which card is the one you cannot afford to lose, and which reading on your dashboard is a wish rather than a measurement.

---

Research write-up from a homelab, not a datacenter. All numbers here come from this box’s own operations logs. Subscribe if you want the rest of them, incidents included - that’s rather the point of this lab.