Fitting Four 32 GB Resizable Bars Behind a PCIe Switch on Linux
Web page · https://doug.sh/posts/gpu-bars-behind-a-pcie-switch
First seen by Oriel on .
AI summary
This post describes a Linux kernel-module workaround that maps four 32 GB AMD Radeon AI PRO R9700 BARs behind a PCIe switch, enabling GPU peer-to-peer transfers.
- A Broadcom PEX88096 gives four AMD Radeon AI PRO R9700s and two RTX 3090s eight lanes each, allowing cards to copy without CPU involvement.
- Linux 7.0’s bridge-window sizing omitted nearly 32 GB gaps between GPU BARs, causing “can’t assign; no space” failures; an unmerged 14-patch series fixes this regression.
- The gpu-rebar-layout initramfs module sets 32 GB BAR sizes, removes and rescans the switch’s root port, and raises compatible BAR alignments before drivers load.
- After reboot, all R9700s exposed 32 GB BARs; peer-to-peer transfers reached 14.0 GB/s per pair and 54.7 GB/s for four-card ring copying.
- The workaround depends on kernel internals, requires resizing before GPU drivers load, and removing or rescanning an already-driven GPU caused a lockup.
#pci express #resizable bar #linux kernel #rocm
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Points
Score reported by the source.
- Hacker News: 92, latest observation ; +84 over observed period since
Comments
Comment count reported by the source.
- Hacker News: 37, latest observation ; +33 over observed period since
Aliases
| Source | External ID | URL |
|---|---|---|
| Hacker News | 50468720 | https://doug.sh/posts/gpu-bars-behind-a-pcie-switch/ Read discussion |