Oriel — AI & technology worth noticing, every day.

Fitting Four 32 GB Resizable Bars Behind a PCIe Switch on Linux

Web page · https://doug.sh/posts/gpu-bars-behind-a-pcie-switch

First seen by Oriel on .

AI summary

This post describes a Linux kernel-module workaround that maps four 32 GB AMD Radeon AI PRO R9700 BARs behind a PCIe switch, enabling GPU peer-to-peer transfers.

  • A Broadcom PEX88096 gives four AMD Radeon AI PRO R9700s and two RTX 3090s eight lanes each, allowing cards to copy without CPU involvement.
  • Linux 7.0’s bridge-window sizing omitted nearly 32 GB gaps between GPU BARs, causing “can’t assign; no space” failures; an unmerged 14-patch series fixes this regression.
  • The gpu-rebar-layout initramfs module sets 32 GB BAR sizes, removes and rescans the switch’s root port, and raises compatible BAR alignments before drivers load.
  • After reboot, all R9700s exposed 32 GB BARs; peer-to-peer transfers reached 14.0 GB/s per pair and 54.7 GB/s for four-card ring copying.
  • The workaround depends on kernel internals, requires resizing before GPU drivers load, and removing or rescanning an already-driven GPU caused a lockup.

#pci express #resizable bar #linux kernel #rocm

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Points

Score reported by the source.

  • Hacker News: 92, latest observation ; +84 over observed period since

Comments

Comment count reported by the source.

  • Hacker News: 37, latest observation ; +33 over observed period since

Aliases

SourceExternal IDURL
Hacker News50468720https://doug.sh/posts/gpu-bars-behind-a-pcie-switch/ Read discussion