PCIe bandwidth is the per-direction data rate of the link connecting a GPU to the rest of the host. It is largely irrelevant once a model is fully resident in VRAM, and it can become the dominant limit the moment weights, KV cache, or tensor-parallel collectives have to cross the host fabric on every token. That single distinction explains most of the contradictory advice about whether a narrow PCIe link “matters” for local inference.
This article is for anyone running a local model who has been told their PCIe slot is too slow — or who has been told it does not matter and wants to know which answer applies to their setup. It covers how the number is derived, when offloading turns it into the bottleneck, and how to measure your own link rather than guess.
How PCIe bandwidth is calculated
A PCIe link’s capacity is the product of three things: the number of lanes, the per-lane signalling rate of the generation, and the encoding overhead. PCIe 3.0 signals at 8.0 GT/s, PCIe 4.0 at 16.0 GT/s, and PCIe 5.0 at 32.0 GT/s; all three use 128b/130b encoding, so roughly two bits in every 130 are consumed by encoding rather than payload.
That works out to approximately 0.985 GB/s of payload per lane per direction on PCIe 3.0, roughly double that on PCIe 4.0, and roughly double again on PCIe 5.0. The figures below are calculated from those signalling rates and encoding ratios. They are encoding-capacity figures, not benchmarks, and they exclude transaction-layer and software overhead — which, as the measured example later shows, is not a small exclusion.
| Link | Approximate GB/s per direction | Practical reading |
|---|---|---|
| PCIe 3.0 x1 | 0.985 | Mining-riser class; highly restrictive for transfers and collectives |
| PCIe 3.0 x4 | 3.94 | Similar headline scale to Thunderbolt 4 PCIe tunnelling, but not equivalent |
| PCIe 3.0 x8 | 7.88 | Intermediate native link; test the resident workload and any TP collectives |
| PCIe 3.0 x16 | 15.75 | The reduced-link state tested in the Qwen TP2 experiment |
| PCIe 4.0 x4 | 7.88 | Same encoding-capacity figure as PCIe 3.0 x8 |
| PCIe 4.0 x8 | 15.75 | Same encoding-capacity figure as PCIe 3.0 x16 |
| PCIe 4.0 x16 | 31.51 | Normal live state of the B70 reference host |
| PCIe 5.0 x16 | 63.02 | Twice the PCIe 4.0 x16 encoding capacity |
| Thunderbolt 4 | 32 Gbit/s PCIe data requirement | About 4 GB/s before tunnelling overhead; Intel cites storage up to 3,000 MB/s |
Two consequences follow immediately. First, generation and lane width multiply: PCIe 3.0 x16 and PCIe 4.0 x8 both land at about 15.75 GB/s per direction. A card reporting “PCIe 4.0” is not necessarily running at 4.0 x16 — many boards, risers, and M.2-to-PCIe adapters negotiate down to x4 or x8, and some slots are physically x16 but electrically x8 or x4.
Second, PCI-SIG’s own rounded figures for PCIe 3.0 — about 1 GB/s per lane per direction, and about 32 GB/s for x16 aggregated across both directions — are consistent with the table, but they are quoted in different units. Comparing a per-direction figure against an aggregate figure is one of the most common sources of confusion when reading spec sheets.
When the link is invisible, and when it becomes the wall
If a model’s weights fit entirely in VRAM and nothing else crosses the host fabric during generation, PCIe bandwidth is close to irrelevant to token throughput. The transfer happens once, at load time. A narrow link makes the model load more slowly; it does not slow down the tokens.
Offloading changes that. CPU offload — including expert offload for mixture-of-experts models — keeps some weights in host memory and streams them to the device as they are needed. In the implementation documented in the experts4bit-qlora offload notes, each layer’s frozen NF4 experts are streamed host-to-device on every forward pass. The same structural problem applies to KV cache offload and to tensor parallelism, where collectives cross the host fabric on every token.
Once a transfer is on the critical path per token, the arithmetic is unforgiving. Take a hypothetical layer whose offloaded weights total 1 GB and which must be transferred once per forward pass over a link measured at 6 GB/s. That transfer alone costs roughly 167 ms — before any GPU compute. Halve the bytes moved through quantization and the cost halves with it. This is an illustrative calculation, not a measurement; the point is that the transfer, not the GPU, sets the floor.
That is why the same link can be described as “fine” and “the bottleneck” in the same week by two people who are both correct. They are running different workloads.
A measured example: PCIe 3.0 x8 under expert offload
The clearest published measurement in the sources available here comes from an offload-diagnostics investigation on an RTX A2000 12 GB installed in a NAS-class host with a Xeon W-1250 on a PCIe 3.0-only platform. The card sat in the chassis’s sole x8 electrical slot, with the full production stack live — load around 45, and the GPU shared with six other dev-agent containers, not quiesced.
Two numbers anchor the analysis:
- Pinned H2D, 256 MB × 20: 6.16–6.18 GB/s.
- Pageable H2D, 256 MB × 20: 4.5–5.5 GB/s.
Against the PCIe 3.0 x8 encoding capacity of about 7.88 GB/s per direction, the pinned measurement is roughly 78% of the theoretical figure. The gap is the transaction-layer and software overhead the capacity table omits — and it is the number every per-layer measurement in that investigation was read against.
The investigation’s conclusion was blunt: the link is the wall, and each copy already saturates it. The team also tested copy consolidation, packing a layer’s four home tensors into one contiguous pinned arena per dtype so a stage issues two H2D copies instead of four. It was bit-identical to the four-copy path, and it did not help throughput, because per-copy fixed overhead was not the limiting factor. The link was.
Two diagnostic flags drove the work and are worth knowing about if you run the same stack: E4B_OFFLOAD_STATS=1 reports CUDA-event-bracketed per-copy timing (bytes, copy count, mean milliseconds, implied GB/s) split by policy, plus prefetch stall versus slack; E4B_OFFLOAD_ARENA=1 enables the consolidated-copy path. A one-shot PCIe-link and H2D-ceiling report prints at load via report_offload_environment.
Treat this as one host’s result, not a universal constant. A quiesced machine, a different chipset, or a different copy size will produce different numbers. What generalizes is the method: establish the link ceiling first, then read every per-copy figure against it.
Terms that are easy to confuse
H2D versus D2H. Host-to-device and device-to-host transfers are separate directions. Most offload workloads are dominated by H2D traffic, but the two are not interchangeable when you are comparing against a spec table.
Pinned versus pageable memory. Pinned (page-locked) host memory is used for efficient DMA transfers. In the measured host, pinned copies ran at 6.16–6.18 GB/s while pageable copies managed 4.5–5.5 GB/s — a difference large enough to change whether a workload is viable.
Encoding capacity versus achieved bandwidth. The table above is arithmetic from signalling rates. Real transfers carry protocol overhead and, on a shared or busy host, contention.
Thunderbolt 4 versus a native slot. Thunderbolt 4 specifies a 32 Gbit/s PCIe data requirement, which is about 4 GB/s before tunnelling overhead; Intel cites storage throughput up to 3,000 MB/s. It is a tunnelled, shared transport rather than a native lane count, so it should not be treated as equivalent to a PCIe 3.0 x4 slot even though the headline scales look similar.
PCIe versus GPU-to-GPU interconnects. NVLink and similar fabric links are a separate topic and are not covered by the sources used here; do not assume PCIe figures apply to them.
How to find out whether you are PCIe-bound
- Confirm the negotiated link. Check both current and maximum generation and width, because a card can idle at a lower generation and because a physically x16 slot may be electrically narrower.
nvidia-smi --query-gpu=pcie.link.gen.current,pcie.link.gen.max,pcie.link.width.current,pcie.link.width.max --format=csvreports all four. - Establish your own ceiling. Run a pinned H2D copy test at a size representative of your workload — the source investigation used 256 MB × 20 — and record the GB/s. This is the number everything else is compared against.
- Measure per-copy time inside the workload. If your runtime exposes per-copy timing, enable it. If not, bracket the transfers with CUDA events or the equivalent in your framework.
- Compare implied GB/s to the ceiling. If per-copy throughput is already at or near the ceiling, the link is the limit and software changes will not help. If it is well below, the problem is upstream — copy size, synchronization, or a stall waiting on data.
- Check whether transfers are actually on the critical path. If prefetch is working, transfer time may be hidden behind compute. If stall time is high, it is not.
What actually helps, and what does not
When the link is the wall, the effective options are to move fewer bytes or to move them less often.
- Reduce bytes moved. Quantization directly reduces the volume streamed per forward pass.
- Reduce transfer frequency. Keeping more of the model resident, or restructuring which components are offloaded, removes transfers from the per-token path entirely.
- Overlap transfer with compute. Prefetch can hide transfer cost when there is enough compute to hide it behind. The measured host reported prefetch stall versus slack precisely to show when this was not happening.
- Use pinned memory. The gap between pinned and pageable in the measured host was roughly 1–1.7 GB/s.
- Widen the link. Moving from x4 to x8, or from Gen3 to Gen4, changes the ceiling proportionally. This is a hardware decision, and on many consumer boards the electrical width is fixed.
What did not help in the documented case was consolidating multiple small copies into fewer larger ones. That is worth stating plainly because it is a common first instinct, and on a link that is already saturated it buys nothing.
Limitations and what remains uncertain
The capacity table is calculated, not benchmarked, and the measured figures come from a single loaded host on a PCIe 3.0-only platform. There is no universal threshold at which a link becomes “too slow” — it depends on how many bytes your particular offload strategy moves per token. Before buying hardware to fix a suspected PCIe bottleneck, measure the ceiling and the per-copy throughput on your own machine; if the copies are already at the ceiling, a wider link will help, and if they are not, it will not.
If you have measured H2D throughput on your own setup — particularly on x1 risers, Thunderbolt enclosures, or electrically narrow slots — sharing the link width, generation, copy size, and achieved GB/s in the community is more useful than a general impression of whether offloading “feels slow.”
FAQ
Does PCIe generation matter more than lane width?
Neither dominates; they multiply. PCIe 3.0 x16 and PCIe 4.0 x8 both work out to about 15.75 GB/s per direction in the encoding-capacity table, so a Gen4 card in an x8 slot can perform like a Gen3 card in an x16 slot.
Can I run a model that does not fit in VRAM on a slow PCIe link?
Yes, but throughput will be bound by the link rather than the GPU whenever offloaded data crosses the fabric per token. The practical question is how many bytes your offload strategy moves per forward pass, not whether the link is fast in the abstract.
Does combining several small copies into one large copy improve offload throughput?
Not necessarily. In the documented PCIe 3.0 x8 host, packing a layer's four home tensors into one contiguous pinned arena per dtype reduced four H2D copies to two, was bit-identical, and did not improve throughput, because each copy already saturated the link.
Why does my measured transfer rate fall short of the spec table?
The table is calculated from signalling rates and encoding alone. Real transfers carry transaction-layer and software overhead, and contend with other activity on the host. The measured PCIe 3.0 x8 host reached about 78% of its encoding capacity with pinned memory.
Is Thunderbolt 4 equivalent to a PCIe 3.0 x4 slot?
The headline scales look similar, but Thunderbolt is a tunnelled, shared transport rather than a native lane count. It specifies a 32 Gbit/s PCIe data requirement, about 4 GB/s before tunnelling overhead, and Intel cites storage throughput up to 3,000 MB/s.