Off-chip data movement

Two independent reasons that compound.

Protocol. A write is a posted transaction: the CPU hands it to the root complex and retires it. A read is non-posted — it generates a request, the device must respond with a completion, and that is a full traversal of the fabric in both directions, 1–2 µs on PCIe Gen5.

Microarchitecture. Device memory is mapped uncacheable, so the load cannot be speculated, cannot be reordered around, and cannot be covered by the out-of-order window. The core genuinely stalls for the whole round trip, retiring nothing. A 2 µs MMIO read on a 3 GHz core throws away about 6000 cycles of issue capacity.

The design consequences are concrete: never poll a device register in a loop; have the device DMA a status word into host memory and poll that (a cached read of ~1 ns when hot). Batch state into one DMA rather than several reads. And where you must write, write — doorbells, not handshakes. This is why descriptor rings look the way they do across every high-performance device: they are structured so that the fast path contains only posted writes and cacheable host-memory reads.

You removed a mean cost and the tail was never in the stack. The stack contributed a fairly consistent 5–15 µs, so removing it shifts the whole distribution left without narrowing it. What remains is Module 5’s territory, and the candidates are ranked by how often they turn out to be the answer:

1. C-states. If your poll loop ever yields, sleeps, or hits a pause path that lets the core drop into C6, you pay 50–100 µs to come back. Pin the frequency, disable deep C-states, and verify with residency counters rather than configuration files.
2. Interrupts still arriving. Bypass does not automatically stop the NIC from raising interrupts on that core, and /proc/interrupts will tell you in seconds. Move IRQ affinity off the polling core entirely.
3. NUMA misplacement. Buffer, thread and device on different nodes. Each is independently wrong-able, so check all three.
4. The other side. Exchange microbursts queue in the switch, and no amount of host tuning addresses that. NIC hardware timestamps versus your application timestamps will separate “the packet arrived late” from “we handled it late” — and that distinction is the single most useful measurement in this whole area.
5. Your own cache misses. The poll loop is now the entire program; if its working set does not stay resident, you have simply moved where the misses happen.

CPU with kernel bypass, roughly:

An FPGA sitting in the datapath does parse, book, features, model, risk and encode as the bits stream through, and never crosses PCIe at all. Total 350–900 ns.

So the two orders of magnitude decompose into three things, and none of them is arithmetic: the two PCIe crossings (~1.3–2 µs), the software stack and poll latency, and the fact that the FPGA overlaps parsing with reception instead of waiting for a complete packet before starting.

One honesty note that separates people who have built these from people who have read about them: quoted “sub-100 ns tick-to-trade” figures are almost always a trigger path — a precomputed order released by a single comparator, with book and feature work done in parallel or in advance. That is a real and legitimate technique, but it is not the same measurement as the full path, and conflating them makes budgets impossible to reason about.

The steelman deserves stating, because the lazy dismissal is wrong in an instructive way.

The strong version of the GPU case: kernel launch overhead can be eliminated with a persistent kernel — a block that never exits, spinning on a flag in mapped memory. GPUDirect RDMA lets the NIC DMA straight into GPU memory, removing the host hop. So you can get to: packet lands in GPU memory (~1 µs over PCIe), a spinning SM notices, does the math in nanoseconds, writes a response. No launch, no host. This is a real technique used in HPC, and anyone who says “kernel launch is 3–5 µs, therefore no” has not engaged with it.

Why it still loses, in the order that actually decides it:

1. The bus is still there. GPUDirect removes the host bounce, not the PCIe crossing. You pay ~1 µs in and ~1 µs out, which already exceeds a full FPGA path that never crosses a bus.
2. Determinism. This is the decisive one. The GPU’s clocks vary with temperature and power; the memory system is shared with whatever else is resident; there is no priority mechanism that guarantees your spinning block runs unmolested; the driver may intervene. You cannot state a worst case, and Module 5’s whole argument is that a system without a stated worst case has no deadline.
3. You are using one SM of 132. At batch 1 there is nothing to parallelise. Every architectural feature you are paying for — the warp scheduler, the huge register file, the memory hierarchy built for thousands of concurrent threads — is dead weight. A GPU is a machine for hiding latency with parallelism, and you have no parallelism.
4. Power and cost for a job a small FPGA does in a fraction of the envelope.

The clean statement: a GPU is a throughput engine, and this is a latency problem at batch 1. The architecture is not slow, it is aimed at a different quantity. The right answer is compute placed in the datapath — the packet is parsed and the decision made as bits arrive, and the bus crossing never appears in the equation at all.

Because +50 ns is the idle number, and three things degrade it under load.

Bandwidth. The inter-socket link is narrower than local memory, so under pressure you queue on it — Module 4’s $1/(1-\rho)$ applied to a link you did not know you were saturating.

Outstanding-request credits. Remote transactions consume limited buffering in the interconnect, so effective memory-level parallelism to a remote node is lower than to local memory. You lose concurrency as well as latency, which Module 1 says is the more expensive loss.

Multiplicity. Thread placement, memory placement and interrupt affinity are three independent decisions, and each can be wrong on its own. A buffer allocated by a thread that later migrates, an IRQ steered by irqbalance, and a NIC attached to the other socket’s root complex compose into something much worse than any single +50 ns.

And it is silent. Nothing fails; the number is just worse than it should be, forever, and it does not show up as an obvious counter. The remedies are all “decide once, explicitly”: numactl for memory and thread, IRQ affinity pinned by hand, and buffers allocated on the node attached to the device that will DMA into them.