12 modules · 5 parts · ~200 min
ML Hardware & Low-Latency Systems
Twelve modules, written for someone who already knows the basics. Starts from roofline and GPU throughput instincts, then inverts them for the batch-of-one regime: tail latency, CPU microarchitecture, PCIe and the wire, fixed-point numerics, and the workloads that answer in nanoseconds.
Part AFoundations
Territory you already own.
- 01Memory hierarchy, roofline, arithmetic intensityWhat the roofline actually bounds, why the ridge point keeps running away from us, and the exact point at which the model stops being useful.
- 02Layout, coalescing, bank conflicts, tilingCoalescing is a transaction count, not a thread count. Bank conflicts, padding versus swizzling, and the three-way tension inside every tile size.
- 03Pipelining, multi-buffering, latency hidingInitiation interval versus depth, prefetch distance as a ratio, and the moment every one of these tools stops working.
Part BThe pivot
Throughput instincts, inverted.
- 04Latency versus throughput as a design axisWhere batching, occupancy and amortization stop being virtues. Why high utilization is a defect, and why a barrier samples the tail.
- 05Tail latency and determinismWhy the mean describes no request that ever happened, where jitter actually comes from, and the measurement error that makes your p99 a fiction.
Part CSubstrate
The CPU and the wire.
- 06CPU microarchitectureFalse sharing, dependent chains, store buffers and memory ordering, the AVX-512 downclock, and the cache-line layout that makes an SPSC ring fast.
- 07Off-chip data movementPosted versus non-posted, why the round trip and not the bandwidth is what kills you, and the rigorous answer to “why not just use a GPU?”
Part DNumerics
Below the framework quantizer.
- 08Fixed-point and roundingQ-format as a contract, why truncation bias grows linearly while random error grows as a square root, and how fixed point buys reproducibility for free.
- 09Nonlinearities without dividersLUT, piecewise-linear, minimax polynomial and CORDIC — and why range reduction is the step that makes any of them work.
Part EWorkloads
What actually runs at nanosecond scale.
- 10Small MLPs at nanosecond scaleWhen you stop executing the network and start being it. Constant-weight multipliers, adder-tree depth, and the resource budget that co-designs the model.
- 11GBDT inference in hardwareRouting, not arithmetic. Why a depth-6 tree costs three levels rather than six, why feature fetch is the real bottleneck, and why trees beat DNNs at this scale.
- 12Hardware/software partitioning and market microstructureOrder books, the tick-to-trade budget allocated stage by stage, the fast-path/slow-path contract, and risk checks that must not trust the strategy.