CPU microarchitecture

Almost certainly a single accumulator, making the loop a dependent chain of FMAs. Each FMA must wait for the previous one’s result, so you run at one per 4-cycle latency instead of two per cycle — one eighth of peak, and a quarter is what you see once the compiler has partially unrolled.

The fix is $\text{latency} \times \text{throughput} = 4 \times 2 = 8$ independent accumulators, summed once at the end. This is Little’s Law applied to the execution engine, and the general form is worth carrying: the number of independent operations you need in flight is the product of an operation’s latency and its throughput. It is the same reasoning that produced “80 sectors per SM” in Module 1 and “8 to 16 warps is not enough” in Module 3.

Two caveats that matter in practice. Compilers will not do this for you in floating point without -ffast-math or an explicit #pragma omp simd reduction, because reassociating a float sum changes the result (Module 8) and the compiler is not permitted to change your answer. And if the data does not fit in cache, none of this helps — you are then bound by the 10–16 line-fill buffers, which cap a single core at 8–12 GB/s no matter how many accumulators you have.

Synchronisation first, and there is essentially none. The producer writes the payload, then publishes with a release store to the write index. The consumer reads the write index with an acquire load, then reads the payload. On x86-TSO both compile to plain mov: no fences, no locked instructions. The release ordering guarantees the payload writes are visible before the index update, which is the only invariant the algorithm needs.

The three things that make it slow anyway are all about layout and traffic:

1. Head and tail on the same cache line. The producer writes one, the consumer writes the other, and they ping-pong a shared line on every single operation. This alone can cost 100×. Separate them by 128 B (adjacent-line prefetch, above).
2. Reading the other side’s index every iteration. Even correctly padded, each read of the consumer’s index pulls a line the consumer keeps dirtying. Cache a local copy of the far index and only refresh it when the local view says the ring is full or empty — typically cutting coherence traffic by the batch size.
3. A non-power-of-two capacity. Wraparound then needs a modulo (20+ cycles) rather than a mask (1 cycle). Always size to a power of two.

Worth stating explicitly: none of this generalises. The absence of atomics is a property of single producer and single consumer. Add a second producer and you need a real RMW, and the whole cost structure changes.

Four mechanisms, and the first is the notorious one.

• Frequency licensing. On Skylake-SP-era Intel server parts, sustained heavy 512-bit operations (FP and integer multiply) drop the core into a lower frequency licence — up to roughly 40% below base — and the transition costs tens of microseconds. Worse, the licence applies to the whole core, so a brief burst of AVX-512 slows down all the scalar code around it. Ice Lake and later reduced this considerably, and AMD’s Zen 4/5 implementation avoids most of it, but on the wrong part it is decisive.
• You are memory-bound. Wider vectors do not move more bytes if you are already limited by bandwidth or by the 10–16 line-fill buffers. You keep the downclock and gain nothing.
• Register pressure and spills. Thirty-two 512-bit registers sound generous until the compiler starts spilling 64-byte values.
• Alignment and line splits. A 64-byte load spans an entire line, so any misalignment splits across two lines every time rather than occasionally.

Practical rule: use 256-bit for mixed or latency-sensitive code, and 512-bit only for long, compute-dense loops where you have measured both the throughput gain and the actual achieved frequency. And measure frequency, not just time — a change that looks neutral may be a real speedup masked by a downclock, or vice versa.

The lock has become the operation, by more than an order of magnitude.

Uncontended, lock-prefixed instructions cost about 20 cycles each, and a lock/unlock pair is two of them plus the surrounding code: call it 40–60 cycles, roughly 15–20 ns against a 5 ns payload. You have quadrupled the cost before any contention at all.

Contended is where it collapses. The lock line must transfer between caches — 100–300 cycles — and if the lock is held when you arrive, pthread_mutex will eventually make a futex syscall and park the thread. That is a context switch: 1–5 µs direct plus tens of microseconds of cache and TLB damage afterwards. Against a 5 ns operation you are now three to four orders of magnitude out, and the operation is serialised on top.

The correct shapes, in order of preference: give each thread its own data and never share; hand off through an SPSC ring (previous probe) so there is no mutual exclusion at all; use a seqlock if the data is read-mostly and readers can retry; and only then reach for a lock, keeping the critical section large enough that 40 cycles of overhead is amortised — which for a 5 ns operation means batching thousands of them.

Because you may be buying determinism rather than speed, and Part B established that those are different objectives.

A predictable branch costs about 1–2 cycles; a cmov or arithmetic mask costs a fixed 1–3 cycles and cannot be predicted away. So on predictable data, branchless loses — that is what your benchmark measured. On unpredictable data the branch costs 15–20 cycles roughly half the time, about 8–10 cycles on average, and branchless wins outright.

The decisive point for a latency path is the distribution rather than the mean. A branch gives you a bimodal 1-cycle/20-cycle result whose mixture depends on the input data; branchless gives a constant. If you are defending a p99, you will take a slower deterministic path over a faster bimodal one, because the tail is what the deadline is written against. And in a decision tree (Module 11) the branches are unpredictable by construction — a predictable split carries no information — so branchless is both faster and more deterministic there.

The general framing: benchmarks report means on whatever data you fed them. Ask what the branch predictor’s accuracy will be on production data, and whether you are optimising the mean or the tail, before believing the result.