Small MLPs at nanosecond scale

DSPs break first, by a wide margin: $128 \times 128 = 16{,}384$ multipliers against 2000–4000 on a mid-size part. Then routing congestion, because 16,384 multipliers each need operands delivered. Then timing closure on the adder trees, where 128-input trees have significant fanout.

Fix in this order, because the ordering reflects cost:

1. Quantise harder. At int8 you get two MACs per DSP; at int4 the multipliers become small enough to build in LUT fabric, which is far more plentiful than DSPs. Combined with constant weights and CSD encoding, many multiplies disappear into a handful of adders.
2. Exploit weight structure. Zero weights cost nothing at all. Structured pruning to, say, 50% sparsity halves the array literally, not statistically. Powers of two are free.
3. Shrink the layer. This is usually the highest-value move and the one engineers resist, because it feels like the model is someone else’s decision. It is not: the model architecture and the device are one design. Going from 128 to 96 units removes 44% of the multipliers and costs a small amount of accuracy that a slightly better feature set usually recovers.
4. Fold. Reuse the array 2–4× and accept the latency multiple. This is the last resort, because folding reintroduces sequencing, control, and intermediate storage — you have started rebuilding a small accelerator with a memory hierarchy, at which point the whole premise deserves a second look.

Because they measure different resources against different constraints. MAC count is area, which you can trade in several directions: fold, quantise, prune, narrow. Tree depth is latency, it is $\lceil \log_2 N \rceil$, and it is much harder to move.

There are only two honest ways to reduce it:

• Fewer terms. A narrower layer or genuine sparsity. Going from 256 to 64 inputs takes the tree from 8 levels to 6 — note that this is a logarithmic return, so halving the layer buys exactly one level. That is why depth reduction by shrinking is a game of diminishing returns and why people overestimate what pruning does for latency.
• Faster levels. Carry-save arithmetic is the standard technique and worth knowing by name: a 3:2 compressor tree takes three numbers to two without propagating carries, so intermediate additions cost one full-adder delay regardless of width, and you pay a single carry-propagate adder at the very end. This removes the $O(W)$ carry chain from every internal node of the tree, which is typically a larger win than anything you can do to the tree’s shape.

The framing that generalises: in a latency design, area is a currency and depth is the price. You should be willing to spend a lot of the former to move the latter, and you should know exactly how many levels each spend actually buys.

As wire count, and as logic levels people forget to budget.

Accumulator width is literal. int8 × int8 gives 16-bit products; a 128-term sum needs $16 + \lceil \log_2 128 \rceil = 23$ bits. Choosing 24-bit accumulators instead of 32-bit removes a third of the adder-tree area, and since adder area is linear in width (Module 8) that saving is directly proportional. Conversely, guessing 16 bits because “it’s int8” overflows on essentially every input.

Requantisation is not free. Between layers you shift, round to nearest even, and saturate. That is a shifter, a rounding increment with its own carry, and a pair of comparators with muxes — about 2–3 logic levels per layer. In a three-layer network that is 6–9 levels, comparable to a whole adder tree, and it routinely goes unaccounted for until timing fails.

Saturation must be real. Module 8’s argument that wraparound converts a small overflow into a sign flip is not theoretical here: with market data, the tail events that overflow are exactly the ones the model exists to react to. A wrapped accumulator does not degrade, it inverts.

So the fixed-point design is not a numerical exercise performed before the hardware design. It is part of the hardware design, and the four numbers per layer — input format, weight format, accumulator width, output format — are structural parameters with area and depth consequences.

There is, and the distinction matters because the slogan from Part B overshoots if taken literally.

You absolutely pipeline for throughput across successive independent inputs. Ticks keep arriving; $II = 1$ with depth 30 means you accept a new input every cycle and each answer emerges 30 cycles later. Both properties hold simultaneously and neither costs the other.

What is forbidden is trading depth for $II$ or for clock frequency, which is the trade that every throughput technique offers and that FPGA tooling actively encourages, because $F_{\max}$ is what appears on the timing report. Adding stages to raise the clock, batching inputs to amortise a setup, deepening to simplify a critical path — all normally free, all now charged directly against the deadline.

So the discipline is: pipeline to exactly the depth that timing closure requires, and not one stage further. Then check the product $\text{depth} \times T_{\text{clk}}$ rather than either factor, and be willing to remove pipeline stages and accept a lower clock if the product improves. That is a genuinely unusual instinct, and it is the main thing that distinguishes latency-oriented FPGA work from ordinary FPGA work.

Three boundaries, and recognising them early saves a lot of wasted effort.

Size. Once the model exceeds the fabric you must fold, and folding reintroduces control logic, sequencing and intermediate storage — you have built a small accelerator with a memory hierarchy, and you now inherit every problem Modules 1 through 3 describe. When that happens, the right question is usually whether the model needs to be that large for the decision quality you actually require, not how to fold it more cleverly.

Weight update frequency. Constant weights are the source of most of the area savings, and they are compiled in. Changing them means a rebuild: full FPGA reconfiguration is 100 ms to seconds, partial reconfiguration 1–10 ms. If weights change daily that is fine; if they change intraday you must make them runtime-loadable registers, which restores full multipliers and forfeits the CSD, zero-weight and power-of-two savings entirely. That can be a 5–10× area difference, so it is a first-order architectural decision rather than a detail.

Precision. If the task genuinely needs floating point, the area advantage largely evaporates. In practice this is rare for small models — Module 8’s analysis usually shows int8 or int16 is sufficient — but it is worth checking rather than assuming.