Latency is "how long?" Throughput is "how many?"
If you feed a number into a multiplier in cycle 10, and the answer pops out in cycle 16, its latency is six cycles. But how often can it take a new number? If it's fully pipelined, its throughput might be one new pair every single cycle, even if each individual calculation takes six cycles to bake.
Lanes just copy-paste the pipeline sideways
In our architecture, Doolittle packs 16 transformed coefficients into a single memory word. So, we build 16 modular multiplier pipelines side-by-side (lanes) to chew through the whole word at once. Lane 0 handles the first coefficient, lane 1 handles the second, and so on. They all step forward together.
Worked example
Latency 8, throughput 16
Imagine a 16-lane pipeline that takes 8 cycles from start to finish. Once you fill it up, it's spitting out 16 finished results every single clock cycle. The fact that each item spent 8 cycles inside doesn't hurt your throughput at all.
Worked example
Don't multiply latency by rows
If we have a 4096-coefficient polynomial, and we process 16 at a time, that's 256 rows (4096 / 16). In a perfect world without stalls, we can feed all 256 rows in just 256 clock cycles. The pipeline latency delays the very first result, but it does NOT multiply the total time! You just add the latency to the end to find out when the last item finishes.
Play with the pipeline yourself
This interactive notebook simulates how items move through a pipeline. Try adding stalls once the pipeline is full, mess with the latency, and watch how the hardware handles the traffic jams while keeping everything in order.
Change downstream stalls and latency while watching offers, transfers, accepted order, and completion order.
Runs entirely in this browser. Python executes in Pyodide WebAssembly with no remote kernel. The construction code stays visible while reactive dependents recompute whenever you change an input.
Check your understanding
You build a pipeline with a 10-cycle latency, but it can accept a new input every cycle. What happens once it's full?
Section summary
- Latency is the delay for a single item. Throughput is the sustained rate.
- Pipelines let us work on multiple items at different stages of completion.
- Lanes give us spatial parallelism to match wide memory buses.
Repository layer · second pass
How do lanes and pipeline stages change throughput without changing results?
Parallel lanes process independent coefficients or limbs simultaneously. Pipeline registers divide one long operation into shorter timed stages. Lane count changes spatial replication; pipeline depth changes latency and achievable clock rate. Initiation interval determines how often new work may enter.
A pipeline can have latency 20 and initiation interval 1: the first result arrives late, then one may arrive each cycle. Resource conflicts, feedback, or insufficient buffering can increase initiation interval even when stages exist.
Reasoning chain
- 1
Measure work items per polynomial.
- 2
Choose independent partition axis.
- 3
Budget resources per lane.
- 4
Place stages by timing paths.
- 5
Calculate latency and initiation interval separately.
- 6
Size buffers for stalls and burst mismatch.
Worked trace
Fill, flow, drain
- Four-stage pipeline accepts item A at cycle 0.
- B,C,D enter on cycles 1,2,3.
- A exits after four edges.
- Then B,C,D follow one per cycle.
Result. Latency is four cycles; steady-state throughput is one item per cycle.
Executable lens · Python
Make the hidden state visible
def completion_cycles(items, latency, initiation_interval=1):
return [i*initiation_interval + latency for i in range(items)]
assert completion_cycles(4,4) == [4,5,6,7]Retype this example, predict each intermediate value, and then change one input that touches a boundary.
Misconception clinic
Tempting mistakes
- Dividing total cycles by latency to estimate throughput.
- Replicating arithmetic without providing enough memory bandwidth.
Retrieval and transfer
Close the book first
- Compute cycles for N items over L lanes.
- Find buffer depth for a five-cycle downstream stall.
- List axes: coefficient, limb, component, batch.