Many hands, one job

Until the mid-2000s, each new chip generation ran faster mostly because its clock did. When Dennard scaling ended, raising the clock further made chips too hot to cool, and extracting more instruction-level parallelism from one thread hit diminishing returns. Since then, performance growth has come mainly from doing more work in parallel: more data per instruction, more threads per core, more cores per chip and more chips per system.

A SIMD instruction operates on a whole register of elements at once. One AVX instruction adds eight 32-bit floats; one AVX-512 instruction adds sixteen. Because only one instruction is fetched and decoded for many operations, SIMD is very energy efficient for regular, data-parallel work such as image processing, audio, linear algebra and machine learning.

SIMD comes in two styles. Packed SIMD (SSE, AVX, Arm NEON) fixes the register width in the instruction set, so each wider generation needs new instructions. Vector architectures, from the Cray-1 to today's Arm SVE and RISC-V V, let the hardware choose the vector length and handle loop remainders with a vector-length register or masks, so one binary runs on any width. Branches inside a SIMD loop are handled with masks: every lane computes, and a mask decides which results are kept.

A GPU runs thousands of lightweight threads. Hardware groups them, 32 at a time on NVIDIA GPUs, into a warp that executes one instruction across all its threads: SIMD hardware presented to the programmer as independent threads, a model NVIDIA calls SIMT (single instruction, multiple threads). When threads in a warp take different branches, the warp runs each path in turn with the inactive threads masked off, so divergent code loses efficiency.

GPUs hide memory latency by switching among many warps rather than by large caches and out-of-order execution. They suit problems with massive, regular parallelism, which is why they dominate machine-learning training and most of the world's fastest supercomputers.

Olukotun and colleagues argued in 1996 that several simple cores on one chip would beat one ever more complex core, and by the mid-2000s every major vendor had followed. A typical multicore processor has:

Key design choices include how many cores to build and how large to make each one, how much cache to share, the interconnect topology, and whether to split the design into chiplets: several smaller dies in one package, which improves manufacturing yield at the cost of slower links between dies.

When two cores cache the same memory block, a write by one must not leave the other reading a stale copy. A coherence protocol guarantees that every core sees writes to the same location in a single order, and eventually sees the latest value. Most protocols are write-invalidate: before a core writes, every other copy is invalidated.

Small systems use snooping: every cache watches a shared bus or broadcast network for other cores' requests. Large systems use a directory (Censier and Feautrier) that records which caches hold each block, so requests go only to those caches instead of to everyone.

Things to try. "Read sharing" needs one bus request per core and then runs without traffic. In "Read, then write", core 0's second write is free because Exclusive upgrades silently. "Two cores writing in turn" generates a bus transaction and a transfer of the whole block on every single write, even if the cores write different variables that happen to share a block.

Coherence works on whole blocks, not variables. If two threads update different variables that sit in the same 64-byte block, the block bounces between their caches exactly as in the ping-pong example above, even though the threads never share data. Torrellas, Lam and Hennessy measured how often this happens and how block size affects it. Padding each thread's data to its own block removes the problem:

Coherence is about one location. Consistency is about the order in which a core's accesses to different locations become visible to other cores. Lamport defined sequential consistency: the result is as if all operations ran in some single interleaving that respects each program's order. Real hardware relaxes this for speed, mainly by letting stores wait in a store buffer while later loads go ahead.

Amdahl's law assumes a fixed problem: if a fraction f of the work runs in parallel, the serial remainder caps the speedup at 1 / (1 − f), however many processors are added. With f = 0.95, 1,024 cores give only about 20×. Measuring this way is called strong scaling.

Gustafson's law assumes the problem grows with the machine, as with a bigger simulation mesh on a bigger computer, so the parallel part grows while the serial part does not. Measuring this way is called weak scaling, and it explains why supercomputers with millions of cores are useful. Neither law is wrong; they answer different questions.

Parallel efficiency is speedup divided by processor count. The Karp–Flatt metric works backward from a measured speedup S on p processors to the experimentally determined serial fraction e = (1/S − 1/p) / (1 − 1/p). If e grows as p grows, overhead rather than inherent serial work is limiting scaling.

Hill and Marty applied Amdahl's law to a fixed chip budget of n "base core equivalents" (BCEs). Combining r BCEs makes one bigger core with performance √r, following Pollack's rule that performance grows roughly with the square root of core area. Three organizations follow:

• Symmetric: n/r identical cores of size r. Big cores speed up serial code but leave fewer cores for parallel code.
• Asymmetric: one big core of size r plus n − r small cores. The big core runs serial code; all cores share the parallel part.
• Dynamic: resources that act as one big core for serial code and as n small cores for parallel code. An idealized upper bound.

Things to try. With the defaults, the best symmetric chip uses cores of about 8 BCEs and reaches about 51×; the best asymmetric chip pairs one 64-BCE core with 192 small ones for about 125×. Lower f to 0.9 and the best designs shift to much bigger cores; raise it to 0.999 and many small cores win. That is the case for heterogeneous chips: a few big cores for serial code, many small ones for parallel throughput.