Architecture and components
Building blocks
The cores are only part of a multicore processor. Everything else, often called the uncore, connects the cores to each other, to memory and to the outside world, and on server chips it can take as much area and power as the cores themselves.
Ways to organize a chip
| Organization | How it works | Typical use |
|---|---|---|
| Monolithic with a ring | Cores and cache slices sit on a ring bus on one die | Client processors with up to about a dozen cores |
| Monolithic with a mesh | Identical tiles, each a core plus cache slice and router, in a 2D grid | Server processors with dozens of cores |
| Chiplets | Several smaller compute dies around an I/O die, or tiles joined by short bridges, all in one package | Large server and desktop processors |
| Clustered heterogeneous SoC | Clusters of performance cores and efficiency cores, each sharing an L2, plus GPU and accelerators sharing memory | Phones, tablets and laptops |
Real processors
| Processor | Cores | Organization | Last-level cache | Memory |
|---|---|---|---|---|
| AMD EPYC 9004 (Zen 4) | Up to 96 | 12 compute chiplets of 8 cores around one I/O die | 32 MB L3 per chiplet | 12 DDR5 channels |
| Intel Xeon, 4th generation | Up to 60 | Four tiles joined by silicon bridges into one mesh | 1.875 MB L3 per core, shared | 8 DDR5 channels |
| NVIDIA Grace | 72 Arm Neoverse V2 | Monolithic die with a mesh fabric | 117 MB L3, shared | LPDDR5X |
| Apple M-series | Mix of performance and efficiency cores | Clusters sharing an L2, plus GPU and neural engine | System-level cache shared with the GPU | Unified LPDDR memory |
Figures as published by the vendors; configurations vary by product.
Power management
A multicore chip almost never runs every core flat out. A power-management controller adjusts each core's voltage and frequency (DVFS), shuts down idle cores completely (power gating), and lets a few cores run above their base frequency when the others are idle and there is thermal headroom ("turbo" or "boost"). This is why a processor's single-core frequency is much higher than its all-core frequency: the power budget is shared.
Scalability challenges
Coherence at scale
Cache coherence is easy with a few cores on a shared bus: every cache snoops every request. Snooping does not scale, because every miss is broadcast to every core, so traffic and snoop work grow with the core count.
Directory protocols, pioneered in machines such as Stanford DASH and SGI Origin, record which caches hold each block, so a request is sent only to the caches involved. The price is storage. A full bit vector needs one bit per core for every block, which grows linearly with the core count. Practical designs shrink it:
- Limited pointers store a few core IDs per block and fall back to broadcast when more cores share it.
- Coarse vectors use one bit per group of cores.
- Sparse directories and snoop filters track only the blocks actually present in private caches, not all of memory.
- Hierarchical coherence keeps traffic within a chiplet or cluster whenever possible.
Martin, Hill and Sorin showed that with these techniques, the storage and traffic costs grow slowly enough that hardware coherence remains practical even for very large core counts.
Directory calculator
On-chip networks
Dally and Towles argued in 2001 that large chips should "route packets, not wires": a shared network of routers and links is more efficient than dedicated wires between every pair of units. As core counts grew, designs moved from buses to rings and then to meshes. Kumar, Zyuban and Tullsen showed that the interconnect's area, power and latency become first-order costs at this scale.
With a mesh or chiplets, the processor itself becomes non-uniform. A core reaches a cache slice or memory controller on the far side of the chip, or on another chiplet, more slowly than one nearby. Core-to-core latency can vary several-fold within one chip. Servers expose this to software as several NUMA nodes per socket (sub-NUMA clustering), so the operating system can keep threads near their data.
| Topology | Average hops (p nodes) | Bisection bandwidth | Main cost |
|---|---|---|---|
| Ring | About p/4 | Constant: 2 links | Latency and congestion grow quickly beyond about a dozen stops |
| 2D mesh | About 2√p/3 | Grows with √p | Router area and power at every tile |
| Crossbar | 1 | Grows with p | Wiring grows with p² |
| Chiplet links | Adds a die-to-die hop | Limited by package wiring | Extra latency and energy per bit crossing dies |
The bandwidth wall
Core counts have grown faster than the number of memory channels, because every channel needs hundreds of package pins and board traces. A 96-core processor with 12 channels of DDR5-4800 has 12 × 4,800 MT/s × 8 bytes = 460.8 GB/s of memory bandwidth, or about 4.8 GB/s per core, less than a single modern core can consume on memory-bound code. Rogers and colleagues called this the bandwidth wall: beyond a certain core count, extra cores add no throughput for memory-bound work unless caches capture more of the traffic.
Designers respond with larger and stacked last-level caches, more and faster channels, and high-bandwidth memory (HBM) in the package. CXL adds memory capacity and bandwidth over PCIe links.
Bandwidth calculator
See where adding cores stops helping a memory-bound workload.
Synchronization
Locks and shared counters are where cores meet, and coherence makes contention expensive. When many cores spin on the same lock variable, every release invalidates every waiting copy and all the waiters rush to re-read it, flooding the interconnect. Mellor-Crummey and Scott's queue-based locks avoid this by having each waiter spin on its own cache line and passing the lock directly to the next in line, so traffic stays constant as cores are added.
Other approaches reduce sharing altogether: per-core data that is combined only occasionally, read-mostly structures, and lock-free algorithms built on atomic instructions. Hardware transactional memory, proposed by Herlihy and Moss, lets a block of code run speculatively and commit only if no other core touched its data; IBM and Intel have shipped versions of it, with mixed success.
Power, heat and software
Power and heat. Each core can be small, but the chip's total power is capped by cooling. With more cores than the budget can run at full speed, some must run slower or stay off, which is the dark silicon problem.
Software. Hardware parallelism is useful only if programs use it. Amdahl's law caps the speedup of code with any serial part, and operating systems, runtimes and libraries must themselves scale without global locks. Keeping data on the NUMA node that uses it, avoiding false sharing and limiting synchronization matter as much as the core count.
Design trade-offs
Design decisions
| Decision | Option A | Option B | What decides it |
|---|---|---|---|
| Core size and count | Fewer, larger cores: fast serial code | More, smaller cores: more throughput per watt | How much of the workload is parallel (see the multicore design calculator) |
| Core types | Homogeneous: simple scheduling | Heterogeneous: big cores for serial code, small for background and parallel work | Workload mix; energy limits; operating system support |
| SMT | Off: full core resources per thread, better isolation | On: higher throughput when threads stall often | Workload latency vs. throughput; security isolation needs |
| Cache sharing | Private L2 and L3: low latency, no interference | Shared L3: more effective capacity, data sharing between cores | How much data threads share; tolerance for interference |
| Coherence | Snooping: simple, fast for few cores | Directory or snoop filter: scales to many cores | Core count; area for directory storage |
| Interconnect | Ring: cheap, low latency at small scale | Mesh or chiplet fabric: scales to many cores | Core count; floorplan; power |
| Construction | Monolithic die: lowest latency | Chiplets: better yield, reuse, mixed process nodes | Die size; volume; product range |
| Memory | More DDR channels: capacity | HBM in package: bandwidth | Bandwidth vs. capacity needs; cost |
| Clock vs. cores | High frequency, fewer cores | Lower frequency, more cores | Power budget; parallel fraction |
Cores or clock speed
Dynamic power is roughly proportional to C × V² × f, and supply voltage must rise roughly with frequency, so power grows close to the cube of frequency. Under the same power budget as one core at frequency f, two cores can each run at about 0.79f, because 2 × 0.79³ ≈ 1.
For perfectly parallel work, the two slower cores deliver 2 × 0.79 = 1.58× the throughput. For serial work, the result is 21% slower. Spreading work across more, slower cores has driven processor design since the mid-2000s, and it is also why chips keep a few fast cores for the serial parts.
Monolithic or chiplets
Manufacturing defects land randomly on a wafer, so the chance that a die has none falls exponentially with its area. A simple model gives yield Y = e−A × D, where A is die area and D is defect density. Splitting one large die into several small chiplets raises yield sharply, and failing chiplets are discarded individually instead of wasting a whole large die.
Chiplets bring other advantages. The same compute die can serve many products, from 8 to 96 cores. I/O, which barely benefits from new process nodes, can be built on an older, cheaper one. Naffziger and colleagues describe how AMD used both effects. The costs are extra area for die-to-die interfaces, extra latency and energy for every access that crosses dies, and more complex packaging, sometimes on silicon interposers or bridges (Kannan, Enright Jerger and Loh).
Chiplet cost calculator
Compare the silicon cost of one processor built as a single die or split into chiplets.
Model assumptions
300 mm wafers. Dies per wafer = π r² / A − π d / √(2A), which subtracts partial dies at the edge. Yield uses the Poisson model e−A × D; real fabs use more detailed models such as Murphy's. The die-to-die overhead applies only when the design is split. Costs cover silicon and packaging only, not test, I/O dies or design.
Things to try. With the defaults, four chiplets cost about two thirds as much as one large die. Lower the defect density to 0.03, as on a mature process, and the advantage shrinks; raise the area to 800 mm² and splitting becomes essential. Set the packaging cost high to see when chiplets stop paying.
Matching the design to the market
| Target | First priority | Typical choices |
|---|---|---|
| Phone | Energy per task | Few big cores plus several efficiency cores; shared memory with GPU and accelerators; aggressive power gating |
| Laptop and desktop | Fast single threads with good multithreaded throughput | Monolithic ring or a few chiplets; mixed core types; high boost frequencies |
| Cloud server | Throughput per watt and per dollar | Many cores; mesh or chiplets; many memory channels; SMT; NUMA-aware software |
| HPC and AI | Bandwidth and vector throughput | Wide SIMD or matrix units; HBM; tight coupling with GPUs |
Resources
Interactive tools and references
Peer-reviewed reading
- Hammond, L., Nayfeh, B. A., & Olukotun, K. (1997). A single-chip multiprocessor. IEEE Computer, 30(9), 79–85.The case for putting several cores on one die.doi.org/10.1109/2.612253
- Agarwal, A., Simoni, R., Hennessy, J., & Horowitz, M. (1988). An evaluation of directory schemes for cache coherence. Proc. ISCA, 280–289.Full, limited and coarse directories compared.doi.org/10.1109/ISCA.1988.5238
- Lenoski, D., et al. (1992). The Stanford Dash multiprocessor. IEEE Computer, 25(3), 63–79.An early scalable directory-based machine.doi.org/10.1109/2.121510
- Laudon, J., & Lenoski, D. (1997). The SGI Origin: A ccNUMA highly scalable server. Proc. ISCA, 241–251.Directory coherence and NUMA in a commercial system.doi.org/10.1145/264107.264206
- Martin, M. M. K., Hill, M. D., & Sorin, D. J. (2012). Why on-chip cache coherence is here to stay. Communications of the ACM, 55(7), 78–89.Why coherence costs grow slowly enough to keep.doi.org/10.1145/2209249.2209269
- Dally, W. J., & Towles, B. (2001). Route packets, not wires: On-chip interconnection networks. Proc. Design Automation Conference, 684–689.The argument for networks on chip.doi.org/10.1109/DAC.2001.935594
- Taylor, M. B., et al. (2002). The Raw microprocessor: A computational fabric for software circuits and general-purpose programs. IEEE Micro, 22(2), 25–35.An early tiled processor with a mesh network.doi.org/10.1109/MM.2002.997877
- Bell, S., et al. (2008). TILE64 processor: A 64-core SoC with mesh interconnect. Proc. IEEE ISSCC, 88–89, 598.A commercial 64-core mesh chip.doi.org/10.1109/ISSCC.2008.4523070
- Kumar, R., Zyuban, V., & Tullsen, D. M. (2005). Interconnections in multi-core architectures: Understanding mechanisms, overheads and scaling. Proc. ISCA, 408–419.The real cost of the on-chip network.doi.org/10.1109/ISCA.2005.34
- Rogers, B. M., et al. (2009). Scaling the bandwidth wall: Challenges in and avenues for CMP scaling. Proc. ISCA, 371–382.How memory bandwidth limits core counts.doi.org/10.1145/1555754.1555801
- Mellor-Crummey, J. M., & Scott, M. L. (1991). Algorithms for scalable synchronization on shared-memory multiprocessors. ACM Transactions on Computer Systems, 9(1), 21–65.Queue locks that keep coherence traffic constant.doi.org/10.1145/103727.103729
- Herlihy, M., & Moss, J. E. B. (1993). Transactional memory: Architectural support for lock-free data structures. Proc. ISCA, 289–300.Hardware transactions as an alternative to locks.doi.org/10.1145/165123.165164
- Kumar, R., et al. (2003). Single-ISA heterogeneous multi-core architectures: The potential for processor power reduction. Proc. MICRO-36, 81–92.The research behind mixed core types.doi.org/10.1109/MICRO.2003.1253185
- Esmaeilzadeh, H., et al. (2011). Dark silicon and the end of multicore scaling. Proc. ISCA, 365–376.Why power limits how many cores can be active.doi.org/10.1145/2000064.2000108
- Kannan, A., Enright Jerger, N., & Loh, G. H. (2015). Enabling interposer-based disintegration of multi-core processors. Proc. MICRO-48, 546–558.The cost and yield case for splitting chips apart.doi.org/10.1145/2830772.2830808
- Naffziger, S., et al. (2021). Pioneering chiplet technology and design for the AMD EPYC and Ryzen processor families. Proc. ISCA, 57–70.Chiplets in volume production, from the designers.doi.org/10.1109/ISCA52012.2021.00014