← tonia.ai
Computer Architecture

Building a chip of many cores

What sits on a multicore processor besides the cores, why adding more cores gets harder with every generation, and how designers balance core count, caches, interconnect, memory bandwidth, power and manufacturing cost. Three calculators put numbers on coherence storage, memory bandwidth and chiplet economics.

Part A

Architecture and components

Building blocks

The cores are only part of a multicore processor. Everything else, often called the uncore, connects the cores to each other, to memory and to the outside world, and on server chips it can take as much area and power as the cores themselves.

CoresEach with its own pipeline, branch predictor and execution units, often two hardware threads (SMT). Some chips mix large performance cores with small efficiency cores.
Private cachesL1 instruction and data caches per core, and usually a private L2. Some designs share an L2 among a small cluster of cores.
Shared last-level cacheSplit into slices, typically one per core or cluster; addresses are hashed across slices to spread traffic.
Coherence hardwareSnoop filters or directories that track which private caches hold each block, so requests go only where needed.
On-chip networkRouters and links (a ring, mesh or crossbar) that carry requests, data and coherence messages.
Memory controllersSchedule DRAM accesses across several channels; their number sets the chip's memory bandwidth.
I/O and acceleratorsPCIe and CXL lanes, chip-to-chip links for multi-socket systems, and on some chips GPUs, neural engines or media blocks.
Power managementA controller that sets each core's voltage and frequency and powers down idle parts of the chip.

Ways to organize a chip

Mesh of tiles corecorecorecore corecorecorecore corecorecorecore corecorecorecore MCMC Each tile: core, L2, L3 slice and a router Chiplets 8 cores+ L3 8 cores+ L3 8 cores+ L3 8 cores+ L3 I/O diefabric, memorycontrollers, PCIe DRAM PCIe Compute dies linked to a central I/O die
OrganizationHow it worksTypical use
Monolithic with a ringCores and cache slices sit on a ring bus on one dieClient processors with up to about a dozen cores
Monolithic with a meshIdentical tiles, each a core plus cache slice and router, in a 2D gridServer processors with dozens of cores
ChipletsSeveral smaller compute dies around an I/O die, or tiles joined by short bridges, all in one packageLarge server and desktop processors
Clustered heterogeneous SoCClusters of performance cores and efficiency cores, each sharing an L2, plus GPU and accelerators sharing memoryPhones, tablets and laptops

Real processors

ProcessorCoresOrganizationLast-level cacheMemory
AMD EPYC 9004 (Zen 4)Up to 9612 compute chiplets of 8 cores around one I/O die32 MB L3 per chiplet12 DDR5 channels
Intel Xeon, 4th generationUp to 60Four tiles joined by silicon bridges into one mesh1.875 MB L3 per core, shared8 DDR5 channels
NVIDIA Grace72 Arm Neoverse V2Monolithic die with a mesh fabric117 MB L3, sharedLPDDR5X
Apple M-seriesMix of performance and efficiency coresClusters sharing an L2, plus GPU and neural engineSystem-level cache shared with the GPUUnified LPDDR memory

Figures as published by the vendors; configurations vary by product.

Power management

A multicore chip almost never runs every core flat out. A power-management controller adjusts each core's voltage and frequency (DVFS), shuts down idle cores completely (power gating), and lets a few cores run above their base frequency when the others are idle and there is thermal headroom ("turbo" or "boost"). This is why a processor's single-core frequency is much higher than its all-core frequency: the power budget is shared.

Part B

Scalability challenges

Coherence at scale

Cache coherence is easy with a few cores on a shared bus: every cache snoops every request. Snooping does not scale, because every miss is broadcast to every core, so traffic and snoop work grow with the core count.

Directory protocols, pioneered in machines such as Stanford DASH and SGI Origin, record which caches hold each block, so a request is sent only to the caches involved. The price is storage. A full bit vector needs one bit per core for every block, which grows linearly with the core count. Practical designs shrink it:

  • Limited pointers store a few core IDs per block and fall back to broadcast when more cores share it.
  • Coarse vectors use one bit per group of cores.
  • Sparse directories and snoop filters track only the blocks actually present in private caches, not all of memory.
  • Hierarchical coherence keeps traffic within a chiplet or cluster whenever possible.

Martin, Hill and Sorin showed that with these techniques, the storage and traffic costs grow slowly enough that hardware coherence remains practical even for very large core counts.

Directory calculator

On-chip networks

Dally and Towles argued in 2001 that large chips should "route packets, not wires": a shared network of routers and links is more efficient than dedicated wires between every pair of units. As core counts grew, designs moved from buses to rings and then to meshes. Kumar, Zyuban and Tullsen showed that the interconnect's area, power and latency become first-order costs at this scale.

With a mesh or chiplets, the processor itself becomes non-uniform. A core reaches a cache slice or memory controller on the far side of the chip, or on another chiplet, more slowly than one nearby. Core-to-core latency can vary several-fold within one chip. Servers expose this to software as several NUMA nodes per socket (sub-NUMA clustering), so the operating system can keep threads near their data.

TopologyAverage hops (p nodes)Bisection bandwidthMain cost
RingAbout p/4Constant: 2 linksLatency and congestion grow quickly beyond about a dozen stops
2D meshAbout 2√p/3Grows with √pRouter area and power at every tile
Crossbar1Grows with pWiring grows with p²
Chiplet linksAdds a die-to-die hopLimited by package wiringExtra latency and energy per bit crossing dies

The bandwidth wall

Core counts have grown faster than the number of memory channels, because every channel needs hundreds of package pins and board traces. A 96-core processor with 12 channels of DDR5-4800 has 12 × 4,800 MT/s × 8 bytes = 460.8 GB/s of memory bandwidth, or about 4.8 GB/s per core, less than a single modern core can consume on memory-bound code. Rogers and colleagues called this the bandwidth wall: beyond a certain core count, extra cores add no throughput for memory-bound work unless caches capture more of the traffic.

Designers respond with larger and stacked last-level caches, more and faster channels, and high-bandwidth memory (HBM) in the package. CXL adds memory capacity and bandwidth over PCIe links.

Bandwidth calculator

See where adding cores stops helping a memory-bound workload.

IdealBandwidth-limited

Synchronization

Locks and shared counters are where cores meet, and coherence makes contention expensive. When many cores spin on the same lock variable, every release invalidates every waiting copy and all the waiters rush to re-read it, flooding the interconnect. Mellor-Crummey and Scott's queue-based locks avoid this by having each waiter spin on its own cache line and passing the lock directly to the next in line, so traffic stays constant as cores are added.

Other approaches reduce sharing altogether: per-core data that is combined only occasionally, read-mostly structures, and lock-free algorithms built on atomic instructions. Hardware transactional memory, proposed by Herlihy and Moss, lets a block of code run speculatively and commit only if no other core touched its data; IBM and Intel have shipped versions of it, with mixed success.

Power, heat and software

Power and heat. Each core can be small, but the chip's total power is capped by cooling. With more cores than the budget can run at full speed, some must run slower or stay off, which is the dark silicon problem.

Software. Hardware parallelism is useful only if programs use it. Amdahl's law caps the speedup of code with any serial part, and operating systems, runtimes and libraries must themselves scale without global locks. Keeping data on the NUMA node that uses it, avoiding false sharing and limiting synchronization matter as much as the core count.

Part C

Design trade-offs

Design decisions

DecisionOption AOption BWhat decides it
Core size and countFewer, larger cores: fast serial codeMore, smaller cores: more throughput per wattHow much of the workload is parallel (see the multicore design calculator)
Core typesHomogeneous: simple schedulingHeterogeneous: big cores for serial code, small for background and parallel workWorkload mix; energy limits; operating system support
SMTOff: full core resources per thread, better isolationOn: higher throughput when threads stall oftenWorkload latency vs. throughput; security isolation needs
Cache sharingPrivate L2 and L3: low latency, no interferenceShared L3: more effective capacity, data sharing between coresHow much data threads share; tolerance for interference
CoherenceSnooping: simple, fast for few coresDirectory or snoop filter: scales to many coresCore count; area for directory storage
InterconnectRing: cheap, low latency at small scaleMesh or chiplet fabric: scales to many coresCore count; floorplan; power
ConstructionMonolithic die: lowest latencyChiplets: better yield, reuse, mixed process nodesDie size; volume; product range
MemoryMore DDR channels: capacityHBM in package: bandwidthBandwidth vs. capacity needs; cost
Clock vs. coresHigh frequency, fewer coresLower frequency, more coresPower budget; parallel fraction

Cores or clock speed

Dynamic power is roughly proportional to C × V² × f, and supply voltage must rise roughly with frequency, so power grows close to the cube of frequency. Under the same power budget as one core at frequency f, two cores can each run at about 0.79f, because 2 × 0.79³ ≈ 1.

For perfectly parallel work, the two slower cores deliver 2 × 0.79 = 1.58× the throughput. For serial work, the result is 21% slower. Spreading work across more, slower cores has driven processor design since the mid-2000s, and it is also why chips keep a few fast cores for the serial parts.

Monolithic or chiplets

Manufacturing defects land randomly on a wafer, so the chance that a die has none falls exponentially with its area. A simple model gives yield Y = e−A × D, where A is die area and D is defect density. Splitting one large die into several small chiplets raises yield sharply, and failing chiplets are discarded individually instead of wasting a whole large die.

Chiplets bring other advantages. The same compute die can serve many products, from 8 to 96 cores. I/O, which barely benefits from new process nodes, can be built on an older, cheaper one. Naffziger and colleagues describe how AMD used both effects. The costs are extra area for die-to-die interfaces, extra latency and energy for every access that crosses dies, and more complex packaging, sometimes on silicon interposers or bridges (Kannan, Enright Jerger and Loh).

Chiplet cost calculator

Compare the silicon cost of one processor built as a single die or split into chiplets.

Model assumptions

300 mm wafers. Dies per wafer = π r² / A − π d / √(2A), which subtracts partial dies at the edge. Yield uses the Poisson model e−A × D; real fabs use more detailed models such as Murphy's. The die-to-die overhead applies only when the design is split. Costs cover silicon and packaging only, not test, I/O dies or design.

Things to try. With the defaults, four chiplets cost about two thirds as much as one large die. Lower the defect density to 0.03, as on a mature process, and the advantage shrinks; raise the area to 800 mm² and splitting becomes essential. Set the packaging cost high to see when chiplets stop paying.

Matching the design to the market

TargetFirst priorityTypical choices
PhoneEnergy per taskFew big cores plus several efficiency cores; shared memory with GPU and accelerators; aggressive power gating
Laptop and desktopFast single threads with good multithreaded throughputMonolithic ring or a few chiplets; mixed core types; high boost frequencies
Cloud serverThroughput per watt and per dollarMany cores; mesh or chiplets; many memory channels; SMT; NUMA-aware software
HPC and AIBandwidth and vector throughputWide SIMD or matrix units; HBM; tight coupling with GPUs
Go further

Resources

Interactive tools and references

Peer-reviewed reading

  • Hammond, L., Nayfeh, B. A., & Olukotun, K. (1997). A single-chip multiprocessor. IEEE Computer, 30(9), 79–85.The case for putting several cores on one die.doi.org/10.1109/2.612253
  • Agarwal, A., Simoni, R., Hennessy, J., & Horowitz, M. (1988). An evaluation of directory schemes for cache coherence. Proc. ISCA, 280–289.Full, limited and coarse directories compared.doi.org/10.1109/ISCA.1988.5238
  • Lenoski, D., et al. (1992). The Stanford Dash multiprocessor. IEEE Computer, 25(3), 63–79.An early scalable directory-based machine.doi.org/10.1109/2.121510
  • Laudon, J., & Lenoski, D. (1997). The SGI Origin: A ccNUMA highly scalable server. Proc. ISCA, 241–251.Directory coherence and NUMA in a commercial system.doi.org/10.1145/264107.264206
  • Martin, M. M. K., Hill, M. D., & Sorin, D. J. (2012). Why on-chip cache coherence is here to stay. Communications of the ACM, 55(7), 78–89.Why coherence costs grow slowly enough to keep.doi.org/10.1145/2209249.2209269
  • Dally, W. J., & Towles, B. (2001). Route packets, not wires: On-chip interconnection networks. Proc. Design Automation Conference, 684–689.The argument for networks on chip.doi.org/10.1109/DAC.2001.935594
  • Taylor, M. B., et al. (2002). The Raw microprocessor: A computational fabric for software circuits and general-purpose programs. IEEE Micro, 22(2), 25–35.An early tiled processor with a mesh network.doi.org/10.1109/MM.2002.997877
  • Bell, S., et al. (2008). TILE64 processor: A 64-core SoC with mesh interconnect. Proc. IEEE ISSCC, 88–89, 598.A commercial 64-core mesh chip.doi.org/10.1109/ISSCC.2008.4523070
  • Kumar, R., Zyuban, V., & Tullsen, D. M. (2005). Interconnections in multi-core architectures: Understanding mechanisms, overheads and scaling. Proc. ISCA, 408–419.The real cost of the on-chip network.doi.org/10.1109/ISCA.2005.34
  • Rogers, B. M., et al. (2009). Scaling the bandwidth wall: Challenges in and avenues for CMP scaling. Proc. ISCA, 371–382.How memory bandwidth limits core counts.doi.org/10.1145/1555754.1555801
  • Mellor-Crummey, J. M., & Scott, M. L. (1991). Algorithms for scalable synchronization on shared-memory multiprocessors. ACM Transactions on Computer Systems, 9(1), 21–65.Queue locks that keep coherence traffic constant.doi.org/10.1145/103727.103729
  • Herlihy, M., & Moss, J. E. B. (1993). Transactional memory: Architectural support for lock-free data structures. Proc. ISCA, 289–300.Hardware transactions as an alternative to locks.doi.org/10.1145/165123.165164
  • Kumar, R., et al. (2003). Single-ISA heterogeneous multi-core architectures: The potential for processor power reduction. Proc. MICRO-36, 81–92.The research behind mixed core types.doi.org/10.1109/MICRO.2003.1253185
  • Esmaeilzadeh, H., et al. (2011). Dark silicon and the end of multicore scaling. Proc. ISCA, 365–376.Why power limits how many cores can be active.doi.org/10.1145/2000064.2000108
  • Kannan, A., Enright Jerger, N., & Loh, G. H. (2015). Enabling interposer-based disintegration of multi-core processors. Proc. MICRO-48, 546–558.The cost and yield case for splitting chips apart.doi.org/10.1145/2830772.2830808
  • Naffziger, S., et al. (2021). Pioneering chiplet technology and design for the AMD EPYC and Ryzen processor families. Proc. ISCA, 57–70.Chiplets in volume production, from the designers.doi.org/10.1109/ISCA52012.2021.00014