Unified Memory on Apple Silicon: What Changes When CPU and GPU Share the Same Pool

The copy tax you didn't know you were paying

On traditional discrete-GPU systems, moving a texture from the CPU to the GPU means allocating a staging buffer, copying data over PCIe, then issuing a blit command to get it into VRAM. That round-trip adds latency, burns bandwidth, and forces you to synchronize two separate memory domains. Apple Silicon removes the domain boundary entirely: the CPU, GPU, Neural Engine, and media engines all address the same physical DRAM. No PCIe hop, no explicit copy, no second allocation. The catch is that the pool is finite and shared, so one subsystem's allocation can starve another.

How the unified pool actually works

Apple's Unified Memory Architecture (UMA) presents a single address space to all compute units. When you create a Metal resource with MTLResourceStorageModeShared, the driver maps the same physical pages into the CPU and GPU page tables. The GPU's memory controller can read and write those pages directly; the CPU sees them as ordinary system memory. Cache coherency is maintained by the fabric, so a write from a compute shader becomes visible to the CPU after the command buffer completes—no manual flush or invalidate required on Apple Silicon (M1 and later).

This design shines for workloads that ping-pong data between compute stages: machine-learning inference that preprocesses on the CPU, runs a model on the Neural Engine, then post-processes on the GPU; video pipelines that decode on the media engine, filter with a compute shader, and display via the render pipeline. Each stage touches the same buffer without a copy.

APIs that make UMA the default

You don't need a special flag to opt in. Metal's MTLResourceStorageModeShared (the default on macOS and iOS for Apple Silicon) gives you shared buffers and textures. Core ML automatically places model weights and intermediate tensors in the unified pool when you compile for the Neural Engine. Accelerate's vImage and BNNS routines operate on the same buffers you hand to Metal. The pattern is consistent: allocate once, pass the pointer or MTLBuffer/MTLTexture to whichever framework owns the next stage.

Worked example: render target → compute input without a blit

Suppose you render a scene to a texture, then run a compute shader that reads that texture to generate a histogram. On a discrete GPU you'd need a blit pass or a copy command. On Apple Silicon you simply:

// 1. Create a texture usable as both render target and shader read
let desc = MTLTextureDescriptor.texture2DDescriptor(
    pixelFormat: .rgba8Unorm,
    width: 1024, height: 1024,
    mipmapped: false
)
desc.usage = [.renderTarget, .shaderRead]
// storageModeShared is default on Apple Silicon
let texture = device.makeTexture(descriptor: desc)!

// 2. Render pass writes to it
let renderPass = MTLRenderPassDescriptor()
renderPass.colorAttachments[0].texture = texture
renderPass.colorAttachments[0].loadAction = .clear
renderPass.colorAttachments[0].storeAction = .store
let encoder = commandBuffer.makeRenderCommandEncoder(descriptor: renderPass)!
// ... draw calls ...
encoder.endEncoding()

// 3. Compute pass reads the same texture—no copy, no barrier
let computeEncoder = commandBuffer.makeComputeCommandEncoder()!
computeEncoder.setTexture(texture, index: 0)
computeEncoder.dispatchThreadgroups(gridSize, threadsPerThreadgroup: groupSize)
computeEncoder.endEncoding()

commandBuffer.commit()

The GPU's render and compute units both reference the same physical pages. Xcode's Metal System Trace instrument will show zero blit or copy commands between the two passes. Run this on an M1/M2/M3 Mac (verify via About This Mac → System Report → Hardware showing Apple Silicon) and you'll see the memory traffic counter stay flat for inter-stage transfers.

Where the model breaks: pressure and tooling

The unified pool is a single budget. A large Core ML batch allocation can evict the GPU's texture cache, causing frame-time spikes. A Metal app that hoards MTLResourceStorageModeShared buffers can push the system into memory compression or swap, throttling the entire SoC. Xcode's Memory Graph debugger and vmmap still report the process's virtual size, but they don't break down residency per compute unit. Some third-party GPU profilers (built for discrete VRAM) misreport "GPU memory" as zero because they look for a separate VRAM region that doesn't exist. Reconfigure those tools to read the shared-memory counters exposed by Metal Performance Shaders or the powermetrics command-line utility.

Alignment still matters: buffers shared with the Neural Engine perform best at 16-byte alignment; textures need the usual tile-aligned dimensions. The coherency model is write-back on the CPU side and write-through on the GPU side, so a CPU write followed by a GPU read is safe after the command buffer completes, but a GPU write followed by an immediate CPU read requires commandBuffer.waitUntilCompleted() or a semaphore.

Verify and profile your own workload

1. Confirm hardware: sysctl -n machdep.cpu.brand_string should return an Apple M-series string.
2. Build a minimal Metal test (like the snippet above) and open Xcode → Product → Profile → Metal System Trace. Filter for "blit" or "copy" commands—there should be none between render and compute passes using the same shared texture.
3. Monitor memory pressure with sudo powermetrics --samplers smc -i 1000 -n 5 while your workload runs; watch the "Memory" and "GPU" power rails rise together, confirming shared usage.
4. If you see frame hitches, reduce simultaneous allocations: stream textures instead of caching full mip chains, release Core ML intermediate buffers after inference, and prefer MTLResourceStorageModePrivate for GPU-only resources that don't need CPU access (they still live in the same pool but can be evicted more aggressively by the driver).

Unified memory eliminates the copy tax, but it replaces it with a sharing discipline. Treat the pool as a single budget, profile with Apple's instruments, and you'll get the latency and bandwidth wins the architecture promises.