NVIDIA and Apple Solved the Hardware. Here's What's Left to Build.

After GTC 2026, one thing is basically settled: the hardware layer for on-device AI is no longer the bottleneck. NVIDIA's RTX Spark packs Blackwell GPU + Grace CPU + 128GB unified memory into a desktop form factor. Apple's M-series chips with unified memory architecture and efficiency-first design let 4B and even 7B parameter models run smoothly on a MacBook. Two different approaches, same destination: consumer hardware now has the compute foundation for running on-device AI agents. Chip vendors have done their part. The next question is: how many layers are still missing between "chip can run an AI model" and "an on-device agent can actually complete useful tasks"? This post maps out the full technology stack for on-device AI agents, examining each layer's maturity, identifying gaps, and tracking what the open-source community has built so far. Layer 1: Silicon (Ready) On-device AI inference has different chip requirements than traditional compute workloads. The core bottleneck isn't peak FLOPS — it's memory bandwidth and unified memory capacity. LLM inference needs model weights fully loaded into memory, with high-frequency data movement between weight matrices and activations during computation. If memory bandwidth can't keep up, raw compute power just sits idle waiting for data. Three main silicon paths exist today: NVIDIA N1X : Blackwell GPU + Grace CPU heterogeneous architecture, 128GB unified memory, petaflop-class compute, targeting desktop workstations Apple M-series (M4/M5) : Unified memory architecture with GPU and CPU sharing memory, optimized memory bandwidth, configurations from 32GB to 192GB Qualcomm Snapdragon X : Targeting laptops and mobile, NPU-accelerated inference, relatively limited memory configurations Different emphases, but one common takeaway: 2026 consumer silicon can run 4B+ parameter models for real-time inference. This layer is ready. Layer 2: Inference Frameworks (Mature) With silicon in place, efficient inference frameworks are neede