Microsoft Commits Maia to Multi-Gigawatt Inference, and Its Efficiency Metric Draws Questions

Microsoft will scale its Azure Maia family to multi-gigawatt deployments starting in 2027, making Maia "the primary inference system for Microsoft products and services." The architecture behind that commitment is unusually specific about data movement. The efficiency claim leading the announcement is not, and practitioners have said so.

Maia 200 is already in production, serving Microsoft Copilot and the MAI team, with GitHub Copilot and Microsoft Foundry to follow by the end of the year. It is deployed across the United States, Europe, and Australia, and Microsoft says it delivers up to 1.7 times more tokens per watt for built-in AI in Teams and Copilot than leading GPU infrastructure. Maia 300 begins production deployments in 2027.

Maia 300 is inference-first, aimed at long-context, memory-intensive, and decode-heavy work rather than training, which neither announcement mentions. Each accelerator carries eight HBM4 stacks, providing 288 GB and 18 TB/s within a 950-watt TDP. A rack holds 96 accelerators in 167 kW across 24 four-accelerator trays, reaching 27.6 TB of HBM and 1.7 PB/s aggregate. Compute chiplets are built on TSMC N2P, with memory-controller and I/O functions on separate dies, which Microsoft says raises compute FLOPs by 36% and contributes to 2.5 times the memory bandwidth and roughly twice the HBM bandwidth per watt of Maia 200. Memory organization moves from uniform access to a NUMA layout across chiplets so software can place data near the compute using it.

The networking is where the design is most opinionated. Maia’s AI network controller implements a custom AI transport built on Ethernet, and Microsoft describes the result as "one fabric rather than separate scale-up and scale-out domains." Accelerators initiate communication to any peer in a superpod "without involving the host or a discrete scale-out NIC." Two accelerators are directly linked within a tray, a switched fabric joins all 96 in a rack, and optical second-tier switches connect up to 128 racks into a superpod of 12,288 accelerators. The post names three latency domains: a direct pair, one hop within the pod, and three hops across pods, and expects software to keep tensor-parallel traffic in the lowest. The split between in-rack and cross-rack bandwidth is "not fixed at the system design time" but set by job packing, which is how an accelerator can drive up to 750 GB/s to any peer when only a subset cross racks.

That fabric carries a specific inference pattern. Prefill and decode workers are sized and placed independently, and after prefill completes, KV state "moves directly to the selected decode workers over the same scale-up fabric rather than leaving the domain through a separate host-mediated scale-out path." The same fabric lets a spare tray in one rack offset a failed tray in another, transparently to software kernels.

Microsoft’s headline figure is that it expects Maia 300 "to deliver 40% more tokens per megawatt compared to leading GPU-based systems," alongside up to seven times the rack-scale token throughput and more than 2.5 times the performance per watt of Maia 200.

That figure is what drew reaction, and the objection was to the unit rather than the number. Rick Bullotta wrote the most-engaged comment on Satya Nadella’s announcement post:

"Tokens per watt";. What an absurd metric. As if there’s a uniform, consistent definition of a token or a generic correlation to compute. Not. And such a metric would inherently reward verbosity.

Commenter Michel Floyd put the dimensional version in one line: "Shouldn’t it either be tokens per second per watt or tokens per KWh?" Vinay Bansal, founder of UBIQS and building a competing inference accelerator, questioned the workload the figure was measured on:

40% more tokens per watt, but at what context length? The workloads you named, sustained multi-step agents, carry long contexts, and every agent’s working memory grows with each step. At that point the binding constraint is memory, not compute, and the specs that lead this launch are memory specs.

If these reactions are representative, the architecture will be easier to assess than the claim. Microsoft never names the GPU systems it compares against, and its own framing is that Maia 300 maximizes "useful tokens per watt and useful tokens per dollar," without defining what makes a token useful. The announcements also differ in confidence: Scott Guthrie and the engineering post say Microsoft expects the 40% improvement, while Nadella’s post states that it delivers it.

A second question concerns who gets access. Every workload named across both posts is Microsoft’s own, including Copilot, GitHub Copilot, Foundry, the MAI team, and Dragon Copilot, and neither post says customers can provision Maia capacity directly. Phillip Pham, chief executive of Pexon Consulting, asked, in German, whether the gains translate into cheaper provisioned throughput reservations or whether the hardware stays reserved for Microsoft’s own services, since with committed capacity it is the price per guaranteed token that decides the benefit in production.

Microsoft has not published a technical paper beyond the engineering blog post by Rani Borkar and Saurabh Dighe, which the announcement links to. Microsoft says it will share more at Microsoft Ignite 2026 in San Francisco next month.