The yield imperative: Turning AI infrastructure into useful intelligence

As we enter the next era, what will be the defining measure of our progress?

Every industry has a word that shapes how it thinks. For pilots, it’s safety. For insurers, it’s risk.

For the semiconductor industry, it’s yield.

Yield does not ask how elegant the solution is, how many years it took or what the roadmap promised. Rather, it asks one simple question: What useful output did we produce?

For more than 60 years, the semiconductor industry has asked that question, relentlessly maximizing the number of usable chips produced from every wafer. Generation after generation, wafer after wafer, it is precisely that discipline that turned the transistor from a laboratory curiosity into the foundation of modern life.

Today, we need to apply the same principle to the unprecedented resources that the world is pouring into AI: capital on a scale once reserved for nations, gigawatts of power and record-breaking fabs and datacenters.

The question that will define this decade is the same one this industry has always asked: What actually comes out? Not just chips and tokens, but as affordable intelligence, as work that matters and as outcomes that improve lives.‌

That is the yield imperative.

The limit of more 

AI has proliferated with remarkable speed, at a rate of adoption faster than the internet, the PC or even the smartphone. Yet, global penetration still stands at just 18% of the working population, and the vast majority of that usage is chat-based. As systems move from answering individual prompts to reasoning, planning, using tools and executing longer agentic workflows, the infrastructure equation changes dramatically. A single agentic task can use more than 3,400 times as many tokens as a typical chat interaction.

We are only in the early innings of agentic adoption, and the infrastructure is already strained. Power is setting the limits on what we can build and when. Packages and racks are growing larger and denser. Memory is becoming an even tighter constraint.

For years, the industry’s rational answer to each new requirement resulted in more: more silicon in the package, more memory beside it, more power to feed it and more fiber to connect it. Each generation delivered meaningful progress. But when each new gain requires more input than the one before it, we are on a treadmill. It moves only as long as we keep adding to it.

I believe we need to pursue two paths forward. The first is evolutionary: we continue improving the architectures we have today, driving incremental efficiency, utilization and economics within each generation. The second is transformational: changing the curve itself with innovation in new architectures, new materials and new approaches to system and model design.

The history of our industry is defined by transformations like these. When increasing CPU clock speeds ran into the power wall, we moved to multicore processors. When planar NAND reached its limits, memory went vertical.

And now, once again, we have an opportunity to challenge our assumptions and rethink the fundamentals. Because the next chapter of AI won’t be defined simply by how much infrastructure we build, it will be defined by how much intelligence we can create from it.

Engineering useful yield 

For decades, the computing industry has optimized yield in the context of manufacturing. Today, that discipline has to extend across layers, from datacenters and silicon through models and the agentic harnesses that orchestrate them. And the work does not stop once the technology is built. We must then deploy and scale it faster, while developing new tools and systems to maximize utilization across our fleet.

From development through execution, each layer has a yield of its own, and losses and gains compound across them. Capacity at any one layer is only a starting point. The real measure is how effectively those layers work together to produce useful output from the system as a whole.

Our experience at Microsoft building and operating AI infrastructure at scale has reinforced two key lessons. First, the biggest constraints are rarely solved in the layer where they appear. Second, when we attack a constraint across the whole stack, tradeoffs that seemed inherent to the problem often turn out to be artifacts of the architecture.

The greatest advances often come when we apply these learnings through co-design, working across layers to turn apparent limits into solvable system constraints. Innovations in memory, networking and power show what this approach looks like in practice.

Memory: More intelligence from every byte 

Today, memory is viewed as a supply problem or a component problem. In reality, it is a system problem.

In AI inference, memory is now setting the limits on system performance. It must hold larger models, preserve longer contexts and deliver data fast enough to keep the compute fed. And agents raise the bar even further. Generation, retrieval, tool use and persistent memory run together in loops that can last minutes or hours. The result is a much longer memory horizon, with far more information kept close to the compute and available across an expanding sequence of turns. Doing that efficiently at scale will define the next generation of AI infrastructure.

Our experience building the Azure Maia platform demonstrates that memory bottlenecks are not resolved by a single layer.

Model architecture, data science and compression can reduce the amount of KV cache, which stores the model’s working context during generation. Software can manage memory hierarchies more effectively, silicon can be optimized for data movement efficiency and compilers can place data closer to compute. No one change removes the constraint. Together, they increase the useful intelligence the system can deliver from the same memory resources.

That is useful yield: not simply adding bytes but getting more useful intelligence from every byte we already have.

Networking: Designing across layers

As we zoom out to the cluster level, we see that intelligence does not come from one chip. It comes from thousands of chips operating as one system. Faster links matter, but the productivity of the system also depends on congestion management, failure recovery, workload placement, programming complexity and the boundaries between silicon, system and software. Together, they determine whether expensive compute is producing intelligence or sitting idle.

When architecting the platform for Maia, we did not begin with an existing networking design. We began with the outcome we wanted to deliver: efficient inference at fleet scale, designing across silicon, networking and system software. Instead of separate scale-up and scale-out fabrics, we built a two-tier scale-up network, integrated the NIC functionality directly into the chip and developed a custom transport layer.

The result is scalable, consistent performance across dense inference clusters, with a unified fabric that simplifies programming, improves workload flexibility and makes better use of available capacity. And with less network hardware needed to deliver this performance, we also lowered the cost of running the entire system.

Our objective is not merely to move data faster. It is to keep more compute productive and deliver more tokens from every watt and every dollar.

Power: Co-designing for efficiency, from grid to chip

Moving from the cluster to the grid, AI has introduced new challenges around power availability, distribution and utilization. Racks have gone from tens of kilowatts to hundreds of kilowatts, and datacenter campuses can operate on the scale of gigawatts. Power used to be something the system simply plugged into. Now, it is something we design around, from the grid to the chip.

That is why the industry is rethinking power across the system. Solid-state transformers and 800-volt direct current power delivery can reduce distribution losses as power moves through infrastructure. Power and cooling are no longer downstream of the design, they are part of the product definition from the start. And increasingly, that co-design is needed all the way into the silicon.

Azure Cobalt 200, our Arm-based server CPU, shows what this looks like in practice. We designed Cobalt so that every core has its own voltage and frequency controls, paired with software-based, per-virtual-machine power capping. This finer-grained control enables targeted power adjustments while protecting the performance of critical workloads, allowing us to run more servers within the same power envelope.

As Cobalt demonstrates, hardware-software co-design enables us to more effectively turn every megawatt into customer value.

The breakthroughs between boundaries 

The pattern we see across memory, networking and power extends throughout the system: start with the useful output, then optimize the whole rather than any one layer. This first requires us to be precise about the output we are optimizing for and which design constraints are truly fixed. Are we optimizing for peak performance or sustained system throughput? Would the workload benefit from significantly more capacity with marginally less redundancy? What creates more value: a broader set of capabilities or significantly earlier customer deployment?

Not every constraint in today’s systems is a law of physics. Some are inherited from decisions made elsewhere in the system and can change only when we work across traditional boundaries. Evolution comes from the steady gains each company drives within its own domain, but transformation comes when we challenge those assumptions together and redesign the system as a whole.

The breakthroughs ahead will emerge from collaboration across the ecosystem, spanning hyperscalers and silicon providers, equipment makers and materials innovators, utilities and datacenter operators, model builders and software developers.

Toward full yield 

But tokens and intelligence are not the finish line. What we produce becomes the input for someone else’s work.

What matters next is how broadly that input translates into productivity across the economy and value in people’s lives, whether it helps a scientist accelerate discovery, a clinician identify a signal earlier, a student get help at the right time or a small business find a new path to growth.

That happens as AI becomes part of everyday work across industries and around the world. People build on it, new uses emerge, the tools improve and the value compounds.

For that cycle to spread, AI must be broadly accessible. And at scale, accessibility depends on efficiency. It is the only way to deploy enough intelligence, and at a cost that allows it to reach everyone.

When every person and every company can access intelligence, build on it and create value of their own, that is full yield.

We will continue to build capacity because the world will need it. But our defining measure of progress must be what comes out: not only chips or tokens, but useful intelligence translated into empowerment, opportunity and human achievement.

That is the yield imperative. And it is work our entire industry must take on together.

Rani Borkar leads the core organizations responsible for planning, architecting, developing and deploying hardware and infrastructure for Microsoft’s leading cloud computing platform — from silicon, to systems, to supply chain.

Learn more about Microsoft’s silicon to systems approach to Azure infrastructure.

Tags: Agentic AI, AI, AI Transformation, Azure, Azure Cobalt 200, Azure Maia