Presentation: Chaos Engineering GPU Clusters

Transcript

Bryan Oliver: I work on the Thoughtworks Radar. It's a publication we make twice a year. It's basically a publication where we share our perspectives and opinions on technologies across the globe and across Thoughtworkers clients. I'm working on a second book, because I don't enjoy free time, with O'Reilly. Actually, we just changed the title to "Designing Intelligent Delivery Systems" because we are reacting to a lot of the new scheduling patterns emerging in AI deployments. I do a lot of speaking and occasionally contribute to Kubernetes and other open-source projects. You would probably guess that that means I'm probably qualified to give this talk. The answer to that is, no, I'm not. This space is incredibly complex. The thing I'm speaking on is changing every day. I don't think any of us have more than a year of experience in this space at this scale. I don't think anybody really is, but we're going to give it a shot.

What Is a Really Big GPU?

To talk about chaos engineering graphics cards, we first need to understand quite a bit about large scale graphics cards because these are not the GPUs that sit in your computers. These are the GPUs that are running Claude, GPT, and that kind of thing. Let's just get a little bit of appreciation for that scale. Your gaming desktop where you play Fortnite and Rocket League and such is probably like a 5080 or a 4090 or something like this. It has 16 gigs of VRAM, maybe 24 if you got lucky, 10,000 CUDA cores. An NVIDIA H100 has 94 gigs of VRAM per GPU, 16,000 CUDA cores, and 400 gigs of network bandwidth for a total of 752 gigs of VRAM, because these things are deployed and purchased in lots of configurations, not just a single chip. Pretty sizable. Let's go bigger. If you've heard of it, it's the GB200 Grace Blackwell Superchip.

This thing is a monster. It has a total of 384 gigs of VRAM per chip, and it is deployed on this rack scale system with 72 of them in a single rack. It is deployed and architected in a way where it acts as a single GPU on the entire thing. It's actually about 13 terabytes to 14 terabytes of total virtual memory on a single GPU. The last time I was talking about some of these concepts, someone was like, how much does it cost? I was like, that's a great question. It's $3 million for one of those racks, approximately. The prices are always changing and varying, but that's an approximate cost. You'll come to appreciate that when we start talking about some of the details of this talk.

GPU Concepts Primer: Hardware

There's a few terms you need to understand for this talk. The first one is XID errors. These are basically just GPU fault codes. It's kind of like kernel panic in the programming world. ECC is Error Correcting Code. This is the memory protection that detects and corrects for bit flips. It's important to know this, but it'll be fine if you don't. Streaming multiprocessor, this is the compute units inside of a GPU. NVLink, this is NVIDIA's high-speed GPU-to-GPU interconnect. We will talk about this more. Thermal throttling. It's fun to know GPUs actually slow down when they're too hot. You probably know this if you're playing a lot of really difficult games on your desktop. It turns out in a single H100 GPU, I think there's like 90 different thermal sensors, all of which can trigger throttling in any given scenario. DCGM, this is short for Data Center GPU Manager. This is a set of two programs, not one program, created by NVIDIA. One of them is called dcgmi, and the other one is DCGM-Exporter. One of them you can actually control and do things to GPUs with, which we're going to talk about. The other one is for only observing, mostly in the Kubernetes world.

GPU Clusters

Let's talk about GPU clusters, because that's why we're here, lots of nodes with GPUs on them. If we go back to 2004, there was a group of researchers at Stony Brook University, the Center for Visual Computing. They wanted to model how a chemical and biological attack would spread through Manhattan's urban areas, namely Times Square, actually. They were trying to basically understand airborne dynamics from various attack patterns to help protect and model defense for the city. They used a method from computational fluid dynamics. I cannot even begin to tell you what that means, or how that works, or how that model works, but they used the Lattice Boltzmann Method, and they developed a simulation in order to do it. When they were doing this, they noticed something interesting. Because they were a visual computing center, they had a large array of GPU nodes inside of their data center, about 30 of them.

These were all just single GPU nodes with a 1 gig link in between them. What they noticed is that when they ran their model on the GPU cluster instead of on the standard cluster, they had about a performance gain of 100x. To get more specific, the simulation could compute a single Lattice Boltzmann Method in 0.31 seconds per step, which is 4.6 times faster than you can do it on a CPU cluster. This was the very beginning of GPU clusters. This paper is famous. It actually won the test of time award at the Supercomputing Conference, which is one of the most prestigious awards these papers can get. A fun fact about this discovery, this was two years before CUDA, so they had to use graphics shaders to express their computations, or in fun terms, they had to trick the GPU into doing science by pretending it was rendering pixels.

Just thought that was fascinating. Going to the history that's contextual to this, back in the '60s and '70s, we had high-performance computing. In 2004, we had the paper we just discussed. In the 2000s, CUDA came out in 2006, GPU clusters began to explode, mostly because of the paper. In the 2020s, we had LLMs. In 2024, we had a call for better AI on Kubernetes. This is the very abbreviated version of the timeline. Back at KubeCon Paris early last year, Tim Hockin got on stage at Contributor Summit, and he said, we have succeeded in creating a general engineering platform for microservices and APIs, now we need to address AI, or our project is going to die and be replaced. It's basically a call to all contributors to start getting things moving, and that conference really was a marking point, because Kubernetes had been focused on supporting AI, but from that point, it became pretty much the focus of the project.

The reason for this is a number, but the main one was the currently existing training platform and technology that you see quite commonly is called Slurm, which has been around since those early days on the timeline that you saw. It's quite complex to manage. It doesn't have a lot of those resiliency features that you see in Kubernetes today that operators wish for. It also doesn't help us with a lot of our telemetry and also chaos concerns you'll see today. The main thing is we wanted to use Kubernetes to handle the rise in complexity, and that's the theme of today.

Complexity in these things comes from a few areas. One of them is multi-tenancy, like you're talking to a ChatGPT and that's running on some GPU. All those requests you're making are going into a multi-tenant environment. There's performance priorities, but the biggest additive to complexity on these things is network enhancements, GPU-to-GPU network links. We're going to talk about what that is so you can begin to understand how to start thinking about chaos testing and observing it. In simple terms, we're talking about linking GPUs together directly through different protocols and technologies. The first one is called RDMA, Remote Direct Memory Access. If you think about, this is an oversimplified version of how packets might flow through a typical server. In this context, instead of going from application to GPU, we are instead going to have our requests go from application to GPU to GPU over a network switch.

You may notice it is skipping the kernel completely. This is by design. To give you a one-liner of what RDMA is, it basically lets one computer read another computer's memory directly, bypassing the operating system. GPUs can do this too. If you think about it in a more traditional sense, a traditional network, you have the application layer, socket layer, TCP/IP layer, device driver, and NIC hardware, where it actually goes out. In the RDMA world, it's going through application layer, which is going to be maybe NCCL or something else. It's then going to go through libibverbs. Then it's going to completely bypass the kernel. If you wanted to look at these requests, you would have to create your own eBPF Hooks just to see them. There are many forms of RDMA. One of them is RDMA over Converged Ethernet, or ROCE. This was used in the early days of RDMA.

It's very cost effective, but it's quite slow because it is literally RDMA over Ethernet. There are now other options out there. One of them is called InfiniBand. This is a fabric technology that allows you to increase that speed that you get from your GPU-to-GPU links. This basically just scales out RDMA at a large data center level. There's also NVLink, and NVIDIA's NVSwitch technology operates very similarly. It's a little bit faster because it's integrated directly with NVLink on GPUs. Point is that these are all building on some of those protocols like RDMA.

GPU Concepts Summary: Networking

To give you a quick primer on a few of the things we just discussed and some of the terms you might want to remember, RDMA, Remote Direct Memory Access. InfiniBand is a high-speed network fabric. You can imagine like a spine-leaf architecture you would see in a data center, but directly for GPUs, not for traditional computing. IP over InfiniBand, this basically just runs regular IP traffic over InfiniBand hardware, so you can do dual purpose if you need to. NCCL is NVIDIA's Collective Communications Library, handles multi-GPU communication. If you want to run an Allreduce job, you would use it through that. You could. It's important to remember this for later. Collective operations like Allreduce, Allgather, when I say these things, what I mean is this is an operation where all of the GPUs are participating in the action, and the slowest GPU that is participating is the bottleneck in these operations. Then, finally, libibverbs and UCX. This is really low level, but if you want to get into making your own observability stuff, you would want to start understanding it. It's basically low-level libraries for RDMA.

The Complexity Begins

Now we have a basic understanding of where we're headed. Let's talk about that complexity stuff. The first thing we have to think about in this GPU cluster world is topology-aware scheduling. What I mean by that is how do I place an AI workload, be it training or inference, on a group of specific nodes? What if one of those nodes dies, and another one dies, and we now need to shift our topology deployment to a different set of nodes? There's some complexity to this, and there's also some reasons why you want to do it. What we started doing in these GPU clusters is defining not just topology as in what region, what data center, what zone, but also what rack, even what block a specific GPU or set of GPUs is on. The reason for this is performance. When I mentioned before we were using RDMA in that cluster, the speeds at which you get are limited to that GPU-to-GPU fabric, and also by physics.

The closer these GPUs are to one another, the faster the communication is going to be. The reality of what's going on is you are trying to get your AI workloads to run as optimally as possible, but the GPU is not as consistent or reliable as CPUs are. The reality of your data center is going to be, you're going to have one that's down, three or four that are performing ok, and two that are performing not so great but are still functioning. You want to place your workloads on nodes that are not experiencing any of those issues, and also in contiguous blocks, meaning nodes that are literally quite right next to each other. This is easier said than done, but it's one of the problems we now face that we have to solve for when we're doing large-scale GPU infrastructure.

If we think of a more laid-out map, and we talked earlier about InfiniBand, you could see or think about a spine-leaf network architecture where you have a group of GPU servers on this architecture. If you're using InfiniBand or NVLink or NVSwitch, you're going to get around 100 gigabits to 600 gigabits per second between each node. NVLink provides, I think, 600. When you run an Allreduce operation, which I mentioned before is where, let's say you're deploying a job to eight GPUs, an Allreduce operation is going to use all of them. The bottleneck of how fast that job runs is the slowest GPU inside of those eight. You can imagine there's some pretty difficult scenarios. The first one is if you manage to get that job deployed to a node that has eight GPUs, that's going to be the most efficient and fastest way to run that operation.

If you deploy it to at least the same rack where the nodes are talking to each other over NVLink or a very short path RDMA, the performance is still going to be pretty good. When you deploy that same job to different racks, your performance hit is pretty massive, actually. I think the speeds are like, you're going from basically 600 gigs per second in this context to 100. If you think about the multi-node context, this problem gets amplified even further. One thing we've noticed is like, say you look at Allreduce latency and compare it to deploying across racks versus deploying across racks that are on the same fabric or switch, you're going to see a 20-millisecond topology-aware Allreduce operation versus 40 milliseconds in the default where you're not using topology awareness. It's pretty important. That's a 46% improvement. Massive. The problem is it's fairly complex. There are all these different components you can use to achieve topology awareness, whether it's in Kubernetes or Slurm.

You can use node labels and topology and affinity rules or custom schedulers and custom scheduler plugins. Then you have to go through a scheduling flow with topology awareness where you are submitting the pod, filtering it based on the nodes, scoring the topology. It's a fairly complex algorithmic process. Topology schedulers are coming out every day or within certain CNCF projects.

Another problem to continue to add to the complexity story is variability. I mentioned before GPUs aren't super reliable. If you compare like a big glob of CPUs to a big glob of GPUs and try to look at just overall resiliency and reliability, the GPUs are going to be much lower. This is because they're not as consistent. Meaning like if you have 100 GPUs that are all exactly the same, they are not going to perform exactly the same. There's going to be some that perform better than others for various reasons, whether it's cooling, power, how it was made on that day. There's a lot of different variances here. There's another paper that came out recently presented at Atlanta Supercomputing 2024 on variability awareness. This is now becoming an important topic as well. In this topic, what we look at is the fact that not all GPUs are performing equally, so we need to schedule our workloads based on the observed performance.

This sounds interesting and tricky because there's a lot of different factors that go into observed performance. If we think about it from our simplified diagram view, let's consider that we have three different job types. Type C, which is a really high priority customer. Type B, which is maybe this is the type of job that's going to be affected by a cooling issue on one of the nodes. Like maybe an Allreduce where the laggard GPU is going to affect the group. A type A where it's maybe a type of job that is not affected by one of the GPUs in the set being less than the others. We're going to prioritize type C first because it's a high priority customer. We're going to put them over here on this green block. Then we're going to prioritize type B next because it's going to be affected by any single GPU being less than the others.

Then, finally, we can place type A on this block where there's maybe some that aren't performing as well because it doesn't care. This is basically what the paper is proposing is if you can get this telemetry and this data on your data center and understand not only the current state of your nodes, but also the performance of job types on your nodes, you could start to classify and inform how you deploy workloads and jobs to these clusters. This is really complicated, though. Again, we're adding more complexity to the story.

Then we have asynchronous snapshotting. If you go back a little bit in training days, snapshotting is basically where you're backing up the current state of your machine learning run. You can now do asynchronous snapshotting where basically every node, every GPU is asynchronously uploading its current state, and that's being organized together. This is good, but also quite complex. The reason we need to do it, though, is in the synchronous world, all of the workers would pause. As you remember earlier, $3 million per rack. Pausing isn't really an option anymore. It's very expensive. On a 100-gig model, I think one synchronous checkpoint takes up to a minute. This gets very costly. If you're checkpointing every 10 minutes, you're losing about 10% of your utilization. The solution is we do fast snapshotting across every single node. All of the GPUs and nodes, it only takes them a few milliseconds.

Training never gets blocked. We commit only when the write completes. It sounds nice, but it's actually fairly complex to implement and get right. There are some issues there as well. Then, lastly, we have something called NUMA, which is Non-Uniform Memory Access nodes. To give you basically a simple version of what this is, it's a logical grouping of a specific CPU or set of cores, and it's directly attached to system memory, to which a GPU is physically connected via shortest path. It's meaning like, if you have a GPU board that is dual socket, there is a shared memory path for the GPUs on that board. You can think about how workloads are scheduled in this context as well. For example, local is faster. When a GPU and the CPU memory it needs to access are within the same node, you're going to have lower latency, higher bandwidth, direct PCIe path, no intersocket hops.

When you are needing to access memory across nodes, like we mentioned in the RDMA context, which we do need to do, but we want to avoid it when we can, we're going to have higher latency, lower bandwidth, more cross intersocket links, and also you're going to be using the fabric that we mentioned earlier. There's some impact on these. Data loading, checkpoint writes can be slower. CPU preprocessing and multi-GPU training gets really complex and can be slowed down by some of these effects. To give you just a basic number, NUMA misalignment can degrade GPU performance by up to 40%. Again, more complexity, but also important, costly. If you think about like NUMA-aware scheduling in default Kubernetes, or non-NUMA-aware scheduling, like let's say we put a pod on Kubernetes and this pod is a GPU workload that needs two GPUs. Simple. It gets scheduled to two different NUMA nodes, NUMA 0 and NUMA 1.

These are not NUMA-aligned nodes. They do not have shared memory. We're going to see a performance hit of about 50%. Whereas if we use Kubernetes' new topology awareness with NUMA nodes, we're not going to have this performance hit. We'll have direct PCIe access from local CPUs, local memory to GPU transfers, and all the resources are going to come from the same NUMA node. It seems like an obvious thing to make, but again, more complexity, more problems that we're adding to how we solve, observe, deal with resiliency in this environment. The reality of it is the default K8s scheduler, like if you're just rolling it out, out of the box, doesn't know about NUMA. It just assigns GPUs without considering CPU affinity. You have to think about implementing your own schedulers, your own topology awareness, your own observability, all these things when you're doing large-scale GPU training and inference.

Just to summarize before we jump into the chaos part. Complexity in this space is extremely high. This was just a few of the things you have to tackle when you're dealing with large-scale GPU clusters. There's more. Just those are the big ones. Those are the costly ones that you have to deal with up front. The traditional approaches to chaos engineering are not going to be sufficient. Pulling the plug in a node is not going to do it. There are different problems in this space that we now have to tackle, as you saw. Lastly, scheduling is a very complex problem, as you saw.

The Chaos Gap

Let's first think about the gap. Compare it to, say, traditional chaos engineering. In traditional chaos engineering, you're going to have stateless workloads, commodity hardware, one network, and CPU-centric failures, meaning things like memory leaks in Java, CPU exhaustion, disk is full, that kind of stuff. In the GPU context, training has state, first of all, so already completely different. You have model weights. You have checkpointing. We talked about quite a few other things as well. Cost. We mentioned $3 million per rack, just an example. These things are incredibly expensive. Just to rent an H100 for an hour is not your typical spin up a node on AWS and give them your credit card. You can. I think that's actually fairly affordable now, but it wasn't. With a GB200, you couldn't just do that. Two networks. You now have your traditional networking model and you have your GPU fabric model that you have to think about with chaos engineering.

Lastly, GPU-specific failures. Things like ECC errors, thermal throttling, and XID events. These are all foreign things. To point out, the current tools, some of them are starting to address these things, but they currently do not understand CUDA or DCGM, which limits their ability to help you do chaos engineering in this context. This may have changed. I'm not sure. I know Gremlin's starting to work on GPU stuff. At least as I know of now.

Let's first split things here a little bit. There's inference and there's training. In the inference world, we have stateless requests. You can load balance GPUs, horizontal scaling is possible, and model weights are read-only. In the inference world, it's fairly similar to the traditional chaos engineering approach. You still need to think about some of the complexity in the network. It's not as complex as the training world. In the world where we have stateful workloads, GPUs have to work in sync. Fixed topology requirements and those checkpointings are critical. There's a few different approaches we can take. Like in that traditional one, we can first just employ the normal chaos engineering approaches you do today, like killing nodes, latency injection, noisy neighbor, that kind of stuff. It's in the training world where we need to start thinking about new solutions, such as fault injection, checkpointing, fabric, and cascading failures.

We'll talk about some of those. There are four things we need to test. First, hardware simulation, this is ECC, thermal events, power problems, and XID errors. The second is resource exhaustion: VRAM, out of memory, and compute saturation. Even on these things with 14 terabytes of memory in a single rack, you still see these things occur when you're training really large models. Network faults, like in that GPU-to-GPU link, NCCL timeouts, fabric partitions. Then, lastly, we need to test workload disruption. What happens when we stop the job on one of the nodes? There's an important note. You are using incredibly expensive hardware when you do this. Try not to damage it while you're running potentially dangerous chaos experiments on your hardware. Dangerous to the hardware, not to yourself, I hope. The first problem is overflow. If we go back to our simple example, one node has eight GPUs.

This is not true of all nodes. It's just we're using it as an example. Some training jobs require set numbers of GPUs. They're calibrated. They're written for that number. Like the weights and everything are all calibrated for this node. I get we'll have eight GPUs. Stopping one is actually stopping the entire node. Because when you think about it, like in that Allreduce operation, for example, I mentioned before, it's using all of the available GPUs within the operation and it is bottlenecked by the slowest one. One of them being dead means the whole thing is kaput. We can think about something like node overflow, how we handle this problem. Where in the typical Kubernetes world or even just general distributed computing world, we've got maybe a set of idle nodes. Maybe we have a little bit of extra capacity in our cluster. Or we have a node autoscaler where we can just cycle in these nodes and put them in and everything comes back up and is running and everything is great.

Sounds nice. In the GPU world, this doesn't work this way. When that GPU goes down, all of the other ones, the entire job is now at risk, effectively. The real problem is, can we have one spare GPU ready? No, because these are all hardware directly linked. Can we have some spare nodes ready? The answer is you can, but it's very expensive. The larger scale you are, the more this is going to cost you. If you are designing for overflow across tons of jobs and customers or internal teams, the more overflow you have, the more costs you're going to be effectively just wasting. I like to summarize this one with, this is why chaos engineering and observability matter. You need to know the health before scheduling.

Training job resilience raises questions. In that context, let's think about a few scenarios. Can we just pull the failed GPU out? No, it's physically linked. Also, that job can't just run on three GPUs now. We would have to stop it, modify it, redeploy it, and then start it up again if we're going to run it on three. Can we pull all four of them and then just reschedule? Where are they going to go in the topology context we talk about? First of all, we need to make sure our checkpointing has completed. Then, secondly, that we're deploying to contiguous nodes, NUMA-aware nodes, all these things we talked about. You can't just simply reschedule, there are things you have to take into consideration. Can we just pull the whole node? This is the most expensive way of handling the problem because now you're going to have to wait for another one to come up or another one to become available from your reservation pool.

Or maybe you have extra ones available already and we're back to the extra cost problem. The goal of chaos engineering in this context is you don't want to get to this problem at all. You want to have been running chaos experiments on your data center, on your hardware, on your infrastructure so that you are already aware of where the problems are in your data center and you don't deploy workloads to where those problems are. You don't have to answer this question because there isn't really a good answer. To summarize a little bit, the cost of having extra nodes at the ready is extremely high. It's much more cost effective, like I said, to know the status of your GPU infrastructure. Knowing the status is pretty challenging though, just so you know. Let's imagine you have a large-scale GPU data center. You would need to have your own control plane monitoring platform that is getting anonymized telemetry data on all of your nodes and GPUs and not collecting information on the jobs running on them.

There's some work you have to do to get that kind of observability, but it's worth it. The techniques that you want to do are monitor the jobs and their performance. This is a wait and see, meaning like, you can build this architecture out where you have an anonymized telemetry data control plane. That's not enough. You need to actually use it and also preempt those problems. You need to figure out where those problems are ahead of time. It's not going to account for random failure. We need to be proactive and this is where chaos engineering comes in.

I have seven cases for you today that you can employ now to accomplish chaos engineering on GPU clusters. The first one is DCGM fault injection. DCGM is the project I mentioned earlier, written by NVIDIA. It allows you to go onto a node and actually inject failure onto GPUs. You can target a specific GPU and give it a failure. This project was written by NVIDIA, or this feature of it, the ability to give it failures to inject them. They did this because they needed to test their own hardware. They wanted to see what it did when they created the failures they were expecting. They were basically doing their own chaos engineering while building their own hardware. Now we can reuse this and put an API in front of it. We can do things with this like NVLink failures, power, thermal throttling, XID errors, and ECC memory errors.

These are all errors we can inject. I have to give credit to Rob Skillington at Chronosphere for this idea. He and I are working together on some of this stuff and this one was his, so it's a good one. To give you an example of some things you would inject, maybe GPU fell off the bus, GPU stopped processing, memory page faults, or maybe some degraded issues like ECC page retirement or preemptive cleanup. You're basically injecting these errors directly onto the GPU. You would then need to ask yourself, in order to have a successful chaos experiment, is your monitoring detecting these? Because they're not easy to find. You need to make sure you have properly implemented monitoring on your GPUs. Is the alerting firing? Is your scheduler responding? Meaning the data from these is being pushed to your observability platform. Then the scheduler that deploys AI workloads to your infrastructure should be able to use that information.

Then, lastly, does the job checkpoint and recover? It's a number of things. This one chaos approach can effectively exercise multiple teams because you can exercise your observability capability, you can exercise your alerting capability, your scheduling and deployment team, as well as the actual machine learning engineers using your infrastructure, making sure they are properly implementing backups and recovery and checkpointing. Very useful.

The second one is probably my favorite. I should have just put a, like, this is fine dog on the front of this slide because it's basically just, light the GPU on fire and see how your platform reacts. It's a little bit more than that, but basically you could use a project like GPU Burn, which is a containerized GPU resource exhaustion tool. He wrote it to test individual GPUs on desktops, like for gaming, but you can use it in a large-scale infrastructure and deploy it as a pod to as many GPU nodes as you want. It can even target specific GPUs when they have multiple GPUs on a single node. This will allow you to test things like OOM handling, CUDA context, maybe memory fragmentation, thermal management, because you're going to basically be ramping up the GPU to its maximum utilization, power delivery, noisy neighbor impact.

One thing you could do with this chaos experiment is basically deploy this pod as a sidecar to a node where another training job is running. Then that's your noisy neighbor test. You're going to see what happens when the job that was supposed to be there runs when something else is using up all the resources. Does it get rescheduled? Does it checkpoint and pull teardown? How do you handle these scenarios? It's a really good one.

To go back to RDMA, if we think about traditional network chaos, we now have to think differently about chaos engineering in the GPU context. Traditional networking chaos isn't going to help us. In order to achieve chaos at the networking level, we're going to have to try a few different techniques. The first one is NCCL environment variables. You can try disabling some, adding overhead, or forcing certain algorithms to run. This is fairly easy to do. You can do this today. The second one is traffic control on IP over IB. This is basically just adding latency, packet loss, bandwidth limits. You can do this in your GPU fabric today. You just are using different tools than you normally would. Then the hard one is eBPF Hooks. Like I mentioned before, to actually observe some of the traffic going from one GPU to another in a RDMA environment, you would need to create eBPF Hooks that monitor both the send and receive events on each GPU because it is bypassing the kernel.

It is not going through IPtables or TC. If you wanted to do some chaos engineering at that level, you would need to write your own eBPF programs in order to get there. This will allow you to test a number of things like multi-node training recovery, collective operation resilience. Things like when you're running a Allreduce on, say, 16 GPUs across multiple nodes, you'd be able to see performance hits from tests like this. To give you a more detailed example of what this looks like, on the left we have NCCL variable environment changing. We can just modify some of these and then run our training job. This is very easy to do. In the middle context, we jump into the nodes and run TC. It does require some level of access to your infrastructure in order to do it, but fairly easy overall. It's an API that's defined and a CLI that's available.

Then the hard one is dropping eBPF Hooks into your node. This is probably the hardest approach because the kernel will first validate and verify every eBPF program you deploy to a node before it is created. Quite a bit of work. You have to write them in C, just so you're aware, but also potentially the most powerful approach.

Number six, checkpointing chaos. I mentioned before, we had the synchronous and asynchronous checkpointing, which is where we're backing up our AI workloads or training jobs. We could do a few different things. Like in the synchronous context, we could just kill the synchronous checkpoint. That would be a nice chaos test, basically just make it stop and see how the team recovers. In the asynchronous context, we could fill the checkpoint's volume. What we mean by this is there's a pre-created volume for every asynchronous checkpoint. You could just fill it up and see how the job checkpoint responds. That might be pretty interesting, actually. I think that'd be a fun one. Then you could also network partition to storage. Just like really slow down one of the asynchronous checkpoints and see what happens. Then, lastly, number seven. I mentioned before how important NUMA awareness is in deploying workloads to GPU clusters.

This is a little bit more tricky, but pretty fun. NUMA has an available CLI API you can use to do some of these. You could wrap an API around it. You could do remote allocation, NUMA node memory pressure, like basically just memory exhaustion. You could do cross-CPU saturation or kill the topology manager completely and see what happens. Some fun things that might happen from this, silent performance degradation. This would be a good test for you to see if your observability is catching something that you would otherwise miss. Noisy neighbor on the wrong node, meaning basically if you are using a workload with NUMA affinity and you kill the topology manager, you might affect the workload on the other NUMA node and not realize it. Some fun stuff there. To give you an example of how this works, we can force memory allocation just with NUMA CTL.

Actually, I think we can use NUMA CTL. Yes, these three. Then to observe some of these problems currently today, you can use nvidia-smi in order to actually observe those problems. They're all fairly simple, though. They all have APIs so you can create these problems on your own simple Python program. It wouldn't be so bad. There are some observability gaps in this context, though. Like I mentioned, you can see what happens with nvidia-smi. You can see the GPU utilization, the memory, the power, the temperature, and everything looks great. In reality, the job took way longer because your GPU was waiting on remote memory, or PCIe bandwidth, or data loader, or 35% slower than optimal config. My point here is this can help you start to identify these gaps that you are going to see out of the box with default monitoring tools so you can start to detect and understand these a little better. You can add things like NUMA stats, NUMA events, training throughput, PCIe bandwidth to your monitoring platform. You start to realize you need these things. They're not just nice to haves.

I mentioned there were seven. Those were the seven. Now we need to think about how do we scale up chaos engineering because what I basically just showed you is a bunch of CLI and script stuff, which isn't very scalable or powerful across large-scale clusters. There's a few approaches you can take to this. The first is, write your own sets of cron jobs and put an API in front of them inside of Kubernetes. Then you can deploy these in a scheduled way through your CI/CD platform. We do some of this today with clients. You could use like Chaos Mesh or Litmus Chaos and write your own chaos experiments, and just use these platforms to execute them for you. Then, lastly, we are kicking off an open-source project. We named it GPUDragon, just like Chaos Monkey, GPUDragon. Why not? That'll be kicking off after the new year.

The URL is already online, though. It's just an empty repo. It's us and Chronosphere and a few others where we're going to actually make a project that implements those seven chaos experiments you just saw for starters, and welcome to have contributors. It'll be MIT-backed, open-source project through Thoughtworks. I want to touch on variability-aware scheduling just a little bit more from earlier just because we mentioned how important that is and how you can maybe think about chaos engineering in that context, even though variability-aware scheduling itself is still pretty bleeding edge. I mentioned before, not all GPUs perform equally. If you start to use chaos engineering in your environment, what you can start to do is discover that variability ahead of time. What I mean by this is we saw earlier the layout, the map of all the nodes, and some of them are green, some of them are red or yellow, you need to get that data somehow.

It's better to have the data up front than it is to find out why your customers are running on GPU infrastructure. When you implement chaos engineering tests like this, you can start to get that information up front. Then you can start to make variability-aware decisions because you have the data up front. It allows you to build baselines and validate your scheduling. All that to say, let's think about why this is important, aside from just cost.

I like to talk about the observability feedback loop. The first part is inject fault, as we saw. The second part is, did we detect the faults? Let's make sure we actually found out what happened. If we injected a thermal issue or a power problem, did we find it? Did we see it? Can we see it? These are important questions to ask, and questions we're asking ourselves all the time in these GPU environments. If we can't see it, we're going to add a new metric or alert. If we can see it, we're going to verify what we did with it. Then we're going to document and repeat. This simple process is so powerful. When we started implementing chaos engineering in order to drive our observability platform at any large-scale GPU infrastructure client or otherwise, that observability capability becomes really powerful, really robust through this process. Because instead of just waiting for things to happen, we are preemptively building our observability platform based on problems that we solved before they occurred.

There are some checklists or things I like to mention. The first is the things you can do today. You can look at GPU utilization, memory use, thermal state. These are the basic things you can look at and observe today. Chaos engineering in the way we've described will help you understand which of these is the most important and which ones you need to start tracking. There are some gaps in GPU observability today. For example, none correlate GPU health with job performance. It's actually quite hard to pin down a single Kubernetes pod to a single GPU in the utilization that it's incurring. This is not easy to do today.

Summary and Key Takeaways

To summarize, we had seven different chaos engineering approaches. There are many more. These are just the ones we talked about today. DCGM fault injection, GPU burning, light it on fire. Network fault injection. There were three different types. Checkpointing and NUMA chaos engineering. There are many more. These are just the ones that are important to us at the moment and for the open-source project we're kicking off. Some of the takeaways from this talk. I think maybe now you understand you need to do this. It's not a nice to have or a want. If you are paying millions of dollars for GPU infrastructure and you are not trying to predetermine the problems with your observability platform, it's going to cost you a lot of money. I would do this now. Start with the basics. DCGM fault injection, testing your networks, checkpointing chaos, and then finally getting into that chaos observability loop.

Just start small, one at a time. One news bulletin. NVIDIA just bought Slurm. I don't know how that's going to affect our overall space, but it'll be interesting. They obviously also work on Kubeflow and NeMo and a lot of other scheduling technologies. It might just mean they have a scheduler technology or it might be good for this space. We'll see. Just so you know.

Resources and Next Steps

Next steps. Check out that repo or just give it a star. Check out DCGM. That's probably the fastest place to start. You can do this with a basic GPU in your gaming desktop today or just rent one online for a few hours. Read those papers, like the PAL paper is a really good one. Then check out our Platform Engineering book at Manning. It doesn't talk much about GPU infrastructure, but it talks about overall platform engineering infrastructure. You can find that on Amazon.

See more presentations with transcripts