Presentation: Designing AI Platforms for Reliability: Tools for Certainty, Agents for Discovery

Transcript

Aaron Erickson: Tools for Certainty, Agents for Discovery. Who here had this moment at some point in the last three years? I know, what if we added AI to this? Imagine what could go right? Or, imagine what could go wrong if you try to just apply AI to whatever, just let's throw tokens at it forever. There's all these other bad ideas that have existed in the world, and why don't we just put another paper stone on the list. One of the things I think we could do is sometimes the road to hell is paved with things that sounded like a good idea, but actually didn't turn out to work as well as we hoped it would work. This is a screenshot from my prior life before NVIDIA. Those of you who've worked in a large organization, who here has ever had an org chart? Like, used in Workday, or maybe you have a special one in your company, and you use it to figure out, who does this person report to?

What team are they on, so on and so forth? I worked to build a startup around that. We actually got some funding, got a few customers. What happened in 2023 was something along the lines of, you have no chance of getting more funding unless you have AI in your product, I think is how the conversation went. Of course, what do we do is we decide, I know, let's build a ChatGPT plugin for your org chart. I wonder how this could go. Why not? We just had GPT-4 made available. This is early to, I'll call it April 2023. I always troll thought that you could use AI to do this at some point. How much worse could a reorg get than using AI for your reorg instead of using a big consultancy? Let's go and find out. We built a plugin where you could ask, I heard you're flattening orgs, can you help with that?

Engineering is pretty important. Maybe you should have fewer layers. Isn't it nicer to have fewer layers in an org? Isn't this fun? We built a plugin. This is alpha. You can see Model Plugins ALPHA. Anybody remember plugins for ChatGPT? That was probably our first place where we could do agents. We might call them MCP servers now. We might call them Claude skills now. Even at the time, I'm like, this is pretty powerful. This is pretty neat. We can just take multiple AI things and mix them together and make new things. I wonder what we can do. I asked it, why don't we do this? We connected an endpoint that would allow it to read a JSON structure that would talk about your organization structure. We gave it a series of actions it could do, such as move somebody from one place to another in an org.

It was, at the core, an event-based system, so it was really easy to translate this to, here are the nine different kinds of moves you can do with people in an org that would allow you to then ultimately do a reorg. We asked it, why don't we do a reorg? It came up with some good ideas. Granted, it's probably somewhat standard what you would do if you called a consultancy to do it. We hit the go button. It was able to generate a restructuring plan. It literally went in, gathered the data from the organization structure, wrote some Python in place to figure out, how do you move the things around? Then would call back to the org's-based system and give you a draft of that reorg. It would take those events that it thought it would be appropriate for doing that reorg, and then went into and generated this flattened organization for you.

I remember showing this to people that, wait a second, we have an AI that can write your reorg email for you? Isn't that a great idea? Whoever has written a reorg email and thought this is going to be a great piece of writing. No, it's almost always the most boring writing you could write, the most anodyne writing you could ever make. This is perfect for LLMs. I can't think of a more perfect thing for LLMs than reorg emails have to be safe, as anodyne as possible. We would take that set of changes, put it into a reorg email.

The NVIDIA Chapter

We didn't become the future of HR, as you could guess. For some of you that are wondering, did I just walk into a Black Mirror episode? You're thinking, what do you AI people do? Do you sit around and watch Black Mirror and go, that's a good idea, we should do that. This turned into something in my mind that was aligned around that, but we're a seed startup. Hopefully you weren't reorged by this tool. If you were, I'm sorry. No, we did not, in fact, become the future of HR. I crashed out. I landed at NVIDIA in 2023. I heard they were this up-and-coming company that made GPUs that were useful for AI. I thought, why don't I take a chance? This might work out. Then I found myself thinking there's a bunch of concepts that are translatable. My first system I was building at NVIDIA was not AI.

It was a system that helped you understand all your GPU clusters and figure out how do you allocate GPUs to all the different research groups in NVIDIA where GPUs are a scarce resource, and you have hundreds of teams that need GPUs to build new models, try new ideas, try new experiments, try to build other open-source stuff that could eventually become things like Nemotron, eventually become things like Cosmos, eventually become things like our AV models, eventually become BioNeMo, these foundational models that we get so excited about at NVIDIA. It started out, I was building this system, and there's some interesting analogs here. We had, in my prior life, human resources. You have open positions. You have employees. You have complex hierarchies. You have performance management, all these things that you would think of that go into an HRIS, like Workday or something like that. It turns out we were dealing with some very similar concepts.

We had GPU resources. People would ask for headcount the same way they would ask for GPUs in my world. It wasn't that different, actually. We would have idle GPU clusters. These are places where some work should go. These are almost like open positions. We have AI training jobs. These are like your employees, to borrow the analogy. We have this complex hierarchy, just like you have an org structure that has like VPs and directors and stuff. You have cloud service providers and regions within those and blocks within those regions. If you want to do a training run, you probably want all your GPUs in the same block so that your InfiniBand networking and all the kinds of networking you need between the GPUs runs hyper-efficiently. There's some really interesting constraint solving that goes into how you actually locate GPU jobs in massive amounts of clusters. The other thing that we have to care about that's somewhat related is we want GPU observability. We want to understand how are these things performing. Are they having fan failure issues? Are they having a lot of wattage fluctuation? It turns out how power goes into these things is actually really important. There's all these things that are kind of similar.

The LLo11yPop Project (Agent Architecture)

About four or five months later, our team emerges with what we called LLo11yPop. This is also why I'm never allowed to name anything in NVIDIA ever again because the name is terrible. I thought it was clever, large language model for observability and put a pop at the end so it's a word. It worked for me. We thought of this in a couple different ways. First of which was we just try to throw AI at the problem. A lot of times what would happen is it's trying to apply too many things at once. It's like trying to take the context in the world like a regular LLM does and try to generalize from that. What we found pretty quickly though was if you constrained the problem, if you constrained what you're asking the LLM to do, it had much higher accuracy on answering those kinds of questions.

We didn't want people going into the system and saying, what will the NVIDIA stock price be? Irrelevant question, it doesn't know. We all know LLMs, they try to answer the question the best they can, and it might say, based on the usage, it's this. No, we didn't want it answering that kind of question. We want it to be constrained to answer the kinds of questions we care about. What we found is if we divide and conquer the tasks that it's trying to do, so build retrieval agents. Our first retrieval agents were things that were just, how do you take this question and convert it into SQL, or convert it into an Elasticsearch query, or convert it into some other kind of API call? You could reliably get data out of them if they were constrained and give them a few good examples of how to do it.

We were just using basic RAG and prompt engineering. We got really great results by building a few dozen purpose-built retrieval agents that just knew how to work against one query or one table. We paired those with analyst agents. Analyst agents don't know anything about databases. They know what kinds of questions you should ask about a given topic. If you're thinking about how do H100s fail, we could take an analyst agent that understands how do heat issues tend to happen. We could look at incident reports where we had a lot of heat issues and we could actually do just some prompt engineering, a little bit of RAG at first and a little fine-tuning later on to build analyst agents that were LLMs that were good at one specific problem, that could then use these retrieval agents to get data relevant to that problem and reason through how you might solve that.

We would orchestrate these with an orchestrator agent that had some idea or a goal that it wanted to accomplish, and so it might work with multiple analyst agents, gather up some ideas about what to do to solve a given problem or to optimize the fleet in some manner that we wanted. Then we would give that to action agents, which we then automated the entire fleet with. No, we didn't. What we would do is we would surface up sometimes into Slack, sometimes into Jira, sometimes into other kinds of systems, recommendations of things you might want to change, recommendations of things, "We've seen this condition. It somewhat is in the same pattern of this incident that happened five years ago or this incident that happened three years ago. How do we raise that context to the right person that owns this set of GPUs or is responsible for this service provider, and let them know that there's a problem?" They would interact with tool agents that were very simple, similar to your retrieval agents that would just, I know how to call into this one thing.

I have examples of how to make this Jira ticket. You might be looking at me going, why don't you use MCP servers for this? Isn't that what they're for? Why don't you use Claude skills for this? Isn't that what they're for? You'd be right. This is all before that stuff. We had to trailblaze that a little bit and figure out, if you don't put in good examples into the context window, it's not going to know what to do. If you let it try to solve too many things, it will reason too long, and sometimes just give you these wildly inaccurate results because you let it think too much about something. Those are the kinds of things we had to figure out as we were going.

Lessons from LLo11yPop

Some of the lessons from the LLo11yPop project, and again, it was interesting for me because we actually moved on to other things this year, but we had a need like what we built with LLo11yPop, and I had one of my engineers that joined later on build the same thing we spent five months doing in approximately six hours, using MCP, using basic LangChain, using just a bunch of things. Either we were really dumb or maybe we were a little early, but we had the opportunity to learn as much as we could so we could really understand how things like MCP and skills and other more advanced techniques would work. One of the first failure modes of a system like this is we have a really great operations crew, and they would always come up with these questions like, where are the zombie nodes? How do you answer that question when you're asked where are the zombie nodes?

The LLM would come up with some good ideas about it. It'd be like, zombie, I don't know what that means, but it might mean where there's no network connectivity to it, so it would look at some DCGM metrics and whatnot and find out, this is almost right. It's almost like you gave an intern access to the GPU infrastructure and didn't tell them what zombie meant, but they made some guesses as to what it meant. It turns out these systems rely on what I call rare context. Rare context are things that are specific to your company, language that your company uses, examples of what zombie means to you. Those things are going to be different per company. If anybody here is a provider of AI agents who are like, we can work in any company, out of the box, zero-shot. BS, no, you can't, because you don't know the terms that are used in that company to tune your models, or to even just prompt engineer your models to give you the right output.

You're not going to know where the zombie nodes are, not in the way that an experienced operator might without that rare context. The other thing we learned was, I used to joke that AI stands for angry intern, so there's your angry intern. If you think of an intern's understanding of what a database looks like, sometimes you might say, I know you can do joins, I know you can do complex queries, but let's, while we're learning this, start with the basics. Let's start with a very wide schema with lots of columns in it, and actually just make you do SELECTS, GROUP BYs, maybe some filtering, and here's a few hints for how to get past pagination issues. Here are some ways that we just read the database. You have read-only access to the database. You can't write anything to it. No Bobby Tables, delete, my name is, delete tables, that kind of thing going on, just here is a very specific query, and here's an API endpoint that only ever allows you to do reads.

We gave it that, and we got much better results in our evals when we used flat schema. This might be obvious to a lot of people, but when you give AI a simpler task to do, it tends to have much higher reliability. One of the other things we learned, so if we're driving to hell, you should have an off-ramp on the way, ideally, and that off-ramp is determinism. Who here has ever heard determinism used as a word when they really mean, I don't want AI? I've seen it a little bit. It's like, you can't argue against AI right now in some places, because it's like, that sounds like you're not on the AI train, but you'll say, I need that to be deterministic. It's your way you can implicitly argue no AI when it's politically unviable to do so. In the industry, I feel like what that has done to us is it's made us force everything into this false dichotomy where it's either deterministic or not.

The reason I do this talk, the reason I care about this talk is because, absolutely, no, it's not like that. Determinism is helpful. Determinism is useful. Determinism is how you ground things for reliability when you have agentic things that are discovering new things. That's the name of the talk. The example I would give you is if you are running into the same kind of problem, the LLM doesn't quite get that join right, or the LLM doesn't quite count the GPUs correctly, or find where the fan failures are correctly, what you can do is you can say, here's a rule of thumb, kind of like you would do in a Cursor rule. When you run into this situation, use a query that has this pattern, and here's how you parametrize it. You simplify the problem for the AI. You still keep that discoverability. You keep that ability for it to reason, but within constraints and given certain tools and given certain almost like runbooks for how to get certain kinds of data. The more explicit you can be with this, the more reliable you're going to be. Even if the system could do this, what you allow it to do is solve higher-order problems because it doesn't have to spend tokens figuring out, how do I count things? I don't want it spending tokens on things that are classify problems.

Purpose-Built Agent Hierarchies

That's a cheesecake factory menu. The best thing about the cheesecake factory is you can have anything you want that's cheesecake and I think almost any other food, but I chose the cheesecake page. Who here has ever seen somebody look at that menu and not make a decision for 20 minutes? If you like cheesecake, how do you even decide? They're all kind of similar, so you might get that wrong in your eval set if you accidentally choose the broccoli cheesecake, as interesting as that might be. Hopefully the LLM doesn't make you do that, but they too can suffer from a paradox of choice. One of the things that we found to be useful is to build things into a hierarchy of agents. You're like, are you obsessed with org structures, or what is going on here? Maybe I am a little bit. I'm obsessed with structure, and maybe that's a better way to put it.

You have a VP agent that has maybe a wider context window, but doesn't know how to do anything specific, like most VPs we know. I don't ask my VP how to write a document. Actually, if I have one that's really good at writing, I might, but most of the time, I ask my VP for VP-level things, but I don't ask them how to do an expense report. My manager agent might have more ideas, it might have more boundaries for what I want to do, but there's a reason why we love individual contributors, because they do things. They don't just say things. They don't just send email. They do things. That's why we have these levels of things, because as it turns out, AI, just like people, and we should never anthropomorphize this stuff too much, but there's some similarities in that people have a limited context window as well.

We develop organizational hierarchies so that you can have people that are good at their thing that operate in a manner that you can then collapse it up to achieve bigger things overall. The exact same thing happens with agent hierarchies. You can build agents that are good at one thing and have it not necessarily involved in this other thing that it's not good at. Why aren't we doing this more? It turns out we are. People invented true AI and all these different agentic systems that made it so we don't have to build one. One of my favorite things about NVIDIA is the minute we find some other company does something better than what we could do in a space like this, we immediately cease the project and we move on to another one, because our goal is to build things that aren't being built anywhere else, open source it, and then make the rest of the world actually implement it at scale so then we can take advantage of your innovation, we can buy the product from you. I feel like that is something we should do more in big tech to help all the startup folks and help all the people in other companies that are focused on specific problems.

One of the other things we found, and I think you're like, this is obvious. Software engineers believe this is obvious. Yes, test pyramid. Who here knows what a test pyramid is? A test pyramid means you're going to test at a lower level for those lower-level agents on very specific questions. Your evals are going to be a little bit like what you might have had with unit tests back in the day, but they're going to be a lot different. We use LLM-as-a-judge. It works well enough for our purposes. I know people have more elaborate techniques, and there's a whole science in how you do good evals. The very basic mental model in my mind is start with low level things that you run at each individual agent level, focus on that unit of whatever that individual data analyst agent is good at. I'm going to have a bunch of evals that check whether that query that counts GPUs is in fact counting GPUs and doing so in a reliable way.

We're actually going to measure the effectiveness of that. We're going to look at operational charts to see how often it's getting the right answer. We're going to sample, make sure it's still getting the right answer. We still have to use LLM-as-a-judge, because as you might understand, when an LLM comes back with an answer to something, it often states it back at in multiple ways because they're, like we said, non-deterministic. If you ask it, who was the first president of the United States? It might say President Washington. It might say George Washington. It might say something else. As long as it's that direction, you can have another LLM to say, does this mean the same thing as George Washington? We use that technique. We use it a lot of places. It's not perfect, but it's good enough for us in this case. You have those at those lower levels.

Analyst agents that are aggregating other agents, they're aggregating the work of multiple worker agents. Again, the more levels of the chain you go up, you might have higher failure rates. You might have higher incorrectness rates just because you're stacking error. That's one of the design constraints you have to have. The analyst agent will then run tests on things that aggregate results from multiple lower-level agents to see, did we get the right answer? It's the equivalent of an integration test in a system, only it's more elaborate and has to do a lot more orchestration and costs a lot more to run. It costs inference tokens to run. You're going to have different dynamics about how you run them. Then you have end-to-end supervisor agent evals. At that top level, when we say, do a run of looking at 1,000 GPUs and we look at the ground truth to see what it actually is.

We sample it, we see, is it getting the right outcome? These are the same kinds of evals, a lot more expensive to run, you have a lot fewer of them, but they're still important to run. Otherwise, you're going to fail like a lot of AI pilots fail, where they get to 80% effective and 80% effective just isn't good enough for financial transactions, or all sorts of other domains where that level of accuracy just good enough for demo, not nearly good enough for production.

Agent Archetypes

Through all of this, we came up with several types of agents that you might run into in the wild that you might think about for agentic systems that you're building. My earlier slide had, I think I said, what if you had an army of idiots that you could control at your whim? I didn't want AI to become that. I framed it as, imagine you have a bunch of AIs hyper-focused on that single area of knowledge and nothing else, but they could do that one thing really well. Wouldn't that be great if you could do that? Can't you think of a lot of use cases for that? As I was thinking about that, of course, you got to have a Michael Burry slide. I was thinking about this scene from Big Short, in this case. In this scene, the person he's talking to is questioning, after he says, "I want to know about every mortgage bond on the market at all." "You want to know what the top-selling mortgage bonds are?" "No, I want to know what's in each one of them, and I want you to analyze it." Anybody ever have a problem like that that they've been asked from somebody to solve?

You've probably had some version of this that looks like this. I've got a worker agent, go look at all 100,000 clusters. Analyze them in detail for wattage fluctuation issues, and find the ones that seem to be not running normally. This is a form of anomaly detection. We've got much better ways to do it now, but this works pretty well, particularly for domains where it might fail for reasons that are fuzzy, or it might be connecting a couple things that it didn't notice before. An agent can come up to that, a conclusion that wasn't immediately obvious, and start to surface that. I think I read in The Hacker's Dictionary or something a long time ago, about problems that are the painting all the rocks on the beach. There are certain kinds of problems that are laborious, they require a little bit of mental energy because every one of these is different in its own little way, I don't have a lot of intellect I'm going to apply to this, but I have to apply a little bit.

I call them low-level intellectual horsepower applied at scale, might be another way of thinking about it. That, to me, is a classic worker agent. Go look at all 100,000 clusters. All sorts of problems. One version of this we actually did was, maybe you want to look at different companies that way. You might say, look at all the companies in my portfolio in this certain way and look for certain things. You might say, take any noun of importance. Look at every server in your organization and see, does it have the right configuration, even if my Terraform says it's supposed to work a certain way. I feel like almost every interesting thing I've done in my career was automating some version of that. Now we don't have to do so with if-then statements, we can do so with stochastic systems, which to me is freeing. There are new things I can try to do.

The next kind is a ruminative agent. I think of this as, what if we applied long-running reasoning to a bunch of exceptions that the worker agents found and then tried to think about something in different ways? You can literally take an agent that's being given different perspectives on how to solve this problem, maybe one oriented towards security, maybe one oriented towards pure cost efficiency, maybe one oriented towards solving for latency. You could have some that are focused on each of those kinds of outcomes you want and actually set up something called a chain of debate between them. Almost like set up a group of agents that could then come to some sort of conclusion after thinking all night. The best thing about this AI is it doesn't have to sit behind a chatbot. In fact, it's probably the least interesting thing I've seen in the last year is yet another chatbot.

I'm tired of it. What I love are things like ruminative agents that can run in a loop and can do new kinds of analysis that I didn't necessarily program the system to do, but I get some value from. Maybe I can figure out a way to connect power fluctuation issues that are happening with GPUs with certain kinds of failure that you don't notice until you run these things at scale, that you can't possibly test for in a factory. There are things like that. Everybody that's solved a SEV-0 incident on some big platform has probably run into some new novel way they would fail. What if you could start to run simulations and start to find some of that stuff in advance so you can be a little bit more proactive about solving those kinds of problems? That's what to me a ruminative agent does is it looks at all that data, figures out new patterns that meet some criteria you deeply care about. You can have as many of them as you can pay for inference for. I work for NVIDIA. I'm giving you new ideas about things that take more inference.

Since we're on the org structure theme, middle manager agents. Please solve for x. This isn't somebody trying to think of new ideas. This is your middle manager that has to take other people's ideas, narrow them down, and figure out a way to goal orient those things within a function that I might have. I might have, latency dot, dot, dot. I care about this customer response time. What are the top three or four or five things I should do to solve that? In fact, why don't you recommend some actions that would actually take all these different perspectives and allow me to either send Jira tickets to the right human to go solve something or figure out a way to have a system figure out a new Grafana dashboard I might need to be able to monitor something. There's all sorts of ways you can use this pattern.

It's this goal-oriented behavior that you give it. You give it a few different tools they you can use to achieve that goal. The agent mostly just uses those tools that you give it. As you start to tune it, you start to think, what if I add this tool? What if I remove this tool? Or this skill, if you're using Claude. There's new forms of this coming up every day. What is it that you do here? Might be the consultant agent. The consultant agent is a little bit more passive. It looks at the patterns of conversations. This might be some agent that might be just observing, finding optimization about how the agents themselves run. It might be looking at patterns of communication where, you know what? I noticed this agent is solving this problem in the same way, or coming to this conclusion in the same way.

Maybe there's a way to delegate this to determinism so I can optimize it. You might have things that look at a larger set of possible outcomes and say, if we wanted to achieve this other thing, how would we maybe simulate it? A consultant agent might work with certain kinds of simulation models or other things like that, that will then just try to recommend other kinds of meta-actions overall, over the entire system.

You have other kinds of agents, tool selector agents. Now, these, I think, often get embedded into the scaffolding of a lot of our new MCP clients. I think almost everybody building any kind of, what I would call compound LLM system at scale, is going to need something like this. It takes a description of what needs to be done, and then figures out what is the right tool, and how do you call that right tool to do the right thing. The earliest version of this, you actually saw in one of our first slides where we talked about the ChatGPT plugin system, which was a tool selection system. Because any question you would ask it, it would figure out the right GPT plugin to answer that question, or the right sequence of plugins to answer that question. Before we really used the term agent a lot, it was probably the first really great tool selection agent system that we had. I think anybody building a compound system in AI is going to end up building something like a tool selector agent. Some of those tools may not be other AI systems. It may be other deterministic systems. It may be certainly not all LLM-based systems.

Then the director agent. This is, I think, still a little bit futuristic. Who here is going to give a director agent a goal and just say, go do it? Nobody is doing that. We're all here. That sounds a little bit audacious. I think what's going to start to happen, though, is you're going to start to see this happen over smaller domains, constrained domains. People are going to start simulating this. They're going to hopefully get better answers then. I think Anthropic tried to have this run their vending machine to hilarious results. Did not make much money in the vending machine. I don't see that as an AI problem. I see that as a design problem. If their future depended on that vending machine working, I'll bet you they would get it to work. It was just an experiment, see what this can do. If you take this more seriously, and you think about how do you recommend actions in an OODA loop, an Observe, Orient, Decide, Act loop, which is really what a director agent would specialize in, what you're going to start to do is say, I'm going to give you a constrained problem.

I'm going to give you these ways to think about how you think about your actions. I'm going to start with something that's a simple metric. How do we reduce latency in these systems? We're going to start to have a system that understands latency a little bit better, understands the kinds of systems that have that latency, and might start to spot issues, and start to make recommendations to people. As you start to get more confidence in that system, as your evals get better and better, you're going to start to see these kinds of systems evolve into more closed-loop systems. Nobody, I think, really believes a lot of people are doing this right now, particularly effectively. From a future-facing perspective, I see this as absolutely something we will be doing in the not-too-distant future.

Accuracy and Hallucinations

How do you make all this accurate? What about hallucinations? Are hallucinations a problem for you? Of course, yes. It's called lying. It happens to be also a problem with people, too. People sometimes hallucinate that this idea was good, and then maybe it wasn't. It's a uniquely human problem. One of the first things I remember, this would be on LinkedIn, I think, which is, somebody will say, no, AI is just a stochastic parrot. It can never count the number of r's properly in strawberry. Who here has ever seen a post like this before? You don't go on LinkedIn enough, apparently. It's like every day this happens. I always wonder, why are people so focused on the number of r's in strawberry? Seems like an odd thing to think about, when the reality is, have it drop to Python and count the r's. You could put this in your system prompt since at least two years ago.

In ChatGPT, and I think almost every modern LLM system has something like this. If your problem involves doing math or numbers, please delegate to Python and write Python to figure this out. It has been getting the right number of r's in strawberry, too, for at least two years, that way. It's almost like a troll. You can just say, no, I know how to do it. Just do this. To me, that was the earliest indicator of this kind of divergence between we have people that expect deterministic results. We have things that, through errors of how LLMs tokenize or through other reasons, necessarily the system can't reason to that yet. I'm glad that criticism happened, because now we have models that have reasoning inside the model that actually get this right. It was a good way to make it so that the biggest LLM providers would figure out a way to solve that problem.

At the end of the day, we've had a practical solution to this problem for as long as, I think, GPT-5, which finally solved this, was able to solve it, or was being built. Other things like this. Who here does long division in their head? Who here knows what long division is? That's how they taught us math in the early days. You wrote it down and did this thing. Of course not, use a calculator. If you're doing a hard math problem, use a calculator. We've known this for years. I knew this when I was using my TI-85, secretly in a calculus class back in 1991. We all saw the problem with Delta Airlines. One of the airlines put in one of these chatbot systems and suddenly somebody tricked the AI into giving it free first-class flights. They, through prompt engineering, got it to give refunds that were wildly out of bounds and everybody went, AI doesn't work.

No, you can give it guardrails. You can say, I'm glad the AI reasoned to this $5,000 refund for your flight to Oakland from San Francisco. We even have that flight, maybe. Does that flight cost $5,000? No. If you had somebody that was an intern that you put on the customer service team, would you give them an ability to write a check for $5,000 without running it by anybody first? No, of course you wouldn't do that. You would govern them with humans. At first, you should be governing these things with humans that are doing anything important. Human in the loop is such a cliche, I almost don't even want to talk about it because it obviously is appropriate in a lot of contexts, particularly things that do anything with money and the equivalent of money, which is like moving GPUs back and forth, which costs a lot of money, or allocating headcount, as we were doing earlier today, or anything else that's important.

Of course, you're going to have human review on it. I don't want humans not reviewing code going into GitHub, for Pete's sake. Of course, we're not going to just give away refunds that are larger than X without a human involved. We're going to be having that consultant agent or some other kind of agent look at the stream of things that are happening to early identify and discover those things that might be problematic.

Who here has ever done ops? Who here has ever been on an incident call? When you're fixing the DNS server, do you always have to remember how you did it? Do you have to reason through from first principles how you're going to reboot your Kubernetes cluster? Is that how you do it? No. You use a runbook to do it. Nobody is saying would reinvent how to do something new every single time. We don't learn that way. We have one thing that works really well and then we write it in a SharePoint document and then we forget it and do it again. That's how we do it. People talk about ground truth. Have you ever looked at your Confluence? How many hallucinations are in your Confluence? How many hallucinations are in your SharePoint? If you want to talk about hallucinations, let's talk to Oracle, like what's in the database.

I think there's plenty of hallucinations in there that an LLM will think is true because set it in the database, it makes it true. One of the things we do in NVIDIA is we build these things called blueprints. This is a NIM blueprint. We use the NIM microservice, but it's really not the same thing. It's an NVIDIA Inference Microservice. Because we want people to use more AI, we want to share patterns around with everybody else about how to do this stuff, one of the things we built was what we call a portable deep research assistant or what we call AIQ. It's a research assistant. Who here has ever used deep research with one of the main labs products, like OpenAI, or Perplexity, or something like that? I do genealogy with mine. It works really well. I don't know if that's a typical use case. It certainly isn't mine.

I quite like it for that. Sometimes they have it do more complex things, but sometimes, and I'm probably not alone here, I don't want to give deep research, as much as I trust OpenAI and Anthropic and all the people that build it, I don't want to give them access to my most precious data, or my most precious GitHub repos, or my most precious documentation. What's becoming obvious to me, and I think to a lot of people, is that you're going to need a version of deep research that can read your database, that can read your repository of hopefully vetted SharePoint documents that you've actually known to be correct or your wiki that you know to be correct. You want to give it access to those things. You want to have this thing run in a loop where it can reason, it can use our Nemotron model, it can use any external model, doesn't really matter.

We like the Nemotron one, but you can use any external model and then put it in a loop and have it generate, build on the idea and ground it every now and then. One of the things that we found, I think anybody that's built a deep research system finds this out, is there is a tradeoff that you're going to make between how often do you ground whatever premises that your given agent has with the reality on the ground from, might check a database, might check a document. It might look at its sources again. It might search the internet for sources. What we found is the dial you can turn is going to be from grounding a lot, which is going to slow it down, might make the analysis more complicated versus not grounding at all, which means it'll just reason, reason, reason, and suddenly you'll think AI for your reorg is a good idea.

That's one of the things, like how frequently you manage that grounding process is really important and really central to how one of these systems work. Deep research doesn't have to be a product on its own, it can be part of any product that you might ever imagine if you think this idea of just doing extended inference and extended reasoning might be something useful to do.

In my mind, one of the most effective things that you'll do when building these agents is give them access to data that you believe in, data that you've vetted. One of my former colleagues, Zhamak Dehghani builds data products at her startup. I think of data products as these things that we've not just dumped data somewhere, but data that's been vetted, data that's been organized, data that people have a lot of high confidence in. Those are the best kinds of things to use AI agents against because you actually have a lot more effort in just making sure that the facts are true in that database. Accurate data, governed by effective guardrails. This sometimes is system-level stuff. The identity an agent is running as should have substantially less rights to the system that you as a user have. There's a lot of people that are building these agentic systems, and you'll see it even with people using things like Claude Desktop or other kinds of systems that allow you much more privileged access than you would have otherwise to different resources, run as the person running the query, so the query runs as you.

I actually don't think that's good enough. I think what you need to do is create identities that are like yours, but have rights that are substantially more constrained than what you would give even an intern in a system. There's no reason why you couldn't build a series of AI agents that have very specific access. You can read this page, or this database, or this thing, and that's your contribution. The nice thing about AI agents, they don't want a promotion. They don't worry about do I have enough scope. They are happy to do their very specific thing over and over again as long as you want, and just learn more about that thing, and get better over time. Agents are the first kind of software that if you build the right reinforcement loops into it, should get better through use if they're getting usage. I think that's one of the reasons why we think about effective guardrails in these well-designed feedback loops, so that it can, in fact, meet that promise that all these worldly valuations are based on, which is it being the first kind of software that once you install it, it gets better every day that you use it.

These feedback loops have to be guided by humans who have taste and judgment to arrange and to solve useful problems. I don't buy this unemployment line at all. I think it's an excuse. What I believe is, if you think of making pottery, this is the best example I can think of. When you're making pottery, you're making pottery. You have a machine around you that's taking that lump of clay, and you're shaping it. You're using your taste as to what a good pottery project would be. I don't know anything about pottery, so I'm making this up, but you shape it. You steer it in the right direction. Your taste, your understanding, your customer empathy, your understanding of what it is to be built, combined with your technical knowledge about what's possible with the technologies we have today, and what good software engineering looks like is making the latency right.

Understanding what happens when you have a larger system at scale, how is it going to manage all these things? That's not necessarily things AI does well yet, where your taste and intuition as an engineer are going to matter far more for the foreseeable future. At least that's the way I see it. Not everybody agrees, but that's where my mind is at.

AI Agents and Tools

I also see, and this is where I talk about platforms a lot, and that is, you're going to have a lot of AI agents, and a lot of them are going to be able to be wrong. A lot of them are going to be completely wrong about a lot of things. I have eight interns a year that join my team. They're usually smarter than the agents, but, also, I allow them to be wrong sometimes. I allow them to hallucinate sometimes. I allow them to have ideas that are a little bit outside of the bounds of what normally our org does. The reason I do that is sometimes, they might be wrong eight times. Sometimes, you get an intern that builds the Tesseract model, which is what we offer today, to do anomaly detection. It's not even an LLM, it's a new foundation model. Sometimes that allowance to be wrong allows you to discover new ways to be right that are outside of what you would normally consider the right answer in an if-then constrained world of tools.

To contrast that, on the tool side, if I'm doing financial transactions, if I'm doing movements of people across an organization in a way that impacts their lives, if I'm doing anything where I'm moving around GPUs where that move might affect tens of millions of dollars of inference being able to be used by our researchers that are trying to build the next model, I have to be careful about that. In fact, those are systems where I want it to be a deterministic system programmed by somebody who cares about it, deeply understands the nuances of how it works, understands where it fails, in ways that AI agents, I could probably do it with enough tokens, but humans are still better at that intuition thing. They're better at figuring out, fast-twitch mentally, what is the right thing to do. That's where I want tools that do the thing I told it to do.

When I have a tool that inserts money into my bank account, I don't want that to be AI at all. I'm not going to probably ever want that to be AI. We don't need it to be that. We don't need that. Just give it access to the tool. Insufficient tool access can be dangerous. That's really where the tradeoffs are.

Who here has ever heard somebody say, AI agents don't work? I rode one of those. That is an AI agent. A Waymo is an AI that picks you up at your house and takes you to the bar, or takes you to the conference, or takes you to something. It interprets lots of situations. It's been programmed to interpret lots of situations. The fact that they're thinking about having one go to the airport. Have you ever tried to pick somebody up at the airport? There's a lot of uncertainty there. There's a lot of discovery there. Specifically, discovering people aren't very good at driving or parking in a coherent manner. I've been in one of these. It navigates. Not quite the airport yet. I haven't ridden that one yet. I've seen this have to go through a parade. It did that effectively. It nudged its way through the intersection little by little, like a human would do.

To me, this is like one of the biggest marvels of engineering, of agentic software engineering I've ever seen. Whenever somebody tells me AI agents don't work, I book a Waymo just for fun. I just drive around the city, listen to some podcasts, listen to Gary Mark. I'll listen to a lot of that stuff. To me, this is proof AI agents work. We're not going to build self-driving cars. I'll be happy to have a self-driving thing that figures out when GPUs are too hot and lets us know. If we can do a self-driving car, we can do that. We just have to combine these things in the right way and not fall into a trap I think a lot of us are falling into the industry, where, remember like monoliths versus microservices? You've seen this debate at QCon probably 90 times. There's a similar thing in the model community where people want one big monolithic model that solves all things, that has all the parameters and just, AGI.

Then I think of models that are composed in multiple models, as you've seen through the rest of this talk, that are, ok, we have one model good at this, one model good at this, one model good at this. It's like microservices learn how to do one thing extraordinarily well and do that super well. We found that that works for the same reason it works with microservices, but also has some of the same complexities that microservices have. There are some tradeoffs. You don't want to just go down to the most atomic level possible, but it is going to be a combination of models and modeling techniques that solves most of this stuff.

When doing a talk like this, I want to at least communicate the way I think about taste. I don't know a good definition of what taste is, but I know it when I see it. What kinds of problems are good agent problems? How do we have good taste about these things? One technique I like to think about is what I call finding the dumb diamonds. These are things where, is the cover sheet on the TPS report properly filled out? Who here has been in management and had to prove dumb things? I always think of that Homer Simpson thing that has like just on the yes button and just clicks the yes button. There's a bunch of stuff like that in life where the human has to do it for procedural reasons, but the human is doing zero checking. They are checking the box. They are ministerially approving the expense reports that are under $300 because it is not worth my time to go through your $300 expense report to vet your pack of gum that you might have bought illicitly.

Sorry, if you did that and I audit that afterwards, I'm like, pay for your own gum. You can afford it. There is a lot of stuff that happens in companies that are dumb diamonds. Classifiers, any workflow where you can sample for correctness with RL and improve the model. Content organizers, we do this thing where we use a prompt to rewrite content from aggregate emails, meeting notes, all these things, and we build a wiki out of it. We have scaled inspectors, similar to what I talked about earlier where I look at everything with a given point of view. We have constraint navigators. I know this will be the last type, and this doesn't necessarily involve an LLM on the other end. With the constraint navigators, what we're often doing is we're using a different technique, like an AlphaGo-like system that can look at statistically what is the best move out of a non-searchable search space to figure out how do we do things like take all of our GPUs of different sizes and arrange them in the right way. That's not necessarily an LLM problem. Even reorg problems turn out to be better suited for constraint navigation.

Conclusion

Small composable skills that do one thing well, that's how you get started. Start small, grow bigger over time. Determinism is good. Start with dumb agents, they'll save you time. You're not making the entire company do it at once. That's good. Lower the political bit of it. They're defined what they enable you to discover. Do it bottom up, rare context. Use evals. Measure what they do, and if they don't work out well, admit it. Design a system to get better over time with valid feedback loops. Really easy. The world will belong to those with the wildest imaginations because that's the one thing AI can't do yet really well is that imagination thing.

See more presentations with transcripts