Presentation: The Multi-Agent Approach: Building Reliable and Controllable Software Development Automation

Transcript

Itamar Friedman: Maybe I'll do the opening a bit, asking questions. I think that my first question is going to be quite straightforward, who uses at least one AI dev tool? Who uses two? Three? Then four? Take a minute to think about it. If you're using Figma and it's spitting code, that counts. If you're using Jira and it helps you with the planning, that counts. If you're using an SRE agent that helps you with that, that counts. I'm trying again. Who uses three and more? Now I'm going to ask the last one. It's a bit tricky. Within half a year to a year, do you think you're going to use more than three tools that are going to generate code for you?

I'm Itamar Friedman, the CEO and co-founder of Qodo. It stands for Quality of Development. Today I'm going to talk about a topic that I believe is not surprising for you, the multi-agent approach for software development. I did see, like in previous talks, people talking about, for example, closing the loop. There were multi-agents there, or workflows, if you like. I'm not going to talk about the difference between agentic and workflow, but all of them are basically right now using LLM. Now I'm going to cover how to make it more reliable, controllable.

Motivation

First, I want to give some motivation. In the last couple of months, we had more than three outages in the cloud. I'm not saying it's related, but just for a minute, let's reflect on that. Some of those companies that had those outages are saying that 10% of their code is generated by AI, now 30%, and some of them are even claiming 50%. That's quite a lot. I don't have a direct connection, but think about it. There could be a world where people are starting to vibe stuff, a lot of things. Sixty-five percent of developers say that at least a quarter of their code is generated by AI. Some say that above 80% is AI influenced: you chat about it, you ask about it. That's a really big change in paradigm. People are using tools not only for code generation, but also to do vibe coding, also to do vibe reviewing.

There was a lot of buzz, I won't say hype, about Anthropic security reviewer. They actually published their prompt. From now on, you are a senior security engineer. Good luck. Then there is actually an exclusion list, which includes DDoS. Maybe it is related to the cloud, maybe somebody vibe reviewed it. The exclusion list is very long. One more thing too, I keep building the motivation here about the multi-agent system. I'll connect the dots later. I believe that most of you are using Cursor or similar tools. We did a survey where we asked people, once you wrote detailed rules or instruction, Markdown, do you think Cursor is following your rules? We asked from completely followed to just never followed. It was 100% on the B, C, D, like, it followed but I'm not sure. Sometimes it follows that, sometimes it follows this. Let me know if the following is familiar to you. You take a prompt, a proper prompt, with rules, and then you try that seven times, you get seven different implementations. Even if you have the rules. You see where I'm going?

Outline

In today's talk, I'm going to talk about mainly topic number two that I'm going to get to. I want to spend time giving you additional motivation, why it's so important to have the multi-agentic system, where a big portion of your investment and efforts are actually on the verification part and context engine, agent. I do want to ground it with some reports. Some of them are from Qodo, but also from Sonar and Faros. The main thing I want to talk about is the multi-agent system, where quality workflows is built in, and specifically continuous learning. I'm going to talk about different paradigms, as well as concrete examples from production.

I want to start with one of the takeaways. It is just a simple graph, but it is part of my takeaway. Basically, we saw the first generation of AI for dev tool, which is autocomplete. Then we saw the second generation, which is basically agentic code generation, where Cursor and Claude Code came into life. Now you're seeing on X, a lot of people talking about code review and code quality workflow, because that's the next ceiling we need to break through to get there, to really automate the Software Development Life Cycle. Then I'm claiming that there's still a glass ceiling there, because basically each one of your software, each one of your workflows is first of all different, even if you're using somewhat similar tools. I think it's always evolving. Would you chase those workflows? Would you chase those changes? Probably you will, but eventually to break through the glass ceiling, we need to have an adaptive quality grounded multi-agent system, and that's what we're going to talk about today. Somewhere in the middle, I'm going to talk about a paper that was released by Google. They claim that a multi-agent system is plus minus bullshit. I'm actually going to talk about that as well, and what to do with it.

Code Quality

A little bit about adoption. Some of the reports that I mentioned, mentioned that 82% developers are using daily, or at least weekly, AI dev tools, especially code generation. Three or more tools are being used by 59% of the developers, and 20% use more than 5, which is roughly I think what I saw here. This is actually in contrast with 28% actually claiming that they don't trust the code. Maybe it's not like that in conflict, talk about it. Basically, a lot of the adoption is mostly in small companies. I am familiar with a few places, and I know that it's being used also in Fortune 500 companies. The massive adoption is a lot in the smaller companies. The question is, how do we mature it for enterprise as well? I'm repeating because I think it's important. Basically, now I'm combining all the reports together. Before it was one report.

People are reporting continuously strong adoption of code generation, but at the same time they're seeing mostly productivity around writing more code, not around getting shit done, like around writing more code, not necessarily end-to-end software development and velocity. One of the reasons for that is because they're not necessarily trusting the quality. You're basically moving the needle, moving the bottleneck from one place to another. Let me elaborate on that. This is from one of the reports. Basically, there's more tasks that are being done. Mostly more PRs are being opened. It's correlated but not one-to-one. It takes much more time to review a PR and actually deal with the bugs and the issues. At least that's what they say in the report. Why is that? It's because code generation is magnificent. It's really changed how we work. I think mostly it's true, that's what we're seeing in the report, in field projects as well as POVs within companies.

Let me show you an example. Cursor just released their tool where you can open a web app. This is specifically a Qodo side project. Forget about the project itself. It's just a ChatGPT-like. I'll ask, are you developing a ChatGPT-like, building that in the company? We're building to get direct access to our context engine. What you're seeing here is one of our developers using the latest feature by Cursor, where instead of going to Figma or whatever, or making the changes in the code, directly making the changes on the web app and then getting the changes from Cursor. No code was written by that developer. That is magnificent. When you're dealing with heavy-duty software, even if on the surface that code and that feature was very simple. All the changes we asked it to do, if you notice, is like changing text, moving a button from one place to another, and we wanted to remove a field.

It was very simple. It did it pretty well. We're going to see in a minute. Eventually, when you're dealing with heavy-duty software, and I don't need to tell you, you need to deal with maintainability, reliability, basically code quality and code governance to some extent. Basically, the quality issues that can come from using those amazing features of code generation, they have multiple dimensions. I'm going to talk about the dimension of the SDLC. There could be problems in planning. There could be problems in development. There could be problems in the review process. There could be problems in the testing, in the deployment. That's one dimension. Sorry that I'm going to put a lot of information about the quality because that's the purpose of this talk. Here's another dimension. I know I'm going to raise a lot of issues that can come from code generation. I'm here to actually think together how can we use a multi-agent system and actually solve or mitigate the problem of quality.

We need to understand the quality problem. One dimension was more process-oriented, and that's mostly what I talked about. I put everything I talked previously mostly on the process, like learning, verification, guardrail, standards. On the code level, we have functional and non-functional, and there's security and a long list of poor error handling and things like that. These are the dimensions of quality you need to think of.

When in those reports, developers were being asked, probably a bit subjective because of the question, what actually is the impact on code quality that you're seeing from AI code generation? Actually, developers said, and there's research about that from METR, that eventually there's 42% developer time actually more spent on fixing bugs. Actually, they do see 65% maybe projects going faster or at the same speed, but 35% of their project delays and 3x security incidents. If you think about it, we just said that AI code generation tools are helping you to triple the amount of code. AI, roughly speaking, is generating the same amount of issues per line of code. It makes sense. These are the dimensions of quality. The question is, what are we going to do? Because we do want to use AI for code generation. We do want to expedite our development. I'm going to start showing you a few examples.

Through these examples, basically I want to show you that there's a lot of specific agents that you can use, that we can combine together. That will lead us to talk about how do we combine them together. What are the paradigms to combine them together? The first suspect if you want to increase quality is actually testing. I like to look at tests as executable specs. I like this format. I do want to tell you, when we asked people, do you use AI for testing or not? Then we slice that to see their response. Those that are actually using AI for testing are reporting much better productivity on AI code generation. I know it might sound weird. Like, how can my AI test my code? If it's wrong in the code, it would be wrong in the test. Don't think about it this way. Think about you as a human.

Like, if you are expending time on tokens and thinking process on testing, eventually it will help you write your code. Even if your tests fail, you are thinking about that test. Did it fail because of the test, because of the code? You're spending tokens and eventually improving them both. What we're seeing is that actually really great improvement. The people that were in the report said that we sliced them, gaining much more of code generation when they're doing testing. I think the first suspect of testing is actually unit testing. That's the first suspect, because also the code generation tools are relatively good in that. Then the next suspect is probably integration testing. Although that is quite complicated, and you need a really strong context engine. I want to show you maybe not a trivial example. The technology is maturing as we speak. For the same example of Cursor changing a chatbot, what you're seeing here is the end of it, where the developer finished a pull request, reviewed it quickly, and opened a PR.

Here you're seeing a tool, a tool that is actually using computer use, and checking different aspects. You're seeing a video that is running on staging or localhost, of a tool that all it's in charge is verifying that things didn't break. That could be quite elaborative. For example, one of the things that this tool is checking is that we didn't break elements that are actually supposedly not touched by Cursor. What happened, although that was a very simple request, actually one of the options in the chat was actually ruined. It was surfaced by that agent, that testing agent. This technology is maturing. I'll go back to what kind of mindset you need to use that.

The second suspect are agents that are specific for code review. Why do we do code review? Roughly speaking, two buckets. One, we want to verify quality. Like it's according to our standards, no bugs, according to the intent. It's a bucket. The second thing is we are there to own the code together as a team. Learning and owning. If there's going to be a problem in the code in production, and that developer that was the individual contributor, eventually, if he or she is on vacation, somebody else is going to fix that. That's why we own it as a team. That's why I'm saying, although it's a bit nuanced, that the purpose of code review is a gateway and a gatekeeper. A gateway, I mean like where we share information, we learn together, and a gatekeeper, where we improve and govern code level and process level quality issues.

When you're thinking about your AI for code review, these are what you need to think. How do I increase the automation of catching those bugs and whatever? How do I help with owning the code together as a team? That's why it's gateway and gatekeeper. From the report, I want to tell you, we also asked people, do you use AI for code review, no matter which? We sliced also the information about that. We saw that developers and teams that are using AI for code review are actually claiming for 2x better code quality. They're claiming for 47% more productivity in the code writing, actually, when you're using code review, because there is some kind of a loop here that helps. That's part of what we're going to talk about next. Specifically, Qodo, that's one of our flagship applications, the code review. One of the things we notice when checking millions of PRs, is that 70% of them, including an issue of high severity.

Think about that? It's not every second, but every fifth could have a high severity issue, like breaking change or something that you would consider high or critical. As you add an AI code generation tool that helps you add more and more PRs, and write more and more code, this could increase in quantity. The second thing, and what's interesting is that we're seeing a lot of code reviews with no human review there. That's really interesting. I think there's two reasons. First of all, only 60% of teams are actually talking about their review in GitHub, GitLab, Bitbucket. Some take that to Teams or Slack. You might be in that 40%. That's a big chunk. Also, there is a trend that is happening, which is funny and sad. I think you saw this meme, but bear with me, pretend something visual. You're waking up in the morning. You're a tech lead.

You have 40 minutes to review. Familiar? You open a PR. If you see 5 lines of code, you're going to give 50 comments. If you see 50 lines of code, you're going to give 5 comments. If you see 500 lines of code, looks good to me, and move on. This is funny, but also sad. Now think about what we just said, Cursor, you give it a prompt, in 5 minutes, you get 1,000 lines of code. Claude Code, even with that, like five Markdowns with more text to read. How do you deal with all that? We're seeing a trend, like 80% no human review. What can you do with that? Basically, one of the things that you need to nail it are, what are your best practices and standards? What are your quality standards? What's inside your tech lead's heads that you want the AI tool to also review?

You can find tools out there, this is specifically Qodo, where you can suggest pair repo, pair organizations, like rules, including mentioning good examples and bad examples of code. Once you set up that rule, that best practice, then when there's going to be a violation of that rule or best practice in the code, that will be surfaced in the PR. This is actually meaning customizing. The AI agent, not only for general code review, according to what it saw on GitHub, according to your training, but actually being specific with customized compliance. This could be like compliance, just like complying to your rules. This could be, for example, just an if statement. If I ask, what's your best practice about if statements? Some will say one thing, and another group will say another thing, an LLM wouldn't know unless you let it know. You need to invest in letting the AI code review know what are your rules and best practices, and that's where the AI code review agents become really powerful.

The AI code review agents are very different than the code generation agents. We just talked about a few minutes ago that you give rules to the AI code generation tools, and they just sometimes use it, sometimes not. The AI code review agents, they take each one of your rules and treat it as must be checked or not. We're going to talk about it a bit later. We're going to talk about architecture, and it's a hint to talk about.

AI Quality - Foundation: Context Engine

Last thing, I'm actually going to talk about context engine, but I just said that I'm going to talk about different agents and how they're different than each other, so when we're seeing the full architecture, we understand why we have a multi-agent. What's the thing with the context engine? Once upon a time a context engine was basically a RAG, Retrieval-Augmented Generation. Today it's actually an agent that is specializing in fetching context. I'm going to talk a little bit about that. When we ask developers, when you have a problem in your code, when you don't trust your code, when it's hallucinating, why do you think it's happening? Eighty-eight percent think it's a context issue. The most requested item from developers, like, what do you want to be developed? Is making those AI tools to improve their context engine. That's the number one. It's 33%, but it's the first from the list of requests from coding agents or AI dev tools, what they're being asked to improve.

Specifically, really interestingly, when we reviewed Qodo on AI Code Review, we saw that 60% of the tool usage of Qodo is for fetching relevant context. Just to back it up, not from the developer's point of view, but actually from the builder's, we are noticing that agents, what they need the most, and also in practice, the tool that they're using the most are tools that are related to bringing the relevant context. I thought that's interesting. Although it's only 8%, but it's actually quite a lot if you think about it. Eight percent of the context fetching is not related to code. It could be related to fetching the company's ARCHITECTURE.md or the rules and best_practices.md, or, in case of Qodo, from a database. Not a small portion is not just from the code itself, it's context from additional sources. There is something about context that you guys need to deal with and making sure it works for you, if you want to make sure that the multi-agent system is actually working.

Just to give you an example, this is roughly speaking, how our context engine at Qodo looks like. You can find it on the internet. In this case, Jensen gave us a shout-out on his GTC keynote. I'm not going into details right now. I just want to show you that it's an agent by itself. Everything there is ingestion agent and queue agents, to retrieval agent, and all of them are agents. It's not any more RAG. It's an agent by itself. This agent needs to know how to fetch your code, different versions, your PR history. Those that do talk inside the PR, actually, there's enough golden nuggets there. One tech lead told another tech lead or a junior, this is not how we do things in our company. Forget about what you learned before, how many times you got that review or so. Then there is the organization rule and the docs and the logs, all of those, what you guys as developers are using when you're doing a review. The context engine is really like a massive agent there.

Recap - Exploiting AI for Quality (The Path Forward)

I'm zooming out for a second, because I'm about to change to the main subject, how to connect all those agents together. What is the paradigm? Just doing a recap, so far, we talked about automated quality gates. Intelligent code review. AI-generated tests. Also, agentic context engine. Somebody wrote to me on social, look at these guys, like they're re-branding CI/CD, like agentic quality gate. It's basically CI/CD. I'm not claiming not, but I think that basically on one hand you can say, yes, it's like putting LLMs and agents into the CI/CD. I think part of the opportunity here is to break the flow of how we're doing software. It was very one way, from left to right, or if you like on the infinity loop of CI/CD, and maybe we can make it different. That's part of what we need to think about when we're thinking about a multi-agent system.

I'm not saying not. Again, I'm doing like a bit of a stop here before I jump into the multi-agent system, just to recap what I wish you take from here until now. One thing is that quality is a competitive edge, like if you invest on quality workflows, and you'll need to invest in order to do that. If you remember my graph, the x-axis was invest, and the y-axis is how much you gain for it. If you invest in agentic workflows, that will be very competitive, because that's how you do code generation with more confidence. AI is the tool, it's not the solution. Sometimes it feels like, yes, we got this one Apple product, and some of you raised that you're using only one tool. Eventually, don't treat it as like AI is the solution for software development, rather AI is a tool for you that you need to put in your software development, and actually put it in many places.

That's what we're going to talk about next, the multi-agent system. Also, you don't need to do everything all at once. I'm going to show you a system, like an architecture, a full system, you should not try everything all at once. What I suggest to you is that you look on your bottlenecks right now, and put AI into work on every part of your SDLC. Fortunately, I think you can do it consequential, you don't need to do it all at once. Just like so far, these are the agents we talked about. Start tomorrow, one by one. We did see in the reports, from all the three reports I mentioned, that people are gaining a lot when doing that. I saw some places in the reports that code review became slower, actually, and now it's going back to be a faster reduction in security vulnerabilities. A lot of people reporting that to get to 80% test coverage from a certain point where it takes three months, now they're seeing it happening in one month, just an example. 3x test coverage, when you implement some of the things I mentioned.

Multi-Agent System Architecture

Finally, I'm going to start talking about the multi-agent system architecture. This is how I think right now, if you put everything we're seeing out there, could look like. Basically, you have agents that are helping you with writing your specs. You have agents that are helping you do tests. You have agents that are helping you with code implementation, because in the future, maybe implementation is not only code, but let's not get to that. Then, we did talk about a RAG solution and context engine, I'm calling it a software DB, because it might include rules. We're talking about tools. You should invest in tools related to verification. Eventually you want to enable the agents to run, because sometimes only when you run software, you actually bump into problems. You need to build secured environments for those agents to run. This is a very holistic approach. It doesn't say a lot of things like how these communicate, how they pass information to each other.

That's a little bit what we're going to talk about. We saw a paper by DeepMind, where they call what we talked about, like MAS. Basically, they were checking different architectures, some property, some dimension of decisions you need to make in the architectures. Then they tested multiple scenarios, financial, software. Roughly speaking, if you see on the fifth line, MAS, they checked centralized, decentralized, independent, hybrid. It's mostly about who is controlling the process. That's the main dimension they checked. They basically got to a point that there's no advantage. Everything I just told you, I'm really sorry. You just need to check one agent sequential, and just prompt it differently. I'm kidding, at least partially, because actually, they checked different types of systems and use cases, and they got to interesting conclusions. It's part of what I'm going to share with you now. How do you think about multi-agent systems?

What should you consider? What are you trying to achieve? We need to align on that. I talked for 30 hours now about quality problems. Just to make sure that we're aligned, what are we trying to achieve? Basically, what we're trying to achieve is moving from vibe coding, but that's because that's all what you're doing every day, to something that we call viable coding. The focus of Vibe-DD, Vibe Driven Development, is flow. We love staying in the flow. That was so exciting. Instead of checking correctness, which is TDD, instead of checking behavior, which is SpecDD, now the goal is my flow. We want to move to something else. I think we want to keep the flow in the center. Basically, we want to take the context and make sure it's properly planned. I'm going to show you how. We want to take the flow and make sure that you have a quality workflow, agentic could help.

You want it to happen all the time as you flow, because you want to keep the flow. We said we want to keep the goal of flow, but make it more viable. The last thing, when we're talking about code review, you want to make sure that you have full control of the code review and everything is governed. Eventually, you have a responsible vibe coding that puts more weight on being proactive about rigor, correctness, and ownership. This is the goal, like, just before we discover different systems. Again, I didn't answer what we should consider. I just want to make sure that we have the goal before we talk about it. I think we should consider architecture design, like async versus sync. I'm going to talk a little bit. That's mostly what Google DeepMind checked. How agents communicate with each other, how they share context engine, and the governance.

About the async, sync and what Google checked, basically, this is what they checked. Should we have a super agent, or should we have a super agent that is interacting with other agents, or should we have an agent swarm? The question is, how do we think about that? How do we decide? Putting that diagram according to Google world, DeepMind, it's centralized, decentralized, independent, and hybrid. What they found in that paper is that you need to think how much you can parallelize. How much can you do in parallel that it's independent? What are other multiple interfaces in your solution? Do you require an arbiter? What is your cost consideration? I'm going to put the cost consideration last, although it's very important. Just to look on the first three. My point in the first three hours that I talked before is that we could and should parallelize things.

Eventually, when I'm coding, I want the flow. I want Cursor to quickly do some basic checks and verify that what I requested in the user interface is working well. Then, if you remember our goal, if I'm not going to look on the code and that's what's starting to happen, I want multiple agents to run mini UI workflows, the happy paths, and they can do that in parallel. I can check different paths. The second thing, do you have multiple interfaces? The answer is yes. We have the IDE. We have the terminal. Do we require an arbiter is one of the things I'm going to talk about. This is one of the hardest questions. Let me double click a little bit about that. I'm going to talk about it a little bit later. I'll talk about the arbiter, because that's one of the hardest things.

I'm jumping to the second point, talking about how agents communicate. The very natural thing that we're being pushed to do from all the vibe hype, maybe, is using protocols. Like MCP, A2A, MCP-UI is really cool. I don't know if I used the best term there, but maybe I should have used traditional on the right side. Let them communicate via some Pub/Sub, or some DB, or something like that. There's a third option, which is not mutually exclusive. Basically, I claim that the most important thing is that you want to make sure that you guys, with your eyes, can actually check the communication between the agents. I'm going to show you an example, which is a mix. In this case, you're seeing Qodo giving a suggestion inside your GitHub. It's inside your GitHub. It's recorded in a draft PR. Now, you can define a workflow in Cursor.

You can define a workflow with Cursor, which uses some workflow with the relevant MCPs, for example, Git MCP and Qodo MCP. There is a set of rules that I just mentioned in that workflow that actually motivates Cursor to eventually react to those comments by Qodo and commit them. What it means for you is that when you're coming later to observe these two agents talking to each other, one is focused on code review, another focused on code generation, you will be able to watch what they did there. If you connected via MCP and there are no commits that are happening, or you commit their communication in the logs, then you need in BigQuery to start looking for it. How can you see which agent did what? Once you do it this way, then eventually you are seeing here is Qodo, here is a commit of Qodo main suggestion.

Here is Cursor main response, and everything is on GitHub, or Bitbucket, or whatever. You don't see that in the thinking that happens, but eventually what one agent was responsible, what another agent was responsible, although they communicated with an MCP or so, or a workflow eventually was recorded. That is very important, my strong suggestion about that. Make sure that you define what are the workflows or the protocol or whatever you do to make the framework to communicate between agents that eventually you can have a human UI. Right now, GitHub is one of them, but you can think of others. Like, for example, a Chat, a Claude desktop, whatever, that enables you to review the communication between those agents. Going back to the arbiter, why is this important? Here is some motivation. Let's say that you have two coding agents, and just to give you an example, you might have Claude Code and Cursor, or Cognition and Figma.

A lot of times developers use the same product and launch the same prompt seven times, because some of the implementation will be right or not, so you're choosing among them. Sometimes they're giving different tasks in parallel, not the same one, and then they might be touching the same file. What do you do with that? Then the coding agent not agreeing with the planning agent. Who decides? Then you have a problem with a testing agent not agreeing with a coding agent. When you're thinking about your software development architecture multi-agent system, you need to think who is the arbiter. I want to double click a little bit on that right now. Once upon a time, like I think not just a long time ago, everything was like from left to right, and a lot of the vibe coding solution was focused on planning and writing. Then already now people are realizing, like it didn't take long because it's quite trivial, that this is really only blue team thinking.

When you're enterprise and you want to fix a bug, you have to deal also with testing and review. It's back and forth. Especially you want to refactor, change a feature, new high-quality feature, and you want it according to your standards. The thing is that basically what it evolved to is like doubling down on code review and testing, as part of the process, not afterthought. Basically, like shifting left. Taking the review and the test and trying to push them as much as you can while you're writing your code and planning. It didn't solve the arbitrary. Like, who is deciding? There's a really funny Tweet where somebody wrote, Cursor didn't necessarily know what to do and Qodo butted in. I checked in ChatGPT if it's a good sentiment or not, and like, who is right? Cursor or Qodo are just an example. It's not solving the problem yet.

Let's keep digging. Basically, like if we try to put that flow into x-axis, one is how we usually do software development. I know you guys, some of you are TDD, but most cases this is the flow. The y-axis is executability. Like your plan is not executable, your code is. Your tests are executable, but not part of the program. I think part of what's promising with AI is basically you're shrinking the V and you can do like code review and tests, not only to writing code, but actually to the plan. I think that's one of the game changer moments where when you can do, not only plan and review to the code or also to the plan. That's part of what we do. I want to share like the way of thinking, is you can see that like agent, all it does, it needs to be an arbiter.

Is the code working according to the planning? It's the context and the right workflows that might be different than all the rest. You have to have an arbiter. You can't avoid that. Here's an example of doing that. In order to help the arbiter, you have to have the right context in the planning. What you're seeing here is a developer writing a description for a new ticket. Then you have the arbiter, first of all, checking that this new ticket actually has enough information to do arbitration from the planning agent to the coding agent. You have to have the arbiter having enough information. That arbiter said, we noticed that you didn't even include which repo you're trying to change. Now the developer is actually copy pasting the repo that is about to change. That arbiter might ask for more and more information in order to make sure that there is enough information in the planning, so later on when the planning agent and the coding agent are in misalignment, the arbiter could have enough information to verify that. This is a real example from a real product. You see all the check boxes there, they become green.

If you remember, I had a slide on generation 3.5. I think if you take that architecture and you start thinking about everything we discussed about different agents and all the considerations, eventually the 3.5 was missing something. First of all, governance, observability, controllability needs to be part of it. Different agents might have different credentials. The testing agents might have access to staging. The coding agents might not because its workflow is so creative. Who knows from the staging it will jump to production. I'm half kidding, but half not. You don't want it to delete your staging DB or whatever, even if it's staging. You see it happening a lot. You need to mature this, and if you're thinking about everything, you have governance, observability, controllability. You need to have some learning that's happening in your database. What are your rules? What are your best practices? You have to have agents that are specifically focusing on making sure that your spec is full enough.

Because if you don't have your spec full enough, you cannot have an arbiter, the testing agent or the code review agents to say who is right, the coding agent or the spec agent. That requires some closing the loop and learning of what are your standards, what are your specs, where is your architecture designs. Eventually with that, you would be able to like, 2026, break through the glass ceiling. You probably took a budget because the CEO told you, I want to see 2X, 10X productivity. Now you're reporting back 20% or whatever, and we still want to make it happen. That's my sharing, that quality is a competitive way, and with a system that actually learns what quality means for you.

Questions and Answers

Participant 1: You had some really good statistics in the beginning from a survey that you ran on productivity of what actually engineers achieve today. Also, what is the usage? You mentioned that small, medium businesses use more AI than Fortune 500. How did you go about creating that survey and getting those answers?

Itamar Friedman: First of all, it's three surveys. We took our own, but also Sonar and Faros, then we tried to find commonality. Basically, what we did is reached out to a variety of at least half of our clients. We have a thousand plus, but also not only, because we wanted to have control. We tried to be a variety and we basically stopped collecting when we saw enough diversity. When we saw the numbers, the hundreds, we reached to like 617, we saw that's enough diversity and is big enough. That's also what I saw. I think in other surveys, I saw even a few thousand of developers in their survey.

See more presentations with transcripts