Presentation: From OTEL to SLMs: Distilling Frontier Model Behaviour from Production Telemetry

Transcript

Ben O'Mahony: First thing I want to bring up is this quote. We shape our tools, and thereafter our tools shape us. John Culkin paraphrased Marshall McLuhan with this quote decades ago, but I think it's never been more relevant. When you build an AI tool, you're creating something that will reshape your workflow, your habits, your thinking. The question isn't just, does it work, but actually, does this make you a better person? What kind of person does this tool create? Look at how we work as developers. I use Neovim, so I've spent hours shaping my editor exactly how I want it, hundreds of lines of Lua, lots of plugins, color schemes. Every key binding, I want to shape my tools. Continuing this theme, I built an AI-powered Language Server Protocol. I wanted that to give me semantic code analysis, inline hints and intelligent suggestions. I crafted the harness, the agent, the instructions, the interactions, but I couldn't do much with the AI API.

I could change models if I needed better model quality, but that was it. I couldn't customize the actual brain of this tool and actually improve it over time in ways that were a little bit more permanent than memory. Also, I was burning tokens, as many of us probably do. The LSP works proactively. I'll show some slides of what it looks like later. Every time I save, I'm sending stuff to an AI agent. Costs add up once you're coding all day. I needed the semantic understanding from large language models though. Then I realized something, that I was actually generating a lot of the data that I needed. I was using OpenTelemetry. It was capturing everything, every interaction, every suggestion I accepted, every suggestion I dismissed, every suggestion I asked to regenerate. That dataset could train a smaller, faster model. This talk is about that journey, and it's also about how you could turn that from something that works in a one-off way into something that is more useful as a platform capability, something that you can adapt and use at scale.

Background

I'm Ben O'Mahony, Principal AI Engineer with Thoughtworks. I work in our global AI team, which is really exciting in that I get to collaborate with many of our awesome Thoughtworks colleagues around the world on many different projects. I'm distilling that knowledge down into a book. I'm writing for O'Reilly, "Building AI agent Platforms," with my colleague Fabian. The good news is that the first few chapters will go into early release. Bad news is, as anyone here who tends towards the more mathematical side of things will understand, I've signed up to do a 10,000-word essay every month, and I didn't quite realize that that was exactly what a book was, which of course it is.

Foundations - Language Server Protocol

Firstly, some foundations, just to make sure we're all on the same page. The Language Server Protocol is a backbone of modern editor language-specific smarts. It's why you can use the same Python language features in VS Code, Neovim, Emacs, any other editor. Your editor acts as the client, and it sends a request to a language server, that is a separate process that understands your programming language, not just the text. The server responds with information, whatever you request, whether that what's at this position, and it will come back with things like documentation, it'll come back with go to references, go to definition, that sort of stuff. We all know about it, but we maybe don't all know that it's called an LSP. When you save the file, the server analyzes it, sends back diagnostics, errors, warnings, suggestions. Traditional LSPs like pyright or rust-analyzer are rule-based, so they parse your code into an abstract syntax tree, they check it against language rules, maybe do some type inference.

They're really fast, precise, reliable. They're really useful to have everywhere. They can't really understand semantic intent. They can't tell you the name of this function no longer matches what this function actually does. That's too complex a thing to be rules-based. Or the error handling looks fragile, you're reinventing this standard library function that you should be using. That's where the AI LSP comes in. For me, I wanted AI suggestions where I work. I didn't want that separate chat panel, I wanted it to be proactive, actually agentic. Not just waiting for me to let it be agentic, but actually agentic. One of the benefits of it being AI is that actually it's language-agnostic, because you don't need to parse the abstract syntax tree, you leave that to proper deterministic checkers. This actually can go in and look at any code or even Markdown and suggest improvements. It can also suggest code fixes, so that's been really useful.

This is what it looks like for me. This is a detailed diagnostics of what I actually have. This is very meta. This is the AI LSP suggesting ways to improve the AI LSP codebase. I actually get these issues in my context, not just the context engineering for the agent, but actually this is easily consumable in the way that I'm used to getting linter errors. It doesn't take me out of flow, I can still code, and I've got these issues coming through. As an aside, it's interesting, like we think so much about context engineering for our agents, but it's surprising to me that we don't really think about that for the engineers using these tools quite often. Like a lot of the tools that we get, the outputs are quite difficult to consume, and the UX for how I use a coding CLI, for example, or I just wait and my UX is a big pull request of 14,000 lines.

For me, that's a crap user experience. I want to think of context engineering as like, what context am I given to make decisions about the code, not just within the realm of the agent. As you can see, I've got code actions as well, because as soon as I had diagnostics, I was like, I want code actions. Firstly, it's generated two options I could choose. Again, it's not like just a suggestion of this is the way to fix it. It's got two options. The diagnostics is located where the problem is, but the fix might actually be somewhere else in the code, so it can make those changes elsewhere. I can accept them and see how it looks. I could also combine them. They might be able to be combined. They might not be, but I can compose them potentially. That wasn't good enough, really, because AI generated code?

I wanted, firstly, an option to dismiss the AI suggestions. If I was getting a lot of these, that's going to take me out of the flow. I don't want too much of that. I added a dismiss, and then I also added a regenerate AI suggestion. I agree with the diagnosis, but not with the solution. Those are the code actions. As you can see, I still have deterministic checkers, because they're way better at the things they do, and I don't want any of the AI to touch the stuff that works perfectly.

Observability - OpenTelemetry for AI Agents

Next foundation, OpenTelemetry. It's industry standard for observability. In my opinion, it's just a no-brainer. It gives you distributed tracing, that is the ability to follow a request through your entire system, across services, agents, everything. See exactly what happened in each step. For traditional web apps, trace might show requests, cache checked, response returned, database query executed, that sort of thing. Each step's a span, with timing information, metadata, and context. For generative AI systems, these traces are even more valuable. This is what a span and trace might look like. Again, maybe I'm preaching to the choir here. Everyone's already using this stuff, but a span is one unit of work. A trace is the path of the overall execution throughout your application, and it's composed of these different spans. Next, what does OTel actually have for generative AI? I don't know if any of you guys have looked at the spec.

It's really useful. You've got events, that is inputs and outputs. This is, you get the slides, you link to all these bits and pieces. That can be like system messages, user message, assistant message, tool calls, that sort of thing. You've got metrics, really useful. Token usage, operation duration, request duration, time per output token, time to first token. A lot of things that are really useful for you debugging an application, but also as a platform. Then the model spans and the agent spans, which are composed of the operations. These are those operations. There's already some legacy in there, as you can see, because of how quickly things move. Fundamentally, agent creation, probably not that exciting. Execute workflow, chat completion, multi-modal generation, tool execution, really interesting. Vector, if you're doing RAG. Legacy completion's not that relevant, but those are they.

This is actually quite a boring trace, but this is what an LSP interaction looks like. This is actually Opik. Other tools are available. Several of the sponsors of this thing will handle OpenTelemetry quite well. It should plug in directly into your existing observability stack and just get going from day zero. This is the point, like every agent should produce traces, in my opinion. Instrument them from day one. Don't wait until you're thinking about distillation, just get the telemetry in place. Capture the context, the prompts, all those things, the user action. I think that's the most important part. Even if you never train a model, this stuff's amazing for debugging, understanding how things go. This is exactly what you want. Actually, auto-instrumentation gets you 99% of the way there. All of those things, like all these frameworks have their own auto-instrument method, and I got bored of finding all of these.

There's lots more. Really, there's no excuse. It's very simple. This is what it looks like in the AI LSP. I'm using Logfire, Pydantic AI, so just setting the OTel exporter endpoint, configuring Logfire and auto-instrumenting. This is what actually generates the gold. This is the training signal. This is actually a span that goes over one of the code actions. This is the regenerate_fix one, and that's all you need to add the instrumentation of that additional span and then that will go into all of the rest of your OpenTelemetry tracing. You now have signals from users, and that's gold. In this case, me, I'm labeling my own data just by using the tool. That's the flywheel that you want to get into. This allows us to use RLHF, any other fine-tuning methodology that you like that incorporates human feedback, super useful.

These are the three spans I have. They're behavioral feedback. Accepted when a diagnostic appears, I click apply fix. It's a strong positive signal that that's actually generated something. It's not only found an issue, but I've accepted that this is how I should fix that issue. I'm basically explicitly saying this is helpful. Regenerated is like a weak positive signal. You're on the right track, not quite. I click regenerate to get an alternative suggestion. That means I probably agree with the diagnostic, but I don't necessarily agree with the solution proposed. I'm saying, go on, try again. Dismissed is strong negative feedback, which is also super useful. Actually, these negative examples are really difficult to find in a lot of products. This goes way beyond the thumbs up, thumbs down that you'll get, because this is not what users say they think. It's how a user's actually using your product. That's where you get really strong signal. Like in traditional machine learning, this stuff is so difficult to gather. It's expensive. You spend a lot of time just labeling data or paying someone to label data, and you still have all these issues. Really nice to have this stuff.

That's where things get really interesting. For me, this is building a data flywheel. Every user interaction with your AI system is an opportunity to collect training data. It's what this looks like over time. Day one, I'm just configuring things, getting started. I take it naturally, coding as I normally would. By the end of the day, I've got several interactions. Not enough to train a model, but it's a good start. End of week one, it's up to tens or hundreds of interactions. The dataset's growing, and I'm starting to see patterns. Things are being accepted more frequently, and other things are being dismissed more frequently. By the end of the month, I've got hundreds of interactions that I've all labeled and annotated and stored. Now I can start actually thinking about what I'm going to do fine-tuning-wise. For me, that was with MLX, just locally, fine-tuning this very simply on this stuff. Took a couple of hours. Now I can use this small language model, and the data flywheel keeps on going. Because if the quality isn't there, I can still give it feedback, and I keep that data flywheel going. That's for me what this is all about. A lot of the stuff that we all know.

The mechanics of actually turning that OpenTelemetry traces into training data, this is where the rubber hits the road, so this is a couple of the methods I've got. They're not particularly interesting. First you need to export your OTel traces somewhere where you can process them. Any OTel-compatible backend works fine: Logfire, Honeycomb, Datadog. I ended up using OTel Collector to write the spans to disk in JSON, but also as well forward them onto Opik. That was a local solution that worked nicely for me. You can do it however you like. The technology for this is pretty mature. My extraction pipeline queries full traces where user action spans exist. Filter out the areas where diagnostics was just ignored, because they're not giving me any signal. Looking at the accepted, regenerated, and dismissed. Output's just a simple JSONL file, and each line is just a training example. Nothing fancy, no complex preprocessing.

You've already got this stuff in the format that you want. The hard work was in instrumentation and a little bit of filtering. Now we have a dataset, several thousand examples, hopefully. I was using Gemini, but you can use any of the frontier models to start the distillation. I've labeled those examples positively and negatively through the usage. Distill that into a small language model. Something in that 7 billion, 13 billion parameter range. One that already knows coding, understands it quite well, but I just want to tune it to my codebase, how I'm dealing with it, maybe my eccentricities, that sort of thing. Training takes a few hours, pretty much. Key's not to over-train. You're not trying to make the model have a general code understanding from scratch. You're taking open-source models that already have a lot of this stuff and just fine-tuning them down to exactly what you want, how you work.

Now I've got something that's local, cheaper, faster. I can use it more easily on client projects that maybe don't allow me to use AI that goes externally. Really useful there. Quality isn't identical. On complex novel evaluations, it's maybe 80%, 85% is good. It's something that's significantly more personalized to me than you would get from just a coding model out of the box. It doesn't suggest things I constantly dismiss as well. Over time, it's learning those patterns, and I'm getting that flywheel.

Platform Capabilities

Let me show you how this generalizes. This is a set of platform capabilities. These are the three that I feel like it brings out as distributed tracing. I think the key here is product usage data. That's where you really start to get a lot of the value. Again, feedback mechanisms are really useful. It's really difficult to get feedback from users. Much better if you can provide them a user experience that becomes a signal for training later on. A good one for Gemini would be export to docs. It's not just giving a thumbs up or a thumbs down on a particular output, but it's actually like, ok, this is great. I want to take this out of the tool now. Very difficult to track when someone copy, pastes something from a chat screen. You need to go one step further and provide that UX that allows you to get out there.

How this generalizes, the core pattern is, instrument your agent with OTel. Capture user actions as training signals. Distill the model behavior into an SLM. Deploy it with safety rails, guardrails, check, and iterate continuously. This is something that you can then build into that ongoing. The architecture is the same across all of these agents. You have a shared instrumentation layer, shared extraction pipeline that can turn the traces into example data. You've got the same fine-tuning and model-serving infrastructure. You can have similar eval tooling as well that will allow you to test how you're going in the right direction. This is what makes it a platform utility instead of a one-off piece of work. Thinking about how you can make it extensible, how you can make it usable by multiple teams, how you can make it general enough that it is easy for them to fit into their own patterns, but also specific enough that you can actually get a signal from it.

Once you build the pipeline once, adding a new agent to the situation, distillation loops should be relatively straightforward, should be quite incremental. Instrument it, define what the user actions are. Define that positive or negative scale of the feedback. I talked about strong positive, weak positive, strong negative, those kinds of things, and plug it into the existing infra.

This is one example of how you could map the feedback to traces, where you've basically got a very simple mapping in Python. There's an economic angle here. Machine learning is expensive, that labeled data, you need to design a labeling process, you need a lot of infrastructure for it. You need a lot of tooling to do all that labeling, and actually using the product feedback is something that makes it much more simple and scales a lot better. Data flywheel approach inverts that long labeling process. Your product usage is the labeling process. Your users, or in my case, me, the labeler, they're labeling for free as a natural byproduct of using the tool, and that's what you ought to be aiming for. The flywheel compounds. Better models lead to more usage, lead to more data, lead to better models. This is something that can really take off. You're not investing linearly in data collection. You're really building a system that generates its own training data continuously, automatically. You can improve continuously, and potentially even automatically.

This is another example that you could provide for users. You can make this interface really simple for engineers who maybe don't have the expertise, for understanding how to build LLMs or even ML models. This can just be simple config. You can just say, here's the list of spans that I think are strong positive feedback, weak positive feedback. Give it a scale of value, and a list of spans that you want to track that they've put into their code, and this generalizes really well. Another example that you could do is actually just have your own platform-specific feedback spans. This would even skip that step of config. You would just say, yes, import the SDK, get the feedback span, fill it out. Again, very similar to what I've showed before in the example of how I did it myself, but actually now with just a wrapper. Or you could do it as a decorator, anything like that, very simple. The key point here is to think about that really simple user experience for the engineers, data scientists, data engineers on your platform. It's truly simple to get started with these things. As soon as they start doing them, they start to realize how relatively straightforward it is to do this stuff.

2025 - The Year of Agents

At scale, the platform gets really interesting. Imagine an organization with dozens of internal AI agents: customer support bots, code review assistants. We've actually seen presentations where people have gone into detail about the number of different agents that they've got that are being used across the organization, data analysis. Each of those agents should generate telemetry, so each of those agents could then improve by distillation. Again, this is probably not something that you do straight out the gate. The instrumentation's probably more going to help with debugging, understanding, prompt iteration. We had a great graphic in one of the talks about when to do RAG, when to fine-tune, when to do pre-training. Again, this is definitely not the first step that you should take, but for me it's something that's really useful as a tool in the arsenal, and it's not difficult to get started with. Crucially, each agent can learn from each other.

The customer support bots' learned understanding of your product can inform the documentation generator, their interactions and the different interactions between these agents will change over time. You can start gathering that data and seeing those interactions. You can see those as spans across multi-agent workflows, and then start to make this more complex and pick out the training data for each different sub-agent. Your AI tools are really learning from that production usage. You're building that learning organization in a very real sense. For me, that's this quote from Andrej Karpathy. I think probably we're all maybe of the same opinion here. We're seeing this a lot with our clients. Agents are here to stay. We're building them, they work. Certainly have many limitations, but I think there's going to be a long time for us getting this right. I think that timescale, rather than it being the year of agents, I think next year really looks more like more agents, in my opinion. I'm biased, I'm obviously writing a book about it.

Summary

I just want to wrap this up and talk about what this really means. We're in the strange moment in the history of developer tools. For me, these tools are really shaping how we behave. That's not always positive. It's not always negative, it's not inherently bad. We have always adapted to use the tools the way that they were built. IDEs changed how we code, how we navigate the codebase. Git changes how we think about version control. I don't know if any of you have tried something like Jiu-Jitsu, or the Meta stats diffs, but you start to realize that you're really embedded in this current way of thinking, this current set of tooling, and those ways of working don't necessarily translate to different tools. There are so many interesting and exciting AI tools that are coming out, but you really do have to think with each of them, is this actually making me a better engineer?

Is this making me more productive, or is this just, I'm going fast, but I'm not going to go far? That's one of the things I think about, and which led me to build this tool that I wanted some of that control. For me, that control means like I want to be able to configure it, be able to shape it. I want to be able to give it feedback myself personally, not just how it works generally. I'm not against using off-the-shelf tools at all. I think they can get you a long way, and actually it's really interesting to look at them and learn from them. For me, the distillation changed that piece, it was the one part of the puzzle that I couldn't really have access to without having a research lab. Being able to do this kind of distillation, like I can distill the behavior that I want into the model, how I want it to work, and actually to some degree I'm still in the driving seat.

I'm the one still shaping the tool. It's for sure shaping me, but I'm able to shape it back. I think that's really important, because as soon as we stop those improvements, we're maybe becoming redundant to some degree. For me it's that agency. It's that ability to, as AI tools become more central to how we work, we remain in control of our own workflow and how we interact with them. For me, I think this is actually a really nice feature, like an idea of many different intelligences rather than a single AGI, many different smaller intelligences that allow us to specialize a lot more. It's a future of AI tooling that I want, which is like personalized, distilled down to that personal task for me, like individual users and organizations and making sure that we're staying in control. We can run things locally. It's much more portable. This fits all of that.

Takeaways

Let me leave you with a couple of concrete takeaways that you can apply immediately. I think first, instrument all your agents with OpenTelemetry. I'm sure most of you are already doing this, but some people for sure still need the message. Don't wait until you're thinking about distillation. Just get it in place. Capture everything that you need. Again, really important, that user action. Think about ways in which you can get feedback from the product that will then tie back to improving the tool. You can get a lot of value from that long before you need to do model distillation. If you've got a dataset that already has these things, you could turn that into evals and start using that as, again, another step that I didn't illustrate in my platform piece, but something that's relatively straightforward to do. Auto-instrumentation does get you 90% of the way there, and most libraries do it.

Crack on with that. Second, really design that user experience to make decisions observable. That's the key. This is the subtle but important piece. If you can get that into the product experience, you're no longer swimming upstream, you're going with the flow of product usage, and you're getting that feedback loop going really nicely. If users interact with your AI, but you can't tell whether they found it helpful, then you're losing signal. The thumbs up, thumbs down, accept or reject, bookmark or stop, these are good first pieces, but if it's in that user workflow, it's 10 times more powerful. That's, I suppose, why I designed the LSP that way. By design, the explicit actions gave me that feedback signal. Third, when you do have enough data, distillation is actually cheaper and easier than you might think. The tooling is really accelerated. I'm sure many of you have distilled models yourself from fine-tuning.

It's really simple. More than that, you can now enable more people to do it. We at a conference like this, are going to be the tip of the spear. Making it really easy for people to follow in your footsteps, learn how to do it, start building out that knowledge and understanding of how to do it, that's what drives that AI adoption. Having more people who understand how these things work, that's also super important. Make sure that you make that platform user experience really simple and easy for people to understand so they can get going from the start. They don't need a massive compute budget. They can do it locally on their laptop. They don't need a big team of ML engineers like many of us are. Fine-tuning a 7-billion parameter model locally on MLX might only take an hour or two, and you've shown someone how they can take something from A to B.

You can see that improvement with evals. Start small. Pick one agent, one workflow, get it instrumented. Collect a month's worth of usage data. Hopefully it's already instrumented, but go and check for those user actions and add them there. Collect a few hundred iterations, and then try some distillation. You'd be surprised, I think, at how well it works. You're not necessarily going to be able to create from a small agent a complete product that was going to blow away something that's been built and has a whole team on it, but actually you can significantly move the needle for your own personal workflow, and for me that's something that's really interesting. For me, future of AI tooling is local, personalized, and continuously improving. Only questions I suppose for me is whether you're going to build tools that shape you or tools that you shape. I've also put up a way to stay connected, and a bit more about the book. We're writing a lot more about this stuff, how to actually make this stuff scale and work at scale in organizations using platforms.

Questions and Answers

Participant 1: Can you tell us a little bit more about the process? Do you do a lot of [inaudible 00:35:51]?

Ben O'Mahony: The question was how I go about deciding which features to instrument? I use it for a bit of everything, really. I only vibe code prototypes, mostly. I'm not fully autonomous agentic vibe coding. For me, I think there's still an element of wanting to have a bit of control there. I think that whole idea of this agent where I worked, rather than in a separate tmux or UI, was something that really appealed to me when building this thing. I pretty much just use it for everything. Again, like a linter, it's not always useful, but when it is, it saves me a lot of time and it can highlight things I didn't think of, which is really useful. I think that UX of it, being proactive. For autocomplete, that's really good, and I use it as well, but that gives me what's coming next. Whereas this actually analyzes the whole page, the whole document that I'm in, and gives me the errors in that area.

It's quite a nice way of doing it, because I could be creating a new feature and it would highlight something that was someone else's contribution that isn't hitting it, and I hadn't looked at that because I was focusing on my new feature, and it can highlight those sorts of things. Then while I'm there, I can tidy it up, or leave it up to me.

Participant 2: How do you see [inaudible 00:38:02]? It's just context engineering that goes out the window, because you're [inaudible 00:38:10].

Ben O'Mahony: Yes, kind of. Actually, there's some improvements I'm thinking about making to this particular tool around actually using something like Tree-sitter, which is a language-independent grammar for abstract syntax trees, to maybe go down and be a bit more granular. At the moment, you just take the whole file and send it off to AI. It will only find a certain number of errors there. If that's the case, and you've got one file to rule them all that's like 20,000 lines long or something, it will fall over. You still have to think about some context engineering, but because you fine-tuned the model on a bunch of the behavior that you maybe have in your different rules or other ways in which you're managing that context with lookups and whatnot, then actually, yes, you can skip some of that stuff. For me, I think these things aren't mutually exclusive.

You very much want to be doing all of these things in combination. This is just one tool in the tool belt. It's an interesting one. It's not the first one I'd pick out. I probably would start with the context engineering or the prompt engineering and then go to this when I've got to a certain point. I've plateaued, retrieval-augmented generation isn't getting me what I need. This might go to that next level.

Participant 3: [inaudible 00:38:10].

Ben O'Mahony: Unfortunately not. It was supposed to be by now, but it's not. It will be very shortly, I'm sure.

Participant 3: [inaudible 00:40:22].

Ben O'Mahony: Yes. My solution at the moment, because it's a personal tool for me, is being to not use it on projects where that's the case. For sure, if this was a commercial tool, that's 100% something that we'd have to work on and would be a major issue if you were to leak a bunch of internal IP code, that sort of stuff. Absolutely, a real important part of the piece of the puzzle. Off the top of my head, I would say you'd start to mutate it with an LLM, but then you probably wouldn't get the accuracy that you need. That's an interesting problem.

Participant 4: You mentioned about like generally to try out a RAG type solution? Did you try one for this, or you're probably unfamiliar with the solution? I'm just curious if you're allowed to talk about it?

Ben O'Mahony: Part of my thought process was to practice distillation. No, I didn't go RAG with this one. I wasn't doing it on large codebases that it had issues with understanding the context. It was relatively small codebases. If you're working on a 2-million-line legacy codebase, a lot of that context is going to be needed. You're going to have to do a lot of like understanding the abstract syntax tree, the graph of execution, all that sort of stuff that's going to really move the needle on that stuff. I steered clear of that on this one. For sure, the reality is, again, like I said, these are combinatorial things. For me, I've considered adding quite a few other tools. They would be like giving it the ability to understand other LSPs, other issues, so that it could maybe fix them for me. It would be things like being able to look up prior art on the internet, potentially. It would be things like a RAG of the codebase of some sort, so when it initializes, it parses all this stuff and it's got go to definition and that stuff handled. There's lots of ways of improving this.

Participant 5: Did you see progression as you were training the smaller model? If you did, how would you catch those [inaudible 00:43:28]?

Ben O'Mahony: For me, I put some evals in place that allowed me to see some small coverage of that. I actually find that without a larger benchmark, coding agents are quite difficult to evaluate. I think it's impressive that people at OpenHands and that sort of thing have been able to evaluate these things effectively. I found I had quite long evals that were mostly LLM as judge, give me some signal, but wasn't as deterministic as many of the other evals that I was able to put together for other agents. For sure, it's not as good as a frontier model, but it was faster, local, and cheaper. There's a tradeoff there.

See more presentations with transcripts