Presentation: From ms to µs: OSS Valkey Architecture Patterns for Modern AI

Transcript

Dumanshu Goyal: I would like to open with a story. This story is about NASA's Space Shuttle program. What you see here is the original vision behind the space shuttle. This program started back in 1970s, continued all the way to 2011 until it was retired. This is an on-paper vision. The idea was to build a reusable spacecraft. The idea was to go back and forth into space and do it cheaply. This was basically the designs behind the spacecraft. Then from the reusability, they came up with this requirement that's similar to the other planes we see out there. We want runway landing, the spacecraft to be able to land on a runway. The runway landing from reusability became a requirement, and that's where they came up with this design. What's important about this design is because of being influenced with the existing structures of planes, they added these delta wings.

What you see there is basically called a large delta wing. This is a real model. Its name is Atlantis. It did 33 space missions over about 25 years, until it was retired. This Atlantis, again, you can see the delta wings over there. What happens is when these spacecrafts, they reenter the atmosphere, there is high heat generated. Then the wing basically slices through that heat. Now, because it's experiencing that heat, it needs to be protected. This led to the requirement of installing these tiles. These are silica tiles. What you see in the picture there, the small brick-shaped structures, these are the silica tiles. This model has about 24,000 silica tiles on it. The idea was to protect the body, the wings, the underbelly of the spacecraft and make it work. When they designed and built this airplane, it was estimated to be a $10 million round trip, what it would cost us.

Then it would require some maintenance and it would be ready in about two weeks' time frame. This was the original goal. When they started using it, in reality, the $10 million ended up to be $1.5 billion. The costs exploded. The two weeks' turnaround time to get ready for the next mission got to be two months. What went wrong? The silica tiles, they had to be replaced. There were unexpected damages during the flight. They were thinking that, ok, they would hold it well, but they were not. The heat was too extreme, a lot of new discoveries. This problem ultimately added to the on-ground complexity as well. The on-ground operations, as you can see here, they got really complex when they had to get this ready for the next mission. The complexity also bloated up not just the cost aspects or the performance aspects around when it can be reused.

In 2011, when NASA retired this program, another program started, the Commercial Crew Program. This is a bit about privatization. This is where you see Starliner there from Boeing. Another top model is called Dragon from SpaceX. These are what we call as capsule designs. They went back to the core requirement of reusability. There was reusability. You want to keep the payload, the crew safe. There is safety of the crew as the core requirement. Then they basically challenged the runway landing requirement. Is that really required? Do we need to really land this on a runway? They let go of that requirement. They let go of the tiles. They let go of the wings. They came up with a simpler design, a capsule. All they had to do was to protect the nose of this capsule, which has this heat shield on the top, so that when it enters the atmosphere, it can sustain high heat.

Not just sustain it, but also move it away from the body of the capsule, so that the rest of the capsule is not impacted. This blunt body of this capsule was a success, and it has been building up on it. This is how they went back to the requirements, and then basically reshaped the spacecraft vision around how do you do reusable space travel.

I want to frame this story around a quote. This quote is about perfection. This is an age-old quote from Antoine de Saint-Exupery. He was an aviator, an engineer, not a software developer like us, or most of us. What he mentioned, when we talk about the spacecraft mission, you have this mindset, I need to solve for runway landing. I'll add the tiles to the system. Then you keep fixing it, keep maintaining it. Versus taking a step back, taking a holistic look, coming up with the capsule design, challenging your requirements. Do you truly need them? Then basically getting rid of the wings, the tiles to simplify the end result. This is basically what I refer to as designing for efficiency. In this talk, we'll cover several examples of what it means to go back to your requirements, do a rigorous requirement analysis, not take them at the surface.

Also, combine them with a holistic tradeoff analysis. This tradeoff holistic analysis, what you could say in the spacecraft mission, it's hindsight now, but you could say it was missing as to what would be the real impact on the tiles. How much it would cost us for real. When you combine this rigorous requirement analysis and the holistic tradeoff analysis, that's when you come up with the efficient designs, whether it's cost-effective or for performance reasons.

Professional Background

I'm Dumanshu. I'm a lead engineer at Airbnb. I lead their data platform. This data platform is responsible for all the bookings you do on Airbnb. All that data gets stored. This is what drives their $11 billion revenue. Before Airbnb, just a little bit about my background. I spent a decade obsessed with durability, where I got to work on internals of several foundational systems. One of them was AWS DynamoDB. After that, eventually, I moved on to the other extreme, which was raw sub-millisecond performance with in-memory caching at Google. This is my talk's title. It shows us basically the new reality about AI, where performance equals efficiency. When we are moving from milliseconds to microseconds, sometimes it's about real-time performance. We'll see how in the AI space this matters. Sometimes it's not about the latency or the performance. Most of the time, it would be about the cost. We'll also go over how moving from these milliseconds to microseconds affects your cost. I'll frame this talk around Redis and Valkey, some of the caching systems, but it's pretty generic in the sense it applies to maybe Memcached or a bunch of other caching systems you might be using.

Roadmap

This is my agenda. We'll cover it in three parts. The first part is about the need of microseconds. We all talk about, we want to get faster. We want microseconds. What's the real need behind microseconds? We'll do a case study, a use case in the AI space to understand why microsecond latency could be truly important, or it might not be in your case. Then the second part talks about our journey of how we are living with these milliseconds architectures, how it got there. I'll frame it around the evolution of Redis and Valkey. Then we'll also cover the architectural tradeoffs, like what's the impact on latency, the performance, the cost, the dollar that you pay for the architecture, and then reliability. Then, in the end, the third part will cover the milliseconds architecture. In the milliseconds architecture, we'll talk about what the price performance equation looks like there, what the reliability equation looks like there.

Part 1: The AI Data Wall - Why Milliseconds Are No Longer Good Enough

Part one, establishing the need. We want to go with requirements. We'll start with what's the need of these microsecond latencies. Why does it matter to us? The example I want to use today is an AI feature store. I'll talk about what the feature store is. This example I'm taking from the DoorDash AI platform. All public blogs. Just was quite relatable to my talk. This example basically starts with a prediction service where you have a 100-millisecond budget, not microsecond, 100-millisecond budget. That's a lot of time, to basically drive a prediction. Then when you combine that with complex data needs, you'll see why it requires an underlying system to serve microsecond latency. This is code from one of the DoorDash's blog posts. This is about a prediction service which is used to detect fraud or provide restaurant recommendations. The number one requirement of this prediction service, as they quote, is basically to be able to serve a prediction within 100 milliseconds.

What does this prediction service do? It looks as something called as features. You can think of features as a piece of real-time data. If you are in the fraud prediction service use case, you would see, is this credit card new? How many logins have happened recently? Stuff like that. You want to be able to feed real-time information to your AI model to be able to drive a prediction out of it. The important thing to note here is for a single prediction, the system needs hundreds of features to be able to make that prediction. It's like quite a diverse dataset, lot of features, these pieces of data coming together to actually make a decision out of it. The underlying challenge is, how do you make it happen so fast? You're talking about hundreds of features. You're talking about 100 milliseconds. This is like a typical architecture.

It's an AI inference use case around the prediction service. The architecture details are not that important, but I think I would like you to focus on the prediction service which basically gets this user contract. The user interacts with the prediction service through the app, and this is where the 100-millisecond constraint comes into picture. This prediction service works with the AI model. Then it also works with this AI feature store. This AI feature store is the contract that we are trying to dive into. Like, what's the contract between the prediction service and the AI feature store? Can it serve millisecond latencies? Is 2 milliseconds good enough, or does it have to be in hundreds of microseconds?

Going back to the DoorDash blog post, this is where they mention, because the single prediction needs hundreds of features, so a natural thing to do is to fetch data in parallel. I have 100 different sources or whatever. I could basically just fan out those calls. When you fan out those calls, what they call out here is the p99 latency. I basically refer to it as tail latency. You need to keep the tail latency low. Why does that matter? Here is a relay race example. When you're basically making hundreds of calls, your latency is decided by the slowest call. When you are having so many calls, your tail latency is bound to show up in every single prediction request. Your latencies, even though in steady state, might be just about 1 millisecond, but there would be that one request that maybe goes all the way to 10 milliseconds.

The analogy I want to map to the relay race where you need everyone to finish, get to the finish line, and then one person basically is lagging behind, and that dictates what happens, how soon you can finish the race. This gets worse. We talked about the AI use case. We also talked about there is a complex data need. What does this complex data need look like? We went over, you need to make 100 parallel calls. Then the next one is a sequential lookup. It's pretty simple. There is a parallel thing. There is a sequential thing. The sequential lookup, all it means is you have data dependencies. In order to get to a piece of data, sometimes you have to traverse the path. Here is an order-related example. You have to fetch a piece of information, and then only you can make the next call. Simply making calls in parallel doesn't help.

Sometimes you have to make it sequentially. When you make calls sequentially, and you are prone to these tail latencies, the high tail latencies, that's where you'll see it starts to eat away your budget. Out of that 100 milliseconds, you want to give as much time as possible to the AI model to do the real thing. Just for fetching the data, you have already eaten up a lot of budget, in this case.

This brings us to the final verdict from the DoorDash AI engineers. What they basically describe here is their AI model latency is in low milliseconds range. We are not worried about that. That's a given. What that means is the underlying store, the underlying feature store, needs to be able to serve the features at a latency which is proportionately lower. This is where it gets even tighter, where we talk about microseconds, for example, to serve the features to this model. Another code, in this example, they build this feature store backed by a data source using Redis. For our talk, we'll talk a bit about Redis. Eventually, we'll go into Valkey. What we have established here is the requirements for this feature store. Like I said, it's backed by another durable data source. Then we are talking about serving these features at a microsecond latency level. I would like to start with a quick introduction to Redis, Valkey.

I'll get back to our favorite LLM to do it for us. Redis, I think in plain, simple terms, it serves numerous use cases, you name it. It has advanced data structures, so many operations available. It's quite simple to use, and that's what basically speaks to its popularity. The important thing to note is this technology already goes back to 2009, since when it started picking up on its popularity. Then, in 2024, that's when the open-source licensing change happened with Redis, and Valkey was born. Valkey is effectively a fork of Redis. You can do all what you want to do with Redis with Valkey as well. That's why I use it interchangeably. Then, it's fully open source. It's backed by the community, a bunch of hyperscalers. That's a little bit history about Valkey. This is a screenshot from the press release around when the Valkey fork was announced, Linux Foundation backed it up. This is early 2024. This is a Stack Overflow 2025 database survey results. What you see here is, in terms of the desired level among these developers, Valkey is trending very close to Postgres. Postgres has really gained community presence, and in terms of the desired level, just this is within one year of its fork.

Before we end the basic introduction, I want to do a typical 101, like a Hello World Valkey application. What does it look like? You have a client. You have the Valkey server. It's a plain, simple key-value lookup, what we have here. This is my terminal running Valkey CLI. You can use the CLI mode. There are a bunch of ways to access it, typical clients. What you do is a SET my_key my_value, and then you basically can get it after that. It's as simple as that plain key-value lookup. We're not talking about anything more advanced than that.

Part 2: The Milliseconds Journey - The Architectural Choice and Its Hidden Costs

Going to our part two. So far, what we talked about is how the DoorDash AI use case that we looked at is basically a requirement for real-time performance, which is measured in microseconds, not milliseconds or not tens of milliseconds. Next, we are going to talk a bit about our milliseconds journey. In the context of Redis, how the architecture has been evolving, what it's like today, and what does it mean in terms of price, performance, latency, reliability. This is a chart from DB-Engines. It's widely popular. They have a way to calculate scores. This showcases, over the last 10-plus years, how the popularity of Redis has been rising. Going back to all the way to 2009, when Redis started to get popular, a lot of applications started using it. They had a simple caching setup, like what we saw with the Valkey 101. In this case, your application is, again, accessing Redis, as simple as that.

However, because of this popularity, the applications started running into the scalability constraint, where the single Redis node was not able to serve the workloads anymore. Either sometimes there's too many client connections, too many operations per second hitting the vertical scaling limits of a single node. There was no native solution available at the time. Then this led to new architectures, which I refer to as the proxied architectures or the gateway architectures, where you basically take the independent Redis nodes and then you stitch them together. You make it look like one giant system, one giant cluster. Your data is sharded across them, and your application basically connects with them. The blueprint, what I have, is the legacy applications and then a gateway layer, and then the Redis servers backing it.

What we see in 2009, over time, so it was 2015, 6 years later when Redis Cluster was released, Redis Cluster improved upon these sharding capabilities. It provided a native sharding capability built into Redis, where a bunch of nodes could work together to provide that one giant cluster view to the end client. Then, the proxies were still there, legacy apps were still there but the trend continued. For one, the Redis Cluster offering was not mature enough. It was new. It was complex. You needed some complicated logic on the client, so not everyone got it right. It was not gaining that much popularity, which doubled down on the proxied architectures. They continued to grow. The legacy applications, even if they wanted to adopt a clustered offering, they couldn't because they didn't want to change the client, so what they did is they, again, continued with the proxy layer in the middle.

You put the cluster behind the proxies, and then you can make it work. You don't have to do the independent nodes behind the scenes. You can still use clusters, and then the proxies would abstract it away from the end client application. Legacy constraints was feeding into this growth, or is still feeding into the growth. The second one was related to simplicity, or abstracting out, or decoupling your client applications from the server, so that's like another one of the non-functional reasons why you see proxies coming into picture. The third one is more technical, I would say, less of a constraint from a legacy perspective. This has to do with connection multiplexing where your application scale is so high, and there are very few applications that I've seen at that kind of scale where you are basically running with 100,000 pods, the client pods. They are trying to connect to the cluster behind the scenes.

If you were to connect to the Redis, Valkey cluster, whatever, directly with those 100,000 connections, at least one each coming from the client, that would definitely kill the cluster. In those cases, you want to have an architectural improvement using proxies, for example, which can take on or offload the connection management. What happens is you get 100,000 connections onto the proxies. They are stateless, so you can scale them out horizontally. Then they would take those requests and multiplex them onto fewer connections against the backend server. The idea is to take away the connection handling responsibilities, move them to the proxies. These are some of the legit or the real requirements around where proxies have served a purpose or continues to serve the purpose.

In this architecture, we use a common open-source Envoy proxy. There are 5 to 10 proxies which are widely used in the industry. I just took one of them, but it has nothing specific to do with the Envoy proxy, so we'll use this example throughout our discussion here. In this example, you have the Envoy proxy layer, which is hosting this logic. The logic could be in two parts. One is, you have independent Redis, Valkey nodes behind the scenes. I've switched to Valkey here instead of Redis. The Envoy could shard the client data using a simple hashing scheme. It's not a database, so there are a little bit of data losing concerns, but it can make it work. Then your client connects to Envoy proxy. It can talk to any Envoy proxy. Then it routes the request to the right node. The other mode is the cluster mode, where Valkey is self-aware of how the data is sharded, and Envoy proxy can also connect to that. That's the advantage of these proxies, that they can speak multiple protocols with the underlying backing system.

We talked about the journey of Redis, how proxies came into picture, how they continue to exist, some of the top reasons behind these proxied architectures. Now we're going to dive into the real-world tradeoffs or the metrics around price, performance, latency for these proxied architectures. What do they actually look like? Let's start with the latency. One could categorize it as a performance dimension. Latency is one way to look at it. Coming to our architecture with three Envoy nodes, three Valkey nodes. Now, to study the latency aspects, the performance aspects, I'm going to simplify this a bit. We are going to just reduce the three Envoys to one, and similarly for Valkey, three to one. It's a pretty simple setup with the client talking to the proxies, proxies talking to the Valkey node behind the scenes. The run I have here, it uses a utility called memtier_benchmark.

Nothing special here. It's just a tool to drive load, and basically measure some characteristics of the system. I have this deployed on EC2, AWS. You can see the VM types. Both of the proxy and Valkey are the exact same machine types. They are in the same availability zones, no cross-zone hop, wanted to reduce the interference. They are both 8-core machines each. This is a benchmark run from my terminal. Don't worry about the grayed-out details. What's important is we talked about the memtier_benchmark. Then the second aspect is it's a read-heavy workload, which you see typically in practice. In this case, it's roughly a 90-10 ratio. More than 90% of the traffic is read traffic. In this setup, what you see is this setup is driving half a million QPS. Half a million queries per second is being driven, which is roughly 90-plus percent reads, and then the remaining is sets.

The latency. Since we wanted to talk about the latency. For these half a million queries per second, the latency that you get, so we measure on two sets, one is the tail, the p99. That's roughly 2.5 milliseconds. This is in the milliseconds category where it could go from anywhere like 1 to 10 milliseconds, roughly speaking. Then there is p50, which is your median latency, and this is close to 1 millisecond. This is what I would refer to as our milliseconds app. These numbers are still super good. Like, they are driving half a million QPS. The latency is super low, single-digit millisecond. Not bad at all.

For each of these setups, we are going to look at a little bit deeper into the latency, price, performance aspects one by one. First, we are going to look deeper into this p50 latency. I like sometimes appreciating the system characteristics, like how it ends up to be where it is right now. In case of the p50 latency, we talked about 1 millisecond, and as you notice, this latency is measured from the client side, so it's a full round trip covered in 1 millisecond. There are two hops going on. These two are network hops, so in order to measure the cost of a network hop, I used a utility, My Traceroute. It's a Linux utility. It's similar to a ping, just has more fancy knobs around it. What you see here is between the client and the proxy node, there is a 300 microseconds hop. This 300 microseconds hop is basically in the same availability zone.

Like I said, not much of an interference, not a cross-zone hop, which gets to be higher. The same 300 microseconds would go from Envoy to Valkey, all round trip. Ultimately, what it ends up to be is 600 microseconds, the two hops, 300, 300, out of the p50 of 1 millisecond gets spent on the network. What you're left with is 400 microseconds, and that 400 microseconds involves the full proxy processing to route the request. It involves the Valkey server processing to serve the dataset. Then it also involves a little bit of client processing, where the client sends the request, gets the response, measures the time taken. All that stuff is basically done in just 400 microseconds. Network, maybe you can't do much about it for a remote lookup, but that's how the latency breakdown looks like for a p50 latency.

Next, we will look at how much do you need to spend to get a certain amount of throughput, measured in queries per second. Cost is important. As engineers, we generally talk about latency, performance. I like to also talk about the dollar. We're going to dive into that. In our setup, recall that we got half a million QPS at 2.5 millisecond tail latency, pretty good. The CPU measurement during this run was 90% CPU utilization on Envoy, which is pretty high, so it means it was almost completely taken. Then the Valkey server CPU utilization is basically 60%. There still seems to be a lot more capacity which is remaining on Valkey to be utilized. Just mapping it back to the diagram, visually you can see there is 90%, 60%. Why is this proxy 90%? Why is it so compute intensive? Why is it consuming so much CPU?

In the caching world, this is where you'll see that it's quite different from the databases world, where we worry usually less about the QPS. It's more about latency, the I/O, and stuff like that. In the caching world, there are two main things that would typically come up. One is the QPS. Sometimes the applications are offloading the QPS from the database to the cache. Generally, caching applications, they would have high QPS, or they have a lot of in-memory storage. You are paying for in-memory storage, which is much more expensive than our typical disk-backed storage. This is where the caching workloads are interesting, because you have to pay attention to the QPS aspects as well. Now, because of this QPS, this traffic is being forwarded through the proxies. What I want to do is, I want to build an intuition around where is this compute going. The 90% consumption on the Envoy, what the breakdown looks like.

How we talked about the latency, where we talked about the network hops, network hops taking majority of the latency. In case of the CPU, so one intuition is basically, when you're forwarding traffic through the proxies, it's doing twice the amount of I/O as compared to the underlying Valkey server. If you look at the Valkey server, it gets a request, sends a response out. In case of the proxy, it's a pretty simple thing. Client sends a request, that's one. Request goes out to the Valkey server. The response comes back from the Valkey server, that's the third one. Then the response goes back to the client. The proxy node has to move this data or the underlying network packets twice the time of what a typical I/O would look like for Valkey. This whole process of moving the network packets is very compute intensive. What you see here is the process routing the request.

Underneath, the network card, once it receives the packet, it has to put that into the host memory because we are running virtual machines, so they are running on a physical host. Once it gets into the memory of the physical machine, another process would route it to the guest VM, the guest OS. Then from there, through interrupts or some mechanism, it would get delivered to the kernel's TCP stack, which is where eventually it would find its way to your application where your code of routing the request runs. Then it would be the reverse path when it has to send something out.

This was about building an intuition. They're twice the I/O. Now let's look at the real data. We want to validate our intuition now with a profiler. Flame graphs are also my favorite, but they're also very hard to read when it comes to slides. Flame graph is just a visualization tool which captures, during the actual run, the stack frames, and then sees what stack frames appear more often than the others. Then if something is appearing more often than the other, you can take enough samples to determine maybe this is the area where it spends more CPU than the others. Don't worry about reading these stack frames. I'm just going to do a higher-level analysis here. We looked at the 90% Envoy CPU consumption. I have a breakdown, which you can also measure from the process. There is user mode, there is system mode. User mode is where your application is running.

The code written, whoever built the proxy would run into the user mode. Then it interacts with the system calls, the TCP stack, that's all basically kernel mode. The split of this 90% is very even, like 45%, 45%, user mode versus kernel mode. Lot of I/O, there is obviously a lot going on with the application. You see the two big boxes I have, these are parts of the flame graph. One part on the left is about the application processing. This is less about the I/O, but it still needs a lot of CPU. This can be heavily optimized, writing different code and all that. The right part is the TCP send and receive path which involves a little bit of application and also the kernel frames. It's like the I/O, what we said is twice as expensive. The I/O does form a major chunk of this whole 90% CPU utilization. The key point we are trying to understand is when you say, proxy consumes a lot of CPU, what is it actually doing? It's actually routing requests. A significant amount of CPU spent there. The rest of the CPU, in this case for Envoy proxy, it's half, it's spent in the I/O.

What we have established is this 90% Envoy, given that the Valkey server has capacity. To me it reads like, Envoy is like the chokepoint here, and then maybe there is something we can do. The idea is to, now I'm going to add one more Envoy proxy to the setup to horizontally scale this layer. Then, using these two Envoy proxies, they are stateless, so I'm going to load balance the request. The client would just pick any one arbitrarily and drive the traffic. Then we're going to just repeat the benchmark. With these two proxies fronting the server, we have 1 million QPS driven from this system. When we were using one Envoy proxy, we had half a million QPS being funneled through the system. Now with two, we have 1 million. That's good because it has a nice scaling characteristic. One was half, it doubled when we doubled the hardware there.

That's good. The latencies are quite similar. p50, again, 1 millisecond. Tail latency, 2+ milliseconds, similar to what we had before. We still retain our milliseconds app roughly. Then we were able to hit 1 million QPS out of this system setup. Since we talked about the dollar cost, let's look at what this setup would cost us. The eighth generation Graviton instance that I'm using costs about $230 a month. The hardware we have provisioned is about $400 roughly for proxies, $200-something for Valkey. This comes out to be about $700 a month to deliver on 1 million QPS at low single-digit millisecond tail latency. This is an important result. Let's take a moment to capture this. I'm going to come back to this later in the discussion. This is not bad. If I gloss over the architectural details, you have an in-memory setup. In-memory is expensive. Everyone knows that.

We are spending $700, justified for the business. Nothing seems wrong. Note that the $700 in reality is actually a lot more. It's going to be close to easily $1,500 or something, because this does not have high availability built in. With high availability, you will have copies and then you'll have cross-zone hops. There will be a lot of network jitter, more cost involved. What that would lead is like worse latencies, higher cost. What we are talking is a little bit of a crafted setup in the same availability zone without those variables.

What we covered so far, we talked about the latency aspects. We talked about the cost associated with delivering 1 million QPS at a certain latency. Typically, architectural evaluations would stop here because we added the proxy. We were analyzing the proxied architecture. It's a stateless system. Stateless systems are usually harmless, from our experiences. Very few times we would look at anything beyond the setup. The third piece is reliability. It has an interesting pitfall associated with it, which is a single point of failure. Our setup has a single point of failure. At least to me, it was not obvious when I first exposed to this problem as to, what is special about this architecture where reliability could take a hit. It's related to head-of-line blocking. It's related to single point of failure. We'll go over that. To understand these pitfalls, we want to drive some traffic. Again, nothing fancy.

We have our architecture. I'm going to let go the single node one and bring back the three proxies, three Valkey servers because more nodes are interesting. We're going to drive some traffic. I have a custom script, not the benchmarking one this time, just to capture the client-side availability, what it looks like. It's driving a small amount of traffic, and the availability, what you see here is 100% every second, which means all the 100 QPS, basically all the requests are successful. Nothing comes without failures, or even worse, slowness. To make things interesting, we're going to take a shard and inject some slowness into it and see how the system behaves. This is Lua scripting. Redis has this powerful feature, Valkey has it as well. It basically is like a server-side function where you can inject some logic and make it run on the server. The code here is a workaround to basically make the main server slow down for about 5 seconds, so just to induce a slowness or a failure there. You can reproduce this easily. Just don't do it in production. It's pretty deadly to the production server.

Now let's see what happens to our client-side availability. We have the 100% run rate, and then for those 4 to 5 seconds when we basically injected the slowness, our availability dropped to zero. The client is experiencing full unavailability, no requests going through for those 5 seconds when we inject this failure. We had evenly distributed traffic in our setup. What even distribution means, like our shard_1, which was experiencing this slowness, was getting one-third of the traffic. The other shards, we did not touch them at all in production or in our setup. They were getting two-thirds of the traffic, but yet our net availability is zero. As you can also see, all my traffic is now routed to shard_1. Shard_2 and shard_3 are actually not even getting any traffic. That hints us to something, like they are all healthy, no CPU consumed, like what is going on?

Imagine if you were running a real server, instead of the three nodes, you had tens or even hundreds of nodes, both for the proxy setup and the shard, one shard going down, taking down your entire cluster. We're going to root cause this. Since the client reported zero availability, so I'm going to just zoom in to the client to understand what is going on. This is what the healthy state looks like. We follow the best practices. The client has to connect to the proxies. You need connections, so you need a connection pool. Do not create connections all the time, take that overhead. This connection pool is capped to five connections. Then these five connections are basically serving the request for the shard_1 or whatever request comes up. Anything more would get throttled. Requests to the shard_1 start to take longer. Because of this, the request to the other shards, they get starved.

Now the entire connection pool becomes occupied with shard_1 requests, because they were slow, so they were not freeing the connection sooner. This is what leads to the outage. As you can see, all my requests to shard_1 have taken over the connection pool on the client side. Shard_2 and shard_3 are not getting any requests. They're all being rejected. This is how a single shard can turn the entire system into a full outage. To summarize what we learned with these proxied architectures. We spent $700 to achieve 1 million QPS with low tail latency. Then we also saw how one shard crashed the whole system.

Part 3: The Microsecond Playbook - The Strategic Path to µs Efficiency and Resilience

Our last part is about microseconds. Let's get deeper into the microseconds architecture. There is a quote from Dr. Werner Vogels, which reminds me, we always say that we have done it this way, so we would want to look for ways to question the current system, to constantly optimize and improve our systems. Just like the space shuttle where we got rid of the wings by questioning the runway landing requirement, getting to the capsules which were much more efficient, I'm going to do the same exercise with these proxies. I'm going to remove these proxies. Maybe they are critical to some use cases where you would definitely need them. For our discussion, I'm going to remove them and see what happens. After the proxies are gone, our clients are able to access the servers directly, but you have multiple servers, so you need to know where the data is.

What you need is a smart client which can understand the topology. The underlying Valkey cluster has a way for the client to learn the topology and route the requests accordingly. What we are going to do is we are going to do an exercise of the latency, price, performance, reliability aspects, but in the reverse order. We're going to start with reliability, then talk about cost, and then finally talk about latency. How we did with the proxied architecture, in the same direct access architecture, we are going to inject slowness in this shard. Use the same script, exactly the same setup. This time what you see is that availability is not zero, like earlier. It's partial outage. It hovers between 60% to 80%, which roughly is two-third of our traffic as expected. When shard_1 failed, my availability reported is 66%. Now let's zoom in into the client, like how we did with the root causing to understand what is going on here.

When your client is connecting directly to the shards, it's as simple as you have different connection pools, one per endpoint. Each connection pool has a bunch of connections to send requests to that shard. Then what happens is when shard_1 gets to slow down, it occupies all the connections in that pool, in the respective pool, and then more requests come up, even more requests, they start to get throttled. What's important is the shard_2 and shard_3's requests are all protected, they are isolated. The fault isolation, what you have on the shard, because the shards are independent of each other, now extends to the client side where the other requests are not impacted. This reminds me of a bulkhead pattern. This is a pattern, goes back to ancient Greece times with ships where they built these bulkheads. They're still there in practice in different shapes and forms. These bulkheads are basically like compartments. They are for the structural integrity of the ship. One other important aspect they serve is fault isolation, similar to our sharded architecture where one part gets damaged, it's not like it will sink the entire ship. It has some impact, but the ship still continues to float.

We just covered the reliability where we saw how with the proxied architecture, one slow shard takes down the whole cluster, and how in the direct access architecture we were able to isolate that fault as expected. Next on our list is price, performance. For price, performance, we have a single Valkey shard, no proxies. We are going to run our benchmark, which is able to achieve 1 million QPS. This Valkey server is able to achieve 1 million QPS without any proxies. The Valkey server just costs $230, like what we are basically able to drive here. This is our latency numbers with our Valkey setup. We have about 567 microseconds at the tail latency instead of the 2+ milliseconds. This is at 1 million QPS.

Summary and References

To summarize and put it all together, we have the cost, which dropped to one-third without the proxies to deliver the same 1 million QPS. We have four times higher latencies with the proxies set up at the tail. Instead of 600 microseconds with Valkey server, we have about 2.4 milliseconds. Here are the references I used for this talk. We basically covered the space shuttle, what you'll see here, Dr. Werner Vogel's Laws of Frugal Architect, and the Valkey's 1 billion RPS innovation. I didn't get to go deeper there, but you can read about it with these references.

Key Learnings

The last slide is about the key takeaways, how we went into the direct access architecture where we understood how the network hop, the 300 microseconds was taking out almost all the latency out of the 360 microseconds for p50. We saw how the architecture impacts the cost, the $700 versus just the $230. How a single point of failure in the proxied architecture goes away with the direct access architecture. Finally, Valkey server is so efficient because of all the community contributions behind the scenes which help us drive this infrastructure cost to be lower.

See more presentations with transcripts