Expedia Uses AI Driven Service Telemetry Analyzer to Accelerate Incident Investigation

Expedia Group has introduced Service Telemetry Analyzer (STAR), an internal AI-assisted observability platform that helps engineers investigate production incidents by analyzing service telemetry and generating structured root cause assessments. The system combines operational metrics with large language models (LLMs) through predefined diagnostic workflows, aiming to reduce the time engineers spend identifying the source of service degradation while keeping humans responsible for validation and decision-making.

Rather than adopting autonomous AI agents, the platform follows a deterministic workflow in which telemetry is collected, analyzed using domain-specific prompts, consolidated into intermediate findings, and summarized into a final report containing potential root causes and recommended next steps.

Describing the project's goal, the team wrote,

Our objective with this service was to minimize the time to know (TTK) and time to recover (TTR).

STAR Architecture (Source: Expedia Blog Post)

The platform is implemented as a FastAPI application that integrates with Datadog to retrieve service metrics and an internal generative AI gateway that manages authentication and access to LLM providers. The workflow uses prompt chaining, allowing multiple specialized analyses to be performed before producing a consolidated diagnosis. Expedia said the current implementation does not use capabilities such as function calling, retrieval-augmented generation (RAG), memory, or autonomous tool use, instead relying on predefined workflows to generate consistent analyses.

STAR focuses on standardized infrastructure telemetry collected from Kubernetes-based services and JVM applications. The system analyzes metrics including request throughput, latency, HTTP, gRPC, and GraphQL error rates, CPU and memory utilization, container restart events, Kubernetes readiness and liveness probe failures, Java heap utilization, and garbage collection activity. Expedia explains that infrastructure metrics provide a consistent view across services developed using different programming languages and frameworks.

As the platform evolved, Expedia replaced FastAPI background tasks with a Celery-based asynchronous architecture using Redis as both the message broker and result backend. The engineering team states that most processing consists of I/O bound operations involving telemetry retrieval and LLM requests. The asynchronous execution model allows STAR to process multiple analysis tasks concurrently while accommodating rate limits imposed by Datadog and the company's internal AI gateway.

The company reports that STAR has been used to support production incident investigations, post-incident analysis, Kubernetes troubleshooting, and JVM memory diagnostics. Engineers review the generated findings before acting on recommendations, making the platform an assistive tool rather than an autonomous operational system.

The blog also describes ongoing work to expand the platform. Planned enhancements include incorporating service dependency information, additional operational metadata, Model Context Protocol (MCP) based tool integrations, and conversational interfaces. Expedia is also evaluating STAR as part of its chaos engineering practices to help analyze the results of controlled failure experiments. Prompt management, tracing, and evaluation are currently supported through Langfuse, while system performance is assessed using subject matter expert reviews and user feedback.