Benchmarking Beyond the Application Layer: How Uber Evaluates Infrastructure Changes and Cloud Service Units

Uber has detailed Ceilometer, an internal adaptive benchmarking framework designed to assess infrastructure performance beyond application-level metrics. The system can leverage repeatable benchmarks with loads close to production environments, assisting Uber in certifying new cloud service units, validating infrastructure changes, and measuring efficiency improvement initiatives. As the heterogeneity of infrastructure continues to increase across different cloud service providers and hardware generations, Uber built Ceilometer to provide consistent data-driven performance signals across various environments. At a scale like Uber's, infrastructure benchmarking has historically been a decentralized manual process. Engineers often relied on one-off scripts, running tests individually and comparing results through spreadsheets, making it difficult to reproduce results or correlate performance metrics across different teams. Ceilometer replaces this approach with a centralized platform that can automatically orchestrate benchmarks, execute tests, ingest results, and analyze them, allowing for standardized comparisons across different servers, workloads, and environments. Ceilometer employs a distributed system architecture to coordinate the execution of benchmarks on dedicated machines. To reflect the behavior of real workloads, tests run in parallel, with raw outputs stored in persistent Blob storage. After the test results are validated and standardized, they are imported into Uber's centralized data warehouse, where they can be queried and analyzed alongside production metrics. According to Uber engineers, this design enables them to identify performance degradation and configuration inefficiencies or hardware-level discrepancies through a unified data model. https://static001.geekbang.org/infoq/4d/4dc2a6b15e30d86e496aa3b64be8ea44.png Ceilometer architecture diagram (Image source: ) The framework supports a wide range of workload types. To reflect the performance of CPU, memory, network, and storage, synthetic benchmarks such as , , and can be used. For stateful systems, Ceilometer integrates with Uber's platform to benchmark database workloads under real conditions. For stateless services, Uber's framework can be used for evaluation, providing adaptive load testing that simulates production traffic patterns. One major application scenario for the framework is server specification and cloud service unit (SKU) certification. Hardware vendors and cloud service providers can run Ceilometer's benchmarking suite in their own environments and share results, allowing Uber to assess expected performance before new SKUs go live. Another key application scenario is infrastructure change validation, where targeted benchmarks help isolate regressions introduced by software upgrades, kernel changes, firmware updates, or configuration adjustments The test results of Ceilometer can be compared across time periods, environments, and workload types, providing engineers with a clearer view of how infrastructure changes impact system behavior, rather than just relying on superficial application-level metrics. In a statement, a senior engineering manager and platform architect from Uber's infrastructure team wrote: As Uber's infrastructure continues to evolve, the company is expanding Ceilometer to gain deeper and more proactive insights. Planned enhancements include: integrating AI and machine learning for predictive regression, root cause identification, and resource scaling optimization; supporting a broader ecosystem, including emerging technology stacks and infrastructure paradigms; advanced anomaly detection for faster identification of unexpected performance deviations; and component-level utilization metrics to provide fine-grained visibility into CPU, memory, storage, and network behavior. Uber engineers also plan to leverage the framework for canary continuous validation testing, regularly automating benchmark tests and alerting the team when performance metrics exceed thresholds, supporting faster and more reliable infrastructure decision-making