We reimagine the way the world moves for the better.~ Uber Mission Statement
Background
Load testing is a key step in production readiness which helps ensure the service reliability and behavior at scale. By speeding up the load tests and achieving an optimal traffic ramp-up curve without manual configuration we can automatically set optimal initial RPS and quickly bring the service into the metastable state.
Ballast
Ballast is an adaptive load test framework that leverages traffic capture using Berkeley Packet Filter (BPF) and replays the traffic using a PID Controller mechanism to adjust the number of requests per second (RPS) to each service. Ballast removes the toil of writing, running, and supervising load tests, improves load test coverage, and performs continuous load testing, providing insight into service capacity and improving deployment safety on an always-on basis. You can read about Ballast in detail here.
Adoption
Ballast was adopted by a number of high throughput Uber services.
Challenges
Ballast was originally designed for Maps/ AI. However, with increasing adoption among several other services one of the major challenges that came up was that one PID values does not fit all services. The current implementation of Ballast holds the static default values for all services which are not optimal. There is value in terms of improved accuracy by having variable PID values for different services because each service behaves differently.
Project Background
My project was called the PID Auto Tuner. The aim of the project was to automatically find the better default P, I, D value for a given service instead of using the same default value.
What I achieved
I majorly worked on Autotuning the value of P throughout my internship. In addition, I worked on a few other tasks like user input validation, creating a validator to validate the payload format and creating a gateway connecting Ballast and CRE (Capacity Recommendation Engine) for auto-tuning the value of P.
Learnings
- To write code using Golang
- Creating a new grpc client for CRE
- Creating a Design Doc
- Writing unit tests
- To test and deploy code into the Uber Infrastructure
- Testing deployed code in the production environment and identifying and fixing bugs
- Following standard coding practices
- Refactoring code to improve code quality
What Lies Ahead
- Auto-tuning I, D values of the PID Controller
Acknowledgements
This project would not have been possible without the support and guidance of my mentor and manager:
- Minglei Wang, for helping me out with the minutest and silliest of questions, motivating me and for always having my back.
- Haarith Devarajan, for being so supportive and encouraging throughout the internship.
A huge shoutout to the entire Maps Production Engineering Team especially Ranjib Dey for the best on-boarding experience, guidance and such a memorable summer. I am looking forward to working with y’all again soon!
Foreword
It was a refreshing experience that the internship had an in-person option too. This was the first time I visited San Francisco (more on that later :P). I got to meet so many other interns which made going to the office even more fun. Also loved the facilities the Uber Office had: a gym, espresso bar, smoothie bar, terrace, lunch, snacks, you name it and it was there! Overall, it was a very productive summer and one of the best internship experiences that I’ve ever had. Looking back, I have a suitcase full of memories I’ve brought home with me which I will always cherish.
