Uber faced challenges with its large, fragmented codebase distributed across thousands of repositories with multiple programming languages, build tools, and configuration tools. To improve developer productivity and streamline the development process, Uber implemented a remote development environment called Devpod. Devpod has resulted in faster Git and build times, improved security, and nearly zero setup and maintenance for developers. In this article, we will delve into the benefits and challenges of implementing Devpod at Uber, and how it has revolutionized the software development process.
Remote Development at Uber
Uber's codebase in 2017 was fragmented and distributed across thousands of repositories with 10+ programming languages, 4000+ services, 500+ web apps, 9+ build tools, and 6+ configuration tools. There were separate repos per microservice and app. This fragmentation resulted in several developer pain points, including dependency management issues, library version fragmentation in production, build tool fragmentation, support for various tools and frameworks, collaboration, and friction in code sharing.
As a global company, Uber has always had a distributed workforce. In the past, this meant that some employees worked from Uber offices worldwide while others worked from home or co-working spaces. However, the COVID-19 pandemic has forced the company to fully embrace remote work, with all non-essential employees working from home.
Uber engineers developed their own remote development environment solution to address challenges and enhance developer productivity that arose from the use of a code monorepo.
What is Devpod?
Devpod is a remote development environment that provides the best possible developer experience for Uber's large-scale monorepos. It runs as a container on Kubernetes, with the necessary tooling and compute resources to enable seamless development.
Devpod is integrated with multiple IDEs, including JetBrains IDEs and VS Code, and is designed to improve performance and productivity. It provides faster Git, build, and IDE experience by utilizing beefy cloud resources, including up to 48 cores and 96 GB of RAM per environment.
In addition to improved performance, Devpod also offers nearly zero setup and maintenance. It comes with a pre-configured and pre-installed toolset and is updated overnight with the latest tooling and security upgrades. This eliminates the need for developers to worry about keeping their environment up to date and allows them to focus on their work.
Devpod also offers enhanced security. It provides a controlled toolchain with a hardened set of tools, and security scans can be performed on the image before it is promoted to the stable version.
Other Advantages of Devpod
In addition to the benefits mentioned above, Devpod offers a number of other advantages for Uber's developers. These include:
Improved productivity: Devpod's faster performance and zero setup time helps developers be more productive. It also allows them to work on projects in any of the main monorepos, with easy provisioning and a ready-to-use IDE.
Multiple remote environments per user: Developers can have multiple Devpod environments, which allows them to work on various projects simultaneously or collaborate with other developers in a shared environment.
Improved laptop battery life: Since Devpod runs in the cloud, it is not subject to laptop battery constraints. This can improve the overall battery life of developers' laptops.
Persistence: Devpods are persistent, which means that developers do not need to worry about losing their individual setup, files, or code changes. This allows them to continue their work on different devices and enables collaboration between multiple engineers in a single environment.
Overall, Devpod is an essential tool for Uber's remote development efforts. It allows the company to provide its distributed workforce with the resources and support they need to be productive while also improving performance, security, and overall developer experience.
Remote Development Challenges at Uber
Uber's Devpod team has faced a number of challenges in creating the perfect environment for engineers. These challenges include:
- Balancing performance and cost efficiency
- Providing automatic upgrades
- Ensuring uninterrupted work
- Pre-configuring IDE settings that work for everyone
Engineers use IDEs daily, so a remote environment without a great IDE experience is not successful. To address this, Uber has provided multiple different IDE options for Devpod, including VS Code remote, JetBrains IDEs via ssh, and Vim and Emacs via terminal. The team has also focused on fine-tuning the hands-on experience by pre-loading indices, pre-configuring settings, and pre-installing tools, extensions, and custom add-ons.
However, the IDE experience has been one of the biggest challenges for Devpod, particularly in the early days when latency and stability issues with web-based IDEs were a problem. Uber is constantly working to improve the performance and intelligent features of these IDEs, including by contributing back to the open-source community.
Keeping the Environments Up to Date
To ensure that Devpod environments do not require manual maintenance, Uber has automated the process of upgrading them with the latest tooling and security updates during non-working hours. Uber has also implemented a gradual rollout of new versions to reduce the risk of a bug. Engineers can choose from one of four release cadence channels: stable, release candidate (RC), dev, or none.
One of the challenges faced by the Devpod team has been balancing cost efficiency with performance. To justify building an in-house solution, the team continuously tracks and improves the cost per Devpod and ensures that the price/performance ratio is better than that of over-the-shelf alternatives.
To ensure efficient use of compute and storage resources, the Devpod team has implemented several improvements:
- Shutting down inactive environments: To save compute resources, an automated job periodically checks whether an environment has been used lately, and upon detecting an inactive one, its container is deleted.
- Rebalancing VMs: If two VMs are only used for half of their capacity, the load is moved to one machine, and the other one is shut down.
- Snapshotting disks of shutdown environments: When a Devpod container is shut down, the disk is no longer used. To save on storage costs, the disk is converted into a low-cost storage option until the environment is started up again.
Overall, the Devpod team has been able to keep the price significantly lower than the competition while still providing a high-performance environment for Uber's developers.
Impact of Devpods on Developer Productivity and Performance
So far, Devpods have effectively reduced the local Git command performance and provided better performance for longer, more complex builds. As a result, there has been an organic adoption of Devpods across the organization, with over 60% of Uber software engineers using the remote development environment as of November 2022.
Devpods have consistently provided p95 Git status times below 4 seconds, while the Git status performance on laptops has improved with implementation of the filesystem monitor and sparse checkouts.
Devpods have also consistently provided a 2.5X improvement over laptops on average for longer, more complex builds and a 1.8X improvement over laptops for average builds.
What Uber Developers Have to Say
Image credit: Uber blog
According to NPS feedback from Uber developers, Devpod has consistently been identified as a major productivity booster. In surveys, developers cited the significant speed improvements in builds as their primary reason for loving Devpod. They reported feeling more productive and efficient while using the remote development environment.
Uber's Future Plans for Remote Development
In summary, the Devpod team at Uber is working on many improvements and new features for their remote development environment. These include reducing the initial creation time for new Devpods, introducing short-lived Devpod environments for specific tasks, improving maintenance and upgrade processes, optimizing cost and performance, providing a seamless IDE experience, implementing a centralized build farm for all environments, allowing teams to customize Devpod configurations, enabling seamless file transfer between laptops and Devpods, and adding Android emulator support for Devpods.