Uber's Success with Remote Development: Enhancing Productivity and Performance

Uber's codebase in 2017 was fragmented and distributed across thousands of repositories with 10+ programming languages, 4000+ services, 500+ web apps, 9+ build tools, and 6+ configuration tools. There were separate repos per microservice and app. This fragmentation resulted in several developer pain points, including dependency management issues, library version fragmentation in production, build tool fragmentation, support for various tools and frameworks, collaboration, and friction in code sharing.

As a global company, Uber has always had a distributed workforce. In the past, this meant that some employees worked from Uber offices worldwide while others worked from home or co-working spaces. However, the COVID-19 pandemic has forced the company to fully embrace remote work, with all non-essential employees working from home.

Uber engineers developed their own remote development environment solution to address challenges and enhance developer productivity that arose from the use of a code monorepo.

Devpod is a remote development environment that provides the best possible developer experience for Uber's large-scale monorepos. It runs as a container on Kubernetes, with the necessary tooling and compute resources to enable seamless development.

Devpod is integrated with multiple IDEs, including JetBrains IDEs and VS Code, and is designed to improve performance and productivity. It provides faster Git, build, and IDE experience by utilizing beefy cloud resources, including up to 48 cores and 96 GB of RAM per environment.

In addition to improved performance, Devpod also offers nearly zero setup and maintenance. It comes with a pre-configured and pre-installed toolset and is updated overnight with the latest tooling and security upgrades. This eliminates the need for developers to worry about keeping their environment up to date and allows them to focus on their work.

Devpod also offers enhanced security. It provides a controlled toolchain with a hardened set of tools, and security scans can be performed on the image before it is promoted to the stable version.

In addition to the benefits mentioned above, Devpod offers a number of other advantages for Uber's developers. These include:

Overall, Devpod is an essential tool for Uber's remote development efforts. It allows the company to provide its distributed workforce with the resources and support they need to be productive while also improving performance, security, and overall developer experience.

Uber's Devpod team has faced a number of challenges in creating the perfect environment for engineers. These challenges include:

Engineers use IDEs daily, so a remote environment without a great IDE experience is not successful. To address this, Uber has provided multiple different IDE options for Devpod, including VS Code remote, JetBrains IDEs via ssh, and Vim and Emacs via terminal. The team has also focused on fine-tuning the hands-on experience by pre-loading indices, pre-configuring settings, and pre-installing tools, extensions, and custom add-ons.

However, the IDE experience has been one of the biggest challenges for Devpod, particularly in the early days when latency and stability issues with web-based IDEs were a problem. Uber is constantly working to improve the performance and intelligent features of these IDEs, including by contributing back to the open-source community.

To ensure that Devpod environments do not require manual maintenance, Uber has automated the process of upgrading them with the latest tooling and security updates during non-working hours. Uber has also implemented a gradual rollout of new versions to reduce the risk of a bug. Engineers can choose from one of four release cadence channels: stable, release candidate (RC), dev, or none.

One of the challenges faced by the Devpod team has been balancing cost efficiency with performance. To justify building an in-house solution, the team continuously tracks and improves the cost per Devpod and ensures that the price/performance ratio is better than that of over-the-shelf alternatives.

To ensure efficient use of compute and storage resources, the Devpod team has implemented several improvements:

Overall, the Devpod team has been able to keep the price significantly lower than the competition while still providing a high-performance environment for Uber's developers.

So far, Devpods have effectively reduced the local Git command performance and provided better performance for longer, more complex builds. As a result, there has been an organic adoption of Devpods across the organization, with over 60% of Uber software engineers using the remote development environment as of November 2022.

Devpods have consistently provided p95 Git status times below 4 seconds, while the Git status performance on laptops has improved with implementation of the filesystem monitor and sparse checkouts.

Devpods have also consistently provided a 2.5X improvement over laptops on average for longer, more complex builds and a 1.8X improvement over laptops for average builds.

According to NPS feedback from Uber developers, Devpod has consistently been identified as a major productivity booster. In surveys, developers cited the significant speed improvements in builds as their primary reason for loving Devpod. They reported feeling more productive and efficient while using the remote development environment.

In summary, the Devpod team at Uber is working on many improvements and new features for their remote development environment. These include reducing the initial creation time for new Devpods, introducing short-lived Devpod environments for specific tasks, improving maintenance and upgrade processes, optimizing cost and performance, providing a seamless IDE experience, implementing a centralized build farm for all environments, allowing teams to customize Devpod configurations, enabling seamless file transfer between laptops and Devpods, and adding Android emulator support for Devpods.