December 8, 2023

PBX Science

VoIP & PBX, Networking, DIY, Computers.

Why does Google update its internal Linux distribution for hundreds of thousands of devices?

3 min read

Why does Google update its internal Linux distribution for hundreds of thousands of devices?


Why does Google update its internal Linux distribution for hundreds of thousands of devices?


Google uses many different operating system platforms internally, including Linux.

More than 15 years ago, Google chose to build its internal Linux distribution Goobuntu based on Ubuntu LTS , mainly because Ubuntu was user-friendly, easy to use, and offered many great features. In addition, LTS will receive 2+ years of security updates from Canonical.


According to the official introduction , Goobuntu is a Linux distribution that Google has been using for a long time.

But the two-year update cycle of Ubuntu LTS means that Google has to upgrade every machine in its more than 100,000 devices before the operating system EOL.

This is a very difficult and time-consuming job, after all having all engineers configure their workspaces from scratch every two years is a serious blow to productivity and an irresponsible option from an economic point of view.


Google also faces the challenge of major version upgrades of software packages during each operating system life cycle, as this may require major changes to the software configuration.

To automate this process, Google engineers wrote an unattended upgrade tool to deal with many common problems. This “automated upgrade” means most Googlers don’t need to manually upgrade by reinstalling their machines and recreating all configurations.

To do this, though, Google needs to thoroughly test the upgrade process and check that all major packages that have changed continue to work (in Ubuntu, upgrades between major versions can be as many as thousands of packages).

Sometimes it’s difficult to provide automation in the event that a package is deprecated and engineers have to decide how to move forward.


It is said that it usually takes a year for Google to complete all Goobuntu upgrades, and the whole process is a huge pressure for the team.

And many times they encounter bugs that have been fixed upstream, but those improvements are never merged into the LTS version used.


So Google turned to a rolling update distribution, but instead of the well-known rolling update distribution Arch Linux, it built gLinux Rodete (Rolling Debian Testing) based on the Debian testing branch.


Why does Google update its internal Linux distribution for hundreds of thousands of devices?


Debian was chosen because of its large community and software repositories, and access to existing in-house packages and tools in Debian format.

And Ubuntu is also based on Debian, making migration easier and smoother in the past.


The Debian stable branch also releases major updates almost every two years, but its beta branch is rolling.

Rolling updates means preventing new releases from breaking existing workflows as much as possible, and to manage all these complex tasks of building all upstream packages from source, Google built a workflow system called Sieve.

Ensure that neither core components nor developer workflows are broken by newer versions of packages by running a virtualized test suite.


Google said the last version of Goobuntu was based on Ubuntu 14.04 LTS (codenamed Trusty).

The development of Rodete started in 2015, and in 2017, Google started migrating machines to Rodete, and by the end of 2018, all the migrations were completed.

The last machines running Goobuntu were shut down in early 2019.

In mid-2020, Rodete has completely kept up with the release of Debian Bullseye.

In the future, Google plans to work more closely with upstream Debian and contribute more internal patches to maintain the Debian package ecosystem.


Copyright © All rights reserved. | Newsphere by AF themes.