Building an Android Firmware Build Machine

  • Wed, Mar 10, 2021
  • 8-minute read

Most commodity desktop computers won’t be able to build the Android Open Source Project in less than 4 hours, and most folk are on a budget when they build a new machine, so I’ve put together a few tips which will help you build a machine capable of building the AOSP in two to three hours.

WWGD?

The first question that most folk ask when considering a machine for building the AOSP is “What Would Google Do?”. Fortunately I spent over a year at Google working on Android and the short answer is; Spend a tonne of money on a very high-end workstation.

Android engineers at Google working on the OS usually get specialised workstations which cost over US$10,000. If you have that kind of money to spend, feel free to head to your nearest Workstation provider (e.g. Lenovo), push all the customisable hardware options to “Max”, drop in ~200 GB of RAM, and enjoy.

If, on the other hand, you’d rather look for something more affordable, read on.

1. Cores are Core

The more cores the better.

To give you an example of how important cores are to an AOSP build I optimised the build for two different machines;

Machine #1 : 6 core i5-9600k CPU, 64GB RAM, NVMe SSD

Machine #2 : Dual 6 core Xeon X5675, 24GB RAM, SATA II SSD

No matter how much I tweaked the build on Machine #1 it was at least 15% slower than Machine #2 even though machine #2 has a slower disk, a lower per-core max speed and less RAM. On these machines 15% equates to about half an hour (yes, you will be looking at multi-hour builds).

You don’t have to break the bank; In my case Machine #2 is actually a 2010 Mac Pro (~US$1000 on eBay) with a Samsung Evo 840 SATA SSD, and a CPU upgrade (Xeon X5675). You can buy similar specification machines that are non-Apple for even cheaper, and if you’re willing to make space for a rack mountable server bought on eBay you can get a 12+ core machine for a lot less than a Core i5-9600k desktop.

So given the choice of two machines with similar prices, go for the one which has more cores.

(The only reason I used a Mac Pro is because I’ve had one for 10 years, so I already had the hardware available)

2. RAM

You’ll need about twice the number of cores in GB of RAM, with a minimum of 16 GB.

A little extra RAM can help, but a lot extra won’t. I’ve tried a few different RAM sizes on my 12-core build machine and while 16 GB RAM was definitely too small for a 12 core machine, any improvement I got from going above 24 GB was close to meaningless. Doubling the RAM from 24 GB to 48 GB in Machine 2 only gave me a less than one percent drop in build times, and similarly bringing Machine 1 down to 16 GB only gave me a low single digit percentage increase.

3. Disks

You’ll need an SSD… preferably two.

There’s a lot of options around buying SSDs, and I’ve already gone down the SSD hardware rabbit hole so you can avoid some common pitfalls when you make your choice. The benefit of getting two SSDs is to split the I/O load between them because compiling the AOSP is a very disk intensive operation. One SSD will hold the AOSP source code, the other will hold the compilation output.

While the “little extra” RAM I mentioned earlier can help when being used as a disk buffer, the build rarely visits the same file twice, so there’s little performance benefit from a read cache beyond reducing the time it takes to load the core tools.

In terms of sizing you should look at 250 GB or more for both. If you want to stay with a single drive then it’ll need to be at least 500 GB. If you want to get one larger drive, and a smaller one, you should make sure your system is configured to use the larger disk for the compilation output to avoid the performance issues of SSDs when they start filling up.

4. OS

Linux.

While it’s possible to build the AOSP on a Mac, the performance tends to be worse than building on Linux, so I’d always advise you to have a Linux machine available. I used to build on macOS, but, after a couple of years of playing catch-up with Xcode I made the move to Linux and haven’t looked back.

My personal preference at the moment is Debian 10.

5. During the Linux setup

There are a few extra things you can do during the installation of Linux which will help the overall performance of your build;

SSD Over provisioning

During the setup, when it comes to partitioning your disks, leave 10% unused. SSDs' use a variety of techniques to improve performance, and having a 10% unused buffer for over provisioning will, in many cases, improve the performance and the lifetime of the SSD.

Create a separate partition(s) for the source code and build output.

The source code comes from a source control system, and so can be easily recreated and not something that the “last access time” is important for, and similarly the compilation output is easily re-creatable, so we can make some tweaks to how we use those partitions which we wouldn’t want to make for the parts of the disk the operating system and other programs use.

If you have a two disk setup then you can create a separate partition on your source code hosting disk, and then use all of your build output hosting disk.

If you’re using one disk you only need to create a single separate partition which will host both the source code and build output.

Use ext4 as the file system.

ext4 is a reliable, trusted, and well tested file system. I’ve tried other filesystems which are designed to be more flash-friendly (e.g. F2FS), there wasn’t a noticeable improvement in build times, and most alternatives had warnings about the state of current maintenance, SSD wear, or other similarly concerning things.

6. Customise how Linux uses the disk(s)

Once you’ve installed linux, and it’s up and running, there are a couple of things you can do to improve your performance; turn off ext4 journaling and access time recording for the source code and build output.

With separate partitions for the source code and build output this is easy to do. If, however, you’ve not created separate partitions I would advise against following these steps because the impact on other programs could cause problems.

To make these changes you need run a command and change a system file for each partition. So if you have your source code checked out to /dev/nvme0n1p1, and your build output is going to /dev/nvme0n2p1, you’ll need to do the following for each one.

As the root user run the following commands;

$ mkfs.ext4 -O ^has_journal /dev/nvme0n1p1 
$ mkdir /mnt/aosp-disk-1

(You can alter the directory in the mkdir command to something more memorable if you want to, the important thing is to use that name consistently.)

Once you’ve run these commands you should edit /etc/fstab and create a line which looks like the following;

/dev/nvme0n1p1  /mnt/aosp-disk-1      ext4    noatime,nodiratime       0       0

There will be lots of similar lines in that file, don’t change them, but you can use them to guide you on the formatting of the file.

The options we’re setting are;

  • noatime - Don’t write an update to the source code hosting partition each time a file is read.
  • nodiratime - Same as noatime, but for directories instead of files.

You should now repeat the above steps for your other partition (/dev/nvme0n2p1) making sure to change aosp-disk-1 to something else (e.g. aosp-disk-2).

Now you can run the following to mount your disks;

$ mount /mnt/aosp-disk-1
$ mount /mnt/aosp-disk-2

7. Two disk setup only: After checking out the source code

Once you’ve followed Googles instructions on checking out the AOSP source code into your source code hosting partition you will, before you start your build, need to create a link from the out directory in the source code checkout to a directory on your build output disk.

If you have checked out the AOSP source code into /mnt/aosp-disk-1/aosp-checkout/ you can run the following;

$ cd /mnt/aosp-disk-1/aosp-checkout/
$ mkdir /mnt/aosp-disk-2/aosp-out/
$ ln -s /mnt/aosp-disk-2/aosp-out out

Then start your build as normal.

8. Before Every Big Build

SSDs have a command called TRIM which allows the operating system to tell the SSD which parts of the disk it isn’t using and can be used for performance optimisations. TRIM can improve the performance and lifetime of SSDs, and is particularly beneficial before you write large amounts of data to a disk, as you would do in a large build.

While it’s possible to have the OS automatically tell the SSD to TRIM a disk, the OS can’t predict the future, and so doesn’t know when you’re going to kick off a build which is likely to write a lot of data. TRIM can also be slow, so you ideally don’t want to have a TRIM start while you’re in the middle of a build, so, to save the Linux developers needing to write a future prediction algorithm, I manually trigger a TRIM operation on all filesystems before I run a large build by running the following command as root;

$ fstrim -a

That’s all folks

Hopefully this guide will help you get a machine setup to build the AOSP efficiently on a reasonable budget. If you have questions or feedback you can find me on Mastodon, GitHub, and Twitter.

AOSP