How an old server machine is serving Android developers

androiddev_excuse

Yeah, every Android developer knows what it means. And it doesn’t stop there. Slow builds drain the battery very fast and fingers suffer from burning keyboard keys. Even colleagues around are distracted by your noisy CPU fan. We are dreaming off better hardware during a build. Even the best MacBook still stuffed into 4C/8T CPU + 16 GB RAM. And Xeon based laptops hardly fit into backpacks. What if we could move the build process to a remote machine?

But we can! Thanks to mainframer community for the solution. Our SRE team picked a retired multiprocessor server with 16C/32T + 252 GB and set up a mainframer there. And they did a great job!

All Android developers are using a single mainframer machine. Servers are managed by Chef using the chef-android-sdk recipe. The tricky part here was to make a separate Android SDK per developer. First, Chef installs a system copy of the android-sdk, then scans for users without android-sdk in their HOME dir and makes a copy if needed. Also add env variable ANDROID_HOME:

echo 'export ANDROID_HOME=$HOME/android-sdk/' >> /home/some_user/.bashrc

All developers eventually get their own copy of android-sdk.

Benchmark

Setup:

  • Local machine: MacBook Pro, Mid 2014, i7 @ 2.2GHz, 16 GB RAM, SSD
  • Mainframer: 2 x E5-2680 @ 2.70GHz, 252GB RAM, SSD
  • Gradle: 4.2.1
  • Android Gradle plugin: 2.3.0
  • Build tools: 26.0.1
  • Kotlin: 1.1.51

I use gradle-profiler5 for the benchmark with following scenarios:

clean_build {
    tasks = ["assembleDebug"]
    cleanup-tasks = ["clean"]
    warm-ups = 3
}

warm {
    tasks = ["assembleDebug"]
    warm-ups = 3
    apply-abi-change-to = "path/to/RealFile.java"
}

By default all benchmarks are run 10 times.

Result

cold

As you see, there is no much win here. Average cold build time increased from 184 to 189 seconds. The sync between laptop and mainframer takes up to 10 seconds each time.

warm

Warm build performs much better: average build decreased from 154 to 114 seconds. Almost 26%. Warm builds require no more than 5 seconds sync time. And the build time on mainframer is more predictable.

How to improve more

Our mainframer server has tons of cores, but the single core is slower than the newer core. Very few Gradle tasks are multithreaded. And we can help Gradle utilize more cores. Currently our modules look like this:

gradle_structure

Orange and green lines are building independently on separate threads. We have plans to split our code into more vertical lines and utilize more server power.

Conclusion

Mainframer gives us a faster, more enjoyable environment and saves tons of money for new hardware. It not only reduced build time, but also allowed development without having to charge the battery. But it is not free lunch. Some trade-offs:

  • Needs an internet
  • Unstable network leads to random long-running sync
  • Android Studio sync still runs on local Gradle
  • Switching build variants in Android Studio doesn’t affect mainframer build
  • File paths are mapped to the server, not local.

The last one could be partially fixed by using mainframer plugin.