CircleCI is popular among Android developers for several reasons: it’s quick to get started, fast to execute your builds with high parallelism, (whether native, cross- or multi-platform), and even supports running Android emulators right from CircleCI with our Android machine images.
This article will show you how to build and test Android applications for an example project on the CircleCI platform. The full source code is available on GitHub - CircleCI-Public/android-testing-circleci-examples, and you can find the working pipelines in the corresponding CircleCI project.
To get the most out of this article you should have
- Working knowledge of Android development
- Familiarity with Android tooling and testing frameworks
- Ability to use the Gradle build system
What’s available in CircleCI for Android
We have made a few tools available for you for building Android:
- Docker image
- Android machine image
- Android orb
Docker image
The Docker image is the main executor teams use. It runs in a Docker container and so is very fast to start, and contains everything you might need to get started building Android projects. The full list of preinstalled tools can be found in the documentation. The images also ship with variants that have Node.js, the Android NDK, and browser tools preinstalled.
Machine image
The biggest change is the new Android machine image. This is a virtual machine image, so takes a bit more time than the Docker image mentioned above, but it contains all the tools you may need to build your Android applications, including the SDK, Google Cloud tools for Firebase, Python, Node.JS, and Fastlane. The biggest change however is the support for nested virtualisation, which allows you to run Android emulators from within this machine image.
Android machine images support nested virtualization, which allow running the x86 emulators in CircleCI jobs. If you have been developing for Android since the early years of the platform, you may remember that emulators were very slow. Then x86 emulators started supporting Intel’s Hyper-V hypervisors, which lets the emulator use the resources of the host computer directly. Using resource directly like this makes the emulators able to run much, much more smoothly and quickly.
For a long time, support was confined to our local hardware. Limited support on CircleCI slowed down emulators and made it difficult to run UI tests. Nested virtualization solves this problem by adding support for fast emulation to a massively reproducible CI/CD environment, making all the benefits of that available.
Android Orb
Finally there is the CircleCI Android orb which provides easy access to both images as executors, as well as handy jobs and commands for installing SDKs, running emulators, caching, testing, and more. You can read more about what the orb offers in the orb docs at this link.
Introducing the Android CI demo project
I have created a demo project, which is a fork of Google’s own Android Testing Codelab sources. The Codelab is a simple TO-DO list manager application and contains a combination of unit tests and instrumentation tests, with and without the UI. The project is also available to view on CircleCI. If you open it in Android Studio, make sure to switch to the Project
view. Project view lets you view the .circleci/config.yml
file we will be reviewing.
The project has a single app
module, with the default debug
and release
variants. The test
directory contains unit tests, while the androidTest
directory contains instrumentation tests. The full CircleCI configuration is included, and I will be referring to it in this article. The configuration is located in .circleci/config.yml
.
Building and testing CI for the sample Android app
Our CI process must do three things:
- Run unit tests inside a virtual machine
- Run instrumentation tests on the emulator
- Build a release version of the app
We will run unit tests and instrumentation tests side by side. If both types of tests are successful, we can assemble the release build. We will also add a condition to continue on to the release build only if new work has been commited to the “main” branch, and only following successful emulator tests on every Android version from 23 to 30. When our build meets this condition, we will have solid assurance that our application will work on many different Android devices.
Note: As I mentioned before, everything from this point on will refer to the CircleCI configuration file in .circleci/config.yml
.
Using the CircleCI Android orb
The first step for creating our CI pipeline is to use the Android orb, which contains many of the steps already pre-written for you. The latest version is available in the orb docs.
After the orb is defined, the script specifies these jobs:
unit-test
android-test
release-build
One workflow, test-and-build
, is also included.
version: 2.1
orbs:
android: "2.1.2"
jobs:
unit-test:
...
android-test:
...
release-build:
...
# We'll get to this later
workflows:
test-and-build:
...
# We'll get to this later
Running unit tests
Our unit-test
job contains an executor from the Android orb: android/android-docker
. As mentioned this runs it in a Docker container which is extremely fast to start up.
In the steps
stanza, there are a few things to note. First, we run android/restore-gradle-cache
Gradle cache helps store our dependencies and build artifacts. After running the test command, we also run android/save-gradle-cache
and android/save-build-cache
to make sure subsequent builds run faster by using the cache. These all come from the Android orb, as indicated by the prefix android/
.
We could also modify what gets cached by passing the find-args
parameter to restore-gradle-cache
. A great advanced example can be found in this open source project.
The main build step happens in the android/run-tests
command, where we call ./gradlew testDebug
within a test-command
parameter. This command runs all unit tests for the debug
build variant. As a bonus, this command comes with a default retry value to help you avoid test flakiness.
jobs:
unit-test:
executor:
name: android/android-docker
tag: 2022.08.1
steps:
- checkout
- android/restore-gradle-cache
- android/run-tests:
test-command: ./gradlew testDebug
- android/save-gradle-cache
...
Running Android tests on the emulator
Our android-test
job is the one that uses the new Android emulator capability. The job uses the same Android machine executor as the build
job. All subsequent jobs will use the same executor, and the cache restoration steps are also the same.
Emulator requires us to run on a machine image, so we specified android/android-machine
as the executor. To improve build times, we specify the resource-class
as xlarge. You can also run it on a large
executor, but I have found that the lack of resources causes slower build times. I recommend that you do some experimentation on your own to find the right size executor for your project.
The next step is even shorter; we need just the android/start-emulator-and-run-tests
command. This command does exactly what it says: it starts the specified emulator and runs the tests. We pass in the test-command
again as the parameter (it uses the android/run-tests
command). The system-image
is the fully qualified Android emulator system image. We currently bundle our Android machine images back to SDK version 23. You could install more by using the sdkmanager
tool.
jobs:
android-test:
executor:
name: android/android-machine
resource-class: xlarge
steps:
- checkout
- android/start-emulator-and-run-tests:
test-command: ./gradlew connectedDebugAndroidTest
system-image: system-images;android-30;google_apis;x86
...
You could code all the emulator setup steps manually using the commands specified in the Android orb: create-avd
, start-emulator
, wait-for-emulator
, and run-tests
.
Storing the test results
There is more to testing than just running the tests, as we can get a lot of value from seeing exactly what has failed from right inside the CircleCI dashboard. For that we can use store_test_results
functions built into the CircleCI platform, which will show our passing (or failing) builds.
The steps differ slightly between unit tests and instrumentation tests, as each of their respective Gradle tasks stores the tests in a different place. For unit tests the tests will be in app/build/test-results
, and for instrumentation tests they will be in app/build/outputs/androidTest-results
.
We suggest you create a new Save test results
step, which uses the find
utility and copies all test results files into a common location - like test-results/junit
, and then store it from there. Also make sure to add the when: always
parameter to the step, so the step runs regardless of whether the tests above succeed or fail.
Storing unit tests
For this project, we want the XML based results. Calling store_artifacts
makes those results parseable on the CircleCI platform. You can also store the HTML output if you prefer.
...
- android/run-tests:
test-command: ./gradlew testDebug
- run:
name: Save test results
command: |
mkdir -p ~/test-results/junit/
find . -type f -regex ".*/build/test-results/.*xml" -exec cp {} ~/test-results/junit/ \;
when: always
- store_test_results:
path: ~/test-results
- store_artifacts:
path: ~/test-results/junit
Storing instrumentation tests
There are different regex command for different test types. The rest of the steps to store test results are identical to the unit tests example.
- android/start-emulator-and-run-tests:
test-command: ./gradlew connectedDebugAndroidTest
system-image: << parameters.system-image >>
- run:
name: Save test results
command: |
mkdir -p ~/test-results/junit/
find . -type f -regex ".*/build/outputs/androidTest-results/.*xml" -exec cp {} ~/test-results/junit/ \;
when: always
- store_test_results:
path: ~/test-results
- store_artifacts:
path: ~/test-results/junit
Making a release build and storing the artifacts
For this project, we need to store the application we are building. To complete this part of the process, we use the release-build
job, which has two parts of note. A run
step calls ./gradlew assembleRelease
. Then, store_artifacts
points to where the built APK is. For a full CI/CD project, you could sign and upload that APK to a beta distribution service. You could even create a release on Play Store with Fastlane. Both of those activties are out of scope for this article, though.
jobs:
...
release-build:
executor:
name: android/android-machine
resource-class: xlarge
steps:
- checkout
- android/restore-gradle-cache
- android/restore-build-cache
- run:
name: Assemble release build
command: |
./gradlew assembleRelease
- store_artifacts:
path: app/build/outputs/apk/release
Running tests on multiple Android versions simultaneously
The Android platform evolves all the time, and each version has its own bugs and pecularities. I am sure you have encountered some of them. A good way to avoid undefined behaviour across different versions of Android is to test on as many of them as you can. The CircleCI job matrix feature makes it easier to run jobs on multiple versions.
To use the job matrix, add a parameter to the job you want to test on multiple versions. Our android-test
job does that using the system-image
parameter. We have also specified a default value for the parameter, so you can run the job without it.
To use the parameter, specify where you need its value by placing it within pairs of angle brackets - << >>
:
android-test:
parameters: # this is a parameter
system-image:
type: string
default: system-images;android-30;google_apis;x86
executor:
name: android/android-machine
resource-class: xlarge
steps:
- checkout
- android/start-emulator-and-run-tests:
test-command: ./gradlew connectedDebugAndroidTest
system-image: << parameters.system-image >> # parameter being used
To pass values to the parameter in the workflows, use the matrix
parameter of the job we are calling.
workflows:
jobs:
...
- android-test:
matrix:
alias: android-test-all
parameters:
system-image:
- system-images;android-30;google_apis;x86
- system-images;android-29;google_apis;x86
- system-images;android-28;google_apis;x86
- system-images;android-27;google_apis;x86
- system-images;android-26;google_apis;x86
- system-images;android-26;google_apis;x86
- system-images;android-26;google_apis;x86
- system-images;android-26;google_apis;x86
name: android-test-<<matrix.system-image>>
Making each run faster by caching
Alongside building our application, installing dependencies, and running tests, we often want to leverage cache as well. This allows us to do repeated tasks less often. Android and especially Gradle provides an excellent mechanism for this - the Gradle Build cache.
This lets you skip pre-built parts of the application.
Choosing what runs when
During development, we create many commits, and developers should be encouraged to push them to a remote repository as often as possible. Each one of these committs creates a new CI/CD build, though, and we might not need every test run on all possible devices for a single commit. On the other hand, if the main
branch is our primary source of truth, we want to run as many tests on that as possible, to ensure it is always working as intended. We might want to initate the release deployment only on main
, and not on any other branch.
We can use CircleCI’s requires
and filters
stanzas to define workflows that fit our process. Our release-build
job requires
the unit-test
jobs. That means the release-build
is put in the queue until all the unit-test
jobs have passed. Only then is the release-build
job run.
We can use filters
to set up logical rules for running a particular job only on a specific branch or git tag. For example, we have a filter for our main
branch that runs the android-test
with a full matrix of emulators. In another example, the release
build triggers only on main
, and only when its two required jobs have run successfully.
workflows:
test-and-build:
jobs:
- unit-test
- android-test: # Commits to any branch - skip matrix of devices
filters:
branches:
ignore: main
- android-test: # Commits to main branch only - run full matrix
matrix:
alias: android-test-all
parameters:
system-image:
- system-images;android-30;google_apis;x86
- system-images;android-29;google_apis;x86
- system-images;android-28;google_apis;x86
- system-images;android-27;google_apis;x86
- system-images;android-26;google_apis;x86
- system-images;android-25;google_apis;x86
- system-images;android-24;google_apis;x86
- system-images;android-23;google_apis;x86
name: android-test-<<matrix.system-image>>
filters:
branches:
only: main
- release-build:
requires:
- unit-test
- android-test-all
filters:
branches:
only: main # Commits to main branch
Improving your Android application CI process
The steps I have described in this article are only a beginning. There are many ways you can improve the flow. Here are just a handful of ideas:
Build once, run many times
If you are an experienced Android developer you know that the connectedAndroidTest
will always run the entire build process from the start. Using CircleCI, we could build the entire application and tests once, pass the artifacts down to following jobs, and simply run the tests on the emulators. This process could potentially save several build minutes per job run.
To make this happen, add three commandline steps in each emulator run: install the app, install the instrumentation app, and run the instrumentation tests. For our application, we would enter:
adb install app/build/outputs/apk/debug/app-debug.apk
adb install app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk
adb shell am instrument -w com.example.android.architecture.blueprints.reactive.test/androidx.test.runner.AndroidJUnitRunner
It is true that this code sends output to the the command line. That is not the tidy test output that Gradle provides. If you are motivated to make the output look better, this StackOverflow topic might help you.
Running on real devices
Emulators are one tool, but real world devices are often (sadly) much different. There is nothing that can replace a real-world device test. For that you can use a solution offering real devices in the cloud, like Sauce Labs or Firebase Test Lab.
Deploying to beta testers, and releasing the application
This build and test example project is just one part of a wider CI/CD story. You can take your CI/CD process much further. For example, you can automatically release new versions of your applications right to the Play Store. CircleCI has a handful of guides to help you. This tutorial is a great place to start. The official Fastlane documentation is another helpful resource.
Conclusion
In this article, I described building an Android application with CircleCI, and testing it. For our sample project, we used unit tests and a matrix of Android platform emulators, using the new CircleCI Android orb and machine images. We also touched on how to store and display test results, and how to orchestrate workflows to run tests will not give us the tidy test output that Gradle provides, and all output is on a portion of devices.
Do let me know if there is another Android topic you might want me to explore, either in an article or a live stream. Contact me on Twitter - @zmarkan.
Top comments (0)