As an Android developer, testing is an integral part of the app development process. By running tests you can verify your app’s functional behaviour, correctness, and usability before you release it to the public. CircleCI is a CI platform that supports running Android emulators from within an Android machine image. We will use CircleCI to set up automated testing for our project. Automated testing involves using tools that perform tests on the project for you, which is faster, repeatable, and gives you more actionable feedback about your app earlier in the development process.
This article describes how to automate instrumentation and unit tests for an open source Android project using CircleCI. I assume you already know how to create an Android app and write tests for your app. The cloned project and working pipeline can be found here.
Instrumentation Testing
Instrumented tests, also known as instrumentation tests, are tests that run on Android devices, whether physical or emulated. It involves testing the overall behaviour of an application by testing the interaction between different components. It checks how the application performs when different components work together, including user interfaces, APIs, and external dependencies. This type of testing helps identify issues that may arise due to interactions between components, such as compatibility issues or performance problems.
Unit Testing
Unit testing involves testing individual components or units of code in isolation to verify their functionality, usually a single class or function. In unit testing, each unit of code is tested separately from the rest of the codebase to ensure that it behaves as expected.
CircleCI
CircleCI is a hosted and automated solution for continuous integration (CI) builds. It uses an in app configuration file that uses YAML syntax. CircleCI is easier to set up and is hosted in the cloud. After learning about instrumentation tests, unit testing, and CircleCI, we can now learn how testing in CI/CD works.
Unit Testing and Instrumentation Testing in CI/CD
As a DevOps principle, it is a good practice to automate testing as part of the CI process to test your code against any bug. Automated tests can be triggered at various stages of your CircleCI pipeline, including development, staging, and/or production. An effective automated test runs immediately after changes are committed to the version control system's (for example, GitHub) staging or development repository. If the test meets the success criteria, then the committed change(s) may proceed to the next stage or environment. If the test failed, the committed changes are blocked from proceeding until those issues are resolved. By automating tests, teams can have confidence that code changes have been thoroughly tested and have met the success and quality standards, thereby, protecting the production environment against disruptions.
Setting up Automated Android Testing on CircleCI
Firstly, we will set up our config.ymlfile to run unit tests. I'll demonstrate two approaches for running instrumentation tests on CircleCI: using the No-orb and Orb examples. The No-orb example is a more complex process because you would have to write the steps yourself, while the CircleCI Android orb example is an easy process with many steps already written for you. There's no need to worry, the No-orb example is not as difficult as it seems because CircleCi offers clear documentation that will make setup simple. Please take note that you should only use one of the examples in your config.yml file. The two instances given here are merely for demonstration. Let's get started.
Create a circleci.yml File
At the root of your project’s folder, create a .circleci folder. Inside the folder create a config.yml file. Our workflow will be in this order:
- Unit Tests
- Instrumentation Tests
Running Unit Test
We will call our job unit-tests. The working_directory specifies the location of the project files. The docker image section specifies the Docker image to use. In this case, it uses the cimg/android:2023.02 image.
Here’s a full look at the unit test job:
jobs:
version: 2.
unit-tests:
working_directory: ~/project
docker:
- image: cimg/android:2023.02
steps:
- checkout
- run:
name: Run Unit Tests
command: |
./gradlew clean test
- store_test_results:
path: app/build/test-results
The steps section includes three steps: first CircleCI checks out the repository where the code is stored. Then, run unit tests using the ./gradlew clean test command, and store the test results at the path specified.
Running Instrumentation Tests
As I mentioned earlier, there are different approaches to doing this. We will cover two examples or approaches. The first example is the No-orb example.
Using the No-orb Example
Our instrumented test requires an emulator to run on a machine image. Below is a breakdown of the instrumentation tests job. The full code is provided below. We will add each step to the instrumented test job in the circleci.yml file.
jobs:
instrumented_test:
machine:
image: android:202102-01
resource_class: large
So in the instrumented test job, we specify the image: android:202102-01 as the machine image and set the resource_class of the machine as "large" to improve build time. Under the step section we have the following steps:
steps:
- checkout
- run:
name: Create avd
command: |
SYSTEM_IMAGES="system-images;android-29;default;x86"
sdkmanager "$SYSTEM_IMAGES"
echo "no" | avdmanager --verbose create avd -n test -k "$SYSTEM_IMAGES"
The - checkout checks out the code from the repository while the command under - Create avd installs the Android system image required for the emulator to run, and creates a new virtual device named "test".
- run:
name: Launch emulator
command: |
emulator -avd test -delay-adb -verbose -no-window -gpu swiftshader_indirect -no-snapshot -noaudio -no-boot-anim
background: true
The Launch emulator command launches the emulator and waits for it to start up. It runs in the background (background: true) so that other steps can execute in parallel.
- run:
name: Generate cache key
command: |
find . -name 'build.gradle' | sort | xargs cat |
shasum | awk '{print $1}' > /tmp/gradle_cache_seed
The Generate cache key command generates a cache key based on the contents of the build.gradle files in the project. This key is used to restore the Gradle cache later, to speed up the build process.
- restore_cache:
key: gradle-v1-{{ arch }}-{{ checksum "/tmp/gradle_cache_seed" }}
restore_cache step attempts to restore the Gradle cache from a previous build, using the cache key generated in the previous step.
- run:
# run in parallel with the emulator starting up, to optimize build time
name: Run assembleDebugAndroidTest task
command: |
./gradlew assembleDebugAndroidTest
assembleDebugAndroidTest command compiles and build the debug version of the Android test APK and runs in parallel with the emulator starting up, to optimize build time.
This step can be modified to suit your project needs. I modified the above☝️ step to this:👇
- run:
# run in parallel with the emulator starting up, to optimize build time
name: Run compileDemoBasicDebugJavaWithJavac task
command: |
./gradlew compileDemoBasicDebugJavaWithJavac
This step compiles the Java code in the project using the javac compiler. This runs in parallel with the emulator starting up, to optimize build time.
- run:
name: Wait for emulator to start
command: |
circle-android wait-for-boot
As the name implies, the Wait for emulator to startstep waits for the emulator to finish booting up and become available for testing.
- run:
name: Disable emulator animations
command: |
adb shell settings put global window_animation_scale 0.0
adb shell settings put global transition_animation_scale 0.0
adb shell settings put global animator_duration_scale 0.0
The Disable emulator animations step disables the emulator animations to speed up the testing process.
- run:
name: Run UI tests (with retry)
command: |
MAX_TRIES=2
run_with_retry() {
n=1
until [ $n -gt $MAX_TRIES ]
do
echo "Starting test attempt $n"
./gradlew connectedAndroidTest && break
n=$[$n+1]
sleep 5
done
if [ $n -gt $MAX_TRIES ]; then
echo "Max tries reached ($MAX_TRIES)"
exit 1
fi
}
run_with_retry
Run UI tests (with retry) runs the instrumented tests on the emulator, with a retry mechanism in case of failures. The connectedAndroidTest command is the main command which runs the tests.
- save_cache:
key: gradle-v1-{{ arch }}-{{ checksum "/tmp/gradle_cache_seed" }}
paths:
- ~/.gradle/caches
- ~/.gradle/wrapper
save_cache saves the Gradle cache for future builds, using the cache key generated earlier.
Let's take a look at the whole config.yml file, which includes both the unit tests and the instrumentation tests jobs:
jobs:
version: 2.
unit-tests:
working_directory: ~/project
docker:
- image: cimg/android:2023.02
steps:
- checkout
- run:
name: Run Unit Tests
command: |
./gradlew clean test
- store_test_results:
path: app\build\test-results
instrumented_test:
machine:
image: android:202102-01
resource_class: large
steps:
- checkout
- run:
name: Create avd
command: |
SYSTEM_IMAGES="system-images;android-29;default;x86"
sdkmanager "$SYSTEM_IMAGES"
echo "no" | avdmanager --verbose create avd -n test -k "$SYSTEM_IMAGES"
- run:
name: Launch emulator
command: |
emulator -avd test -delay-adb -verbose -no-window -gpu swiftshader_indirect -no-snapshot -noaudio -no-boot-anim
background: true
- run:
name: Generate cache key
command: |
find . -name 'build.gradle' | sort | xargs cat |
shasum | awk '{print $1}' > /tmp/gradle_cache_seed
- restore_cache:
key: gradle-v1-{{ arch }}-{{ checksum "/tmp/gradle_cache_seed" }}
- run:
# run in parallel with the emulator starting up, to optimize build time
name: Run assembleDebugAndroidTest task
command: |
./gradlew assembleDebugAndroidTest
- run:
name: Wait for emulator to start
command: |
circle-android wait-for-boot
- run:
name: Disable emulator animations
command: |
adb shell settings put global window_animation_scale 0.0
adb shell settings put global transition_animation_scale 0.0
adb shell settings put global animator_duration_scale 0.0
- run:
name: Run UI tests (with retry)
command: |
MAX_TRIES=2
run_with_retry() {
n=1
until [ $n -gt $MAX_TRIES ]
do
echo "Starting test attempt $n"
./gradlew connectedAndroidTest && break
n=$[$n+1]
sleep 5
done
if [ $n -gt $MAX_TRIES ]; then
echo "Max tries reached ($MAX_TRIES)"
exit 1
fi
}
run_with_retry
- save_cache:
key: gradle-v1-{{ arch }}-{{ checksum "/tmp/gradle_cache_seed" }}
paths:
- ~/.gradle/caches
- ~/.gradle/wrapper
- store_test_results:
path: app\build\outputs\androidTest-results
Using Android Orb Example
Since CircleCi has already done some of the work, this step is a bit shorter than the first example. So, you can use this instead of the No-orb example and it'll still function the same.
version: 2.1
orbs:
android: circleci/android@2.1.2
jobs:
android-test:
executor:
name: android/android-machine
tag: "202102-01"
resource-class: large
#run instrumentation tests
steps:
- checkout
- run:
name: installing emulator and Running Instrumentation tests
command: |
sdkmanager "platform-tools" "platforms;android-29" "build-tools;30.0.0" "emulator"
sdkmanager "system-images;android-29;google_apis;x86"
echo no | avdmanager create avd -n test-emulator -k "system-images;android-29;google_apis;x86"
emulator -avd test-emulator -noaudio -no-boot-anim -gpu off -no-window &
adb wait-for-device shell 'while [[ -z $(getprop sys.boot_completed) ]]; do sleep 1; done;'
adb shell wm dismiss-keyguard
sleep 1
adb shell settings put global window_animation_scale 0
adb shell settings put global transition_animation_scale 0
adb shell settings put global animator_duration_scale 0
./gradlew connectedAndroidTest
It's time to see our CI test in action on CircleCI.
How to Setup CircleCI
Make sure you have created at the root of your project’s folder, a .circleci folder, and inside of the folder you have a config.yml file. Before moving on to CircleCI, let's add a build job at the beginning of the config.yml file and define the workflow of our build pipeline at the end of the file. Using the no-orb example, the final config.yml file will look like this:
version: 2.1
jobs:
build:
working_directory: ~/project
docker:
- image: cimg/android:2023.02
steps:
- checkout
- restore_cache:
keys:
- v1-dependencies-{{ checksum "build.gradle" }}
- v1-dependencies-
- run:
name: Install dependencies
command: ./gradlew androidDependencies
- save_cache:
paths:
- ~/.gradle
- ~/.android
key: v1-dependencies-{{ checksum "build.gradle" }}
- run:
name: Build project
command: ./gradlew clean assemble
unit tests:
working_directory: ~/project
docker:
- image: cimg/android:2023.02
steps:
- checkout
- run:
name: Run Local UnitTests
command: |
./gradlew clean test
- store_test_results:
path: app\build\test-results
instrumented_test:
machine:
image: android:202102-01
resource_class: large
steps:
- checkout
- run:
name: Create avd
command: |
SYSTEM_IMAGES="system-images;android-29;default;x86"
sdkmanager "$SYSTEM_IMAGES"
echo "no" | avdmanager --verbose create avd -n test -k "$SYSTEM_IMAGES"
- run:
name: Launch emulator
command: |
emulator -avd test -delay-adb -verbose -no-window -gpu swiftshader_indirect -no-snapshot -noaudio -no-boot-anim
background: true
- run:
name: Generate cache key
command: |
find . -name 'build.gradle' | sort | xargs cat |
shasum | awk '{print $1}' > /tmp/gradle_cache_seed
- restore_cache:
key: gradle-v1-{{ arch }}-{{ checksum "/tmp/gradle_cache_seed" }}
- run:
# run in parallel with the emulator starting up, to optimize build time
name: Run compileDemoBasicDebugJavaWithJavac task
command: |
./gradlew compileDemoBasicDebugJavaWithJavac
- run:
name: Wait for emulator to start
command: |
circle-android wait-for-boot
- run:
name: Disable emulator animations
command: |
adb shell settings put global window_animation_scale 0.0
adb shell settings put global transition_animation_scale 0.0
adb shell settings put global animator_duration_scale 0.0
- run:
name: Run UI tests (with retry)
command: |
MAX_TRIES=2
run_with_retry() {
n=1
until [ $n -gt $MAX_TRIES ]
do
echo "Starting test attempt $n"
./gradlew connectedAndroidTest && break
n=$[$n+1]
sleep 5
done
if [ $n -gt $MAX_TRIES ]; then
echo "Max tries reached ($MAX_TRIES)"
exit 1
fi
}
run_with_retry
- save_cache:
key: gradle-v1-{{ arch }}-{{ checksum "/tmp/gradle_cache_seed" }}
paths:
- ~/.gradle/caches
- ~/.gradle/wrapper
- store_test_results:
path: app\build\outputs\androidTest-results
workflows:
build_and_deploy:
jobs:
- build
- instrumented_test:
requires:
- build
- unit tests:
requires:
- build
Next, login to CircleCI using your GitHub, BitBucket, or GitLab account. In the CircleCI web, click on projects at the sidebar and then select which project on your version control you want CircleCI to build on. In my case, I selected the androidlibraryproject.
In the “Select your config.yml file” modal, select Fastest, and a pop-up will appear like so:
Under fastest select to choose your preferred project and the branch where the project is located, then click Set Up Project to start building your project on CircleCI. You can monitor the build process on your dashboard. On a successful build, you'll have a green build like below:
Note that the instrumentation tests job may take a few more minutes than other jobs to build. The time taken is solely dependent on your project. It took almost five minutes for the instrumentation test to finish building in my case.
Recap
In this article, you learned how to automate testing for your Android project using CircleCI. How to set up unit tests and also learned two approaches to setting up instrumentation tests on CirCleCI. You've learned the benefits of automating testing in CI and you've seen the CI test in action on CircleCI.
Top comments (0)