All you need to know about CircleCI 2.0 with Firebase Test Lab

Problem with CircleCI 1.0

Solution

What about instrumented testing?

Getting started

version: 2references:
# We will define reusable references here
jobs: # Build debug APK for unit tests and an instrumented test APK
build_debug:
# ...
# Build release APK
build_release:
# ...
# Run unit tests
test_unit:
# ...
# Run instrumented tests
test_instrumented:
# ...
# Submit JaCoCo coverage report
report_coverage:
# ...
# Deploy release APK
deploy:
# ...
workflows:
version: 2
workflow:
jobs:
- build_debug
- build_release
- test_unit
- test_instrumented:
requires:
- build_debug
- report_coverage:
requires:
- test_unit
- test_instrumented
- deploy:
filters:
branches:
only:
- master
requires:
- build_release
- test_unit
- test_instrumented
  • references: In this section, we will define some useful constants that will be referenced repeatedly, such as the cache key and the workspace paths.
  • build_debug: This is to run the Gradle tasks, assembleDebug and assembleAndroidTest. Since assembleAndroidTest depends on assembleDebug, it is generally a good idea to run them together so that we can save the time spent on preparing the task dependencies. We will store the task dependencies to cache for later use by dependent tasks. The output APKs will be saved to the workspace shared by all tasks.
  • build_release: This is for running assembleRelease, which usually takes a lot more time than assembleDebug. I suggest to run it as a separate task because your release build may fail due to ProGuard mis-configuration while the debug build succeeds.
  • test_unit: Run unit tests after we have our debug APK built.
  • test_instrumented: Run instrumented test on Firebase Test Lab.
  • report_coverage: This is an optional but recommended task to generate a test coverage report.
  • deploy: Finally, when all tasks run successfully, we may want to deploy our release build.
The suggested Android build workflow

Gradle build tasks

build_debug:
<<:
*android_config
steps:
- checkout
- *restore_cache
- run:
name:
Download dependencies
command: ./gradlew androidDependencies
- *save_cache
- *export_gservices_key
- *decode_gservices_key
- run:
name:
Gradle build (debug)
command: ./gradlew -PciBuild=true :app:assembleDebug :app:assembleAndroidTest
- *persist_debug_workspace
- store_artifacts:
path:
app/build/outputs/apk/
destination: /apk/

Container environment setup

<<: *android_config
android_config: &android_config
working_directory: *workspace
docker:
- image: circleci/android:api-27-alpha
environment:
TERM:
dumb
_JAVA_OPTIONS: "-Xmx2048m -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap"
GRADLE_OPTS: '-Dorg.gradle.jvmargs="-Xmx2048m"'

Cache project dependencies

- checkout
- *restore_cache
- run:
name:
Download dependencies
command: ./gradlew androidDependencies
- *save_cache
cache-{{ checksum "gradle/wrapper/gradle-wrapper.properties" }}-{{ checksum "build.gradle" }}-{{ checksum "app/build.gradle" }}

Load secret keys dynamically

- *export_gservices_key
- *decode_gservices_key
echo 'export GOOGLE_SERVICES_KEY="$GOOGLE_SERVICES_KEY"' >> $BASH_ENV
echo $GOOGLE_SERVICES_KEY | base64 -di > app/google-services.json

Build the APKs

./gradlew -PciBuild=true assembleDebug assembleAndroidTest

Persist workspace

- *persist_debug_workspace

Store artifacts

- store_artifacts:
path:
app/build/outputs/apk/
destination: /apk/

Running tests

test_instrumented:
<<:
*gcloud_config
steps:
- *attach_debug_workspace
- *export_gcloud_key
- *decode_gcloud_key
- run:
name:
Set Google Cloud target project
command: gcloud config set project newspaper-84169
- run:
name:
Authenticate with Google Cloud
command: gcloud auth activate-service-account firebase-adminsdk-p9qvk@newspaper-84169.iam.gserviceaccount.com --key-file ${HOME}/client-secret.json
- run:
name:
Run instrumented test on Firebase Test Lab
command: gcloud firebase test android run --type instrumentation --app app/build/outputs/apk/debug/app-debug.apk --test app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk --device model=Nexus5X,version=26,locale=en_US,orientation=portrait --environment-variables coverage=true,coverageFile=/sdcard/tmp/code-coverage/connected/coverage.ec --directories-to-pull=/sdcard/tmp --timeout 20m
- run:
name:
Create directory to store test results
command: mkdir firebase
- run:
name:
Download instrumented test results from Firebase Test Lab
command: gsutil -m cp -r -U "`gsutil ls gs://test-lab-3udbiqpdyp0d0-miwcp7d69v80m | tail -1`*" /root/workspace/firebase/
- *persist_firebase_workspace
- store_artifacts:
path:
firebase/
destination: /firebase/

Build environment for Firebase Test Lab

gcloud_config: &gcloud_config
working_directory: *workspace
docker:
- image: google/cloud-sdk:latest
environment:
TERM:
dumb

Load Google Cloud Service Account key dynamically

echo $GCLOUD_SERVICE_KEY | base64 -di > ${HOME}/client-secret.json

Configure your Firebase Test Lab project

gcloud config set project <your Firebase project ID, e.g. newspaper-84169>
gcloud auth activate-service-account <your Firebase service account ID> --key-file ${HOME}/client-secret.json
gcloud firebase test android run \
--type instrumentation \
--app app/build/outputs/apk/debug/app-debug.apk \
--test app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk \
--device model=Nexus5X,\
version=26,\
locale=en_US,\
orientation=portrait \
--environment-variables coverage=true,\
coverageFile=/sdcard/tmp/code-coverage/connected/coverage.ec \
--directories-to-pull=/sdcard/tmp \
--timeout 20m
  • type instrumentation: We want to run Espresso instrumented tests.
  • app: This is our debug build APK generated by build_debug task.
  • test: This is our instrumented test APK, also generated by build_debug task.
  • device: I use a real device, Nexus 5X, in this example. You may use an emulator if you want.
  • environment-variables: This is necessary if we want to generate a test coverage report from a device on Firebase Test Lab. This will set the environment variables for our Android device. Setting coverage=true will enable test coverage report generation when the tests complete. The test coverage report will be written to the path specified by coverageFile.
  • directories-to-pull: The test coverage report can only be written to a path of the device, so we have to copy the report file to the artifacts directory of Firebase Test Lab.
  • timeout: It is generally a good idea to limit the test execution time. If you are on a paid plan, it is unwise to not setting a timeout.

Copy test results from Firebase to CircleCI

gsutil -m cp -r -U "`gsutil ls gs://test-lab-<some random ID>-<some other random ID> | tail -1`*" /root/workspace/firebase/
https://console.cloud.google.com/storage/browser/test-lab-<some random ID>-<some other random ID>/2017-12-04_04:16:23.701595_DgjM/Nexus5X-26-en_US-portrait?authuser=1

Report test coverage

mkdir -p app/build/outputs/code-coverage/connected && cp firebase/Nexus5X-26-en_US-portrait/artifacts/coverage.ec app/build/outputs/code-coverage/connected/coverage.ec

Deploy

Conclusion

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store