At Rally, we focus a lot on automated testing. We spend a significant amount of time writing tests and thus need to make important decisions around testing strategies. As a good practice, we try to measure and maintain a high code coverage for our projects. With a variety of testing tools available, this post is about how we measure code coverage and the common gotchas encountered in Android projects. Most of this knowledge is already out there in some form of Stack Overflow question or forum question, but we have made an attempt here to consolidate all the information a developer needs to set up unified code coverage for an Android project.

For Android, any test that requires a hardware or an emulator is an instrumented test. Such tests are placed in the androidTest folder. Tests that do not require an emulator/hardware are placed in the tests folder. This blog post covers extracting code coverage for unit tests, instrumentation tests, Robolectric tests and tests written in Kotlin.

If you are writing tests in Kotlin you may encounter an issue mocking final classes. Since Kotlin classes are final by default, one may have to open the classes or functions, or create interfaces. Both these options are not appealing. Mockito 2 allows mocking final classes. In order to do that, one needs to create a file named org.mockito.plugins.MockMaker and place this under the resources/mockito-extensions directory in the test folder. This file contains a single line:

mock-maker-inline

That’s it! Now you can mock final classes in Kotlin using Mockito 2.

Code coverage setup

Android Studio generates HTML coverage reports only for instrumentation test. To view HTML coverage reports for unit testing, one will need Jacoco (Java Code Coverage). With Android plugin 3.0.0 onwards, one cannot configure Jacoco using the android DSL block. Thus, in the project level gradle file, one needs to add the Jacoco dependency with the gradle dependency, forcing the instrumented tests to use Jacoco.

buildscript {
  repositories {
    google()
    jcenter()
  }
  dependencies {
    classpath 'com.android.tools.build:gradle:3.0.1'
    classpath 'org.jacoco:org.jacoco.core:0.8.1' //Use latest version
  }
}

Now, for the module containing the tests, update the respective gradle file to apply the Jacoco plugin and specify the same toolVersion as used in the project gradle file.

You may encounter an issue while getting test coverage for Robolectric tests. To include Robolectric tests in the Jacoco report, one will need to set the includeNolocationClasses flag to true. This can no longer be configured using the android DSL block, thus we search all tasks of Test type and enable it.

To get coverage reports for instrumentation tests, in the android DSL block, set the testCoverageEnabled flag to true.

AndroidJUnitRunner runs on the same instrumentation process so basically your tests are running stateful, this might wreak havoc if your tests have some sort of dependency on the process state. To avoid this, Android recommends using ORCHESTRATOR. In Android test orchestrator each test is run in its own process so dependencies aren’t an issue.

You may also disable the animations during the test execution, and enable the Android resources for the unit tests so that Robolectric tests won’t fail.

For the purpose of this post, we will be running the tests for the debug build variant.

The following is what your gradle file might look like after incorporating the above changes.

apply plugin: 'jacoco'

jacoco {
    toolVersion = '0.8.1' //Use latest version
}

tasks.withType(Test) {
    jacoco.includeNoLocationClasses = true
}

android {
    buildTypes {
        debug {
            testCoverageEnabled true
        }
    }
    testOptions {
        execution 'ANDROID_TEST_ORCHESTRATOR'
        animationsDisabled true

        unitTests {
            includeAndroidResources = true
        }
    }
}

Instrumentation tests

To get coverage reports for the intrumentation tests, one can run the gradle task createDebugCoverageReport. This will generate the coverage report for the instrumentation tests only. This report is generated under build/reports/coverage/debug. If you don’t see createDebugCoverageReport, check the testCoverageEnabled flag, that should be true for the build variant, you are interested in.

Local unit tests

To get a quick peek at the code coverage for unit tests, one can click the “Run test with coverage”, to the next of the debug button(highlighted with a yellow circle). This will show the code coverage within Android Studio:

Run with coverage

However, if you are interested in getting the raw coverage files, you can run the gradle task testDebugUnitTest. This generates the raw coverage file in the jacoco folder within build/outputs. The raw coverage file will be called testDebugUnitTest.exec.

As mentioned before, Android Studio does not generate Jacoco reports for the unit tests. However, you do have the raw coverage file (testDebugUnitTest.exec). This can be used to generate the Jacoco reports by writing a gradle task:

task jacocoUnitTestReport(type: JacocoReport, dependsOn: ['testDebugUnitTest']) {

    $buildDir = // Location of the build directory for the build Variant

    def coverageSourceDirs = [
        "src/main/java"
    ]

    def fileFilter = [
        '**/R.class',
        '**/R$*.class',
        '**/*$ViewInjector*.*',
        '**/*$ViewBinder*.*',
        '**/BuildConfig.*',
        '**/Manifest*.*'
    ]

    def javaClasses = fileTree(
        dir: "$buildDir/intermediates/classes/debug",
        excludes: fileFilter
    )

    classDirectories = files([ javaClasses ])
    additionalSourceDirs = files(coverageSourceDirs)
    sourceDirectories = files(coverageSourceDirs)
    executionData = fileTree(dir: "$buildDir", includes: [
            "jacoco/testDebugUnitTest.exec"
    ])

    reports {
        xml.enabled = true
        html.enabled = true
    }
}

This task first runs its dependency, to make sure the raw unit test coverage files are available. The next set of commands specify the source code to evaluate coverage against. We then filter to remove some of the unwanted files (e.g. R files, manifest, configs, etc.) from the code coverage calculations. javaClasses gives the location of the files in the build directory. All this data is then fed into the relevant parameters for the Jacoco report (e.g. classDirectories, sourceDirectories). The executionData is then made to point to the raw coverage data file. Finally, one can specify the types of reports to be generated (HTML, XML). After running this task, the coverage report will be available under build/jacoco/jacocoUnitTestReport. You should be able to see something along the following lines:

Coverage report

Kotlin

To include Kotlin files, add "src/main/kotlin" to the coverageSourceDirs first. For Kotlin, the location of the debugTree files are different than that for Java. The Kotlin files are located at tmp/kotlin-classes/debug instead of /intermediates/classes/debug. Thus, you will need to add the following after the javaClasses variable:

def kotlinClasses = fileTree(
            dir: "$buildDir/tmp/kotlin-classes/debug",
            excludes: fileFilter
    )

And then include that in the classDirectories: classDirectories = files([ javaClasses ], [ kotlinClasses ]). This should be sufficient to generate the coverage reports for Kotlin tests.

Aggregate coverage report (Instrumentation + Unit)

Unit test raw files are of the .exec format, while the instrumentation tests are of the .ec format. That is not an issue while creating the aggregate coverage report. We can use both these raw files while setting up the execution data for the Jacoco report tasks. Thus to get an aggregate coverage report, update the executionData to include the instrumentation test coverage reports:

executionData = fileTree(dir: "$buildDir", includes: [
        "jacoco/testDebugUnitTest.exec",
        "output/code-coverage/connected/*.ec" //Check this path or update to relevant path
])

Multi module app

If you have multiple modules in your project and tests are spread across these modules, you could create a gradle task in every module to get the unit test coverage reports, and then create a root level gradle task to generate an aggregate coverage report by specifying the classDirectories, sourceDirectories and executionData from the different modules.

Since, we were going to use a similar task in every module to generate the coverage reports for unit tests, we decided to abstract this out into a gradle method, which accepted the module and build variant as the parameters and generated the coverage reports for that module:

ext.enableJacoco = { Project project, String variant ->
    project.plugins.apply('jacoco')

    final capVariant = variant.capitalize()

    StringBuilder folderSb = new StringBuilder(variant.length() + 1)
    for (int i = 0; i < variant.length(); i++) {
        char c = variant.charAt(i)
        if (Character.isUpperCase(c)) {
            folderSb.append('/')
            folderSb.append(Character.toLowerCase(c))
        } else {
            folderSb.append(c)
        }
    }
    final folder = folderSb.toString()

    project.android.testOptions {
        unitTests.all {
            jacoco {
                includeNoLocationClasses = true
            }
        }
    }

    project.jacoco {
        toolVersion = '0.8.1'
    }

    project.tasks.create(
            name: 'jacocoTestReport',
            type: JacocoReport,
            dependsOn: "test${capVariant}UnitTest"
    ) {
        def buildDir = project.buildDir

        def coverageSourceDirs = [
                "src/main/java",
                "src/main/kotlin"
        ]

        def fileFilter = [
                '**/R.class',
                '**/R$*.class',
                '**/*$ViewInjector*.*',
                '**/*$ViewBinder*.*',
                '**/BuildConfig.*',
                '**/Manifest*.*'
        ]

        def javaClasses = fileTree(
                dir: "$buildDir/intermediates/classes/$folder",
                excludes: fileFilter
        )
        def kotlinClasses = fileTree(
                dir: "$buildDir/tmp/kotlin-classes/$variant",
                excludes: fileFilter
        )

        group = "Reporting"
        description = "Generate Jacoco coverage reports for the ${project.name} with the " +
                "$variant variant."
        classDirectories = files([ javaClasses ], [ kotlinClasses ])
        additionalSourceDirs = files(coverageSourceDirs)
        sourceDirectories = files(coverageSourceDirs)
        executionData = files("${project.buildDir}/jacoco/test${capVariant}UnitTest.exec")
        reports {
            xml.enabled = true
            html.enabled = true
        }
    }
}

We then call this from the gradle file of all the submodules and pass the project and the build variant as follows:

enableJacoco(project, 'Debug')

And finally, we apply the plugin and this function in the root gradle file:

apply from: rootProject.file('gradle/enable-jacoco.gradle')

Now you have the unit test coverage reports for all modules with the instrumentation test reports using createDebugCoverageReport. To generate the aggregate coverage report across all these modules and test types (instrumentation, unit etc.), there is a gradle plugin created by the folks at Palantir - com.palantir.jacoco-full-report. This needs to be applied in the root level gradle file:

plugins {
    id "com.palantir.jacoco-full-report" version "0.4.0"
}

Now, to create an aggregate coverage report across all modules (basically for the Android project), run the following gradle command: ./gradlew jacocoFullReport. The aggregate report will be available in the root level build directory under reports/jacoco/jacocoFullReport.

Firebase TestLab

You probably want to generate the coverage reports in your continuous integration (CI) environments. In order for instrumentation tests, the CI will need to run an emulator or hardware device. Jenkins has an Android emulator plugin, which can be used for this purpose. However, setting up an emulator in an emulated environment comes with its own set of challenges. But wouldn’t you love to run your tests on an actual device? Or a range of devices and OS versions? That is possible using Firebase TestLab.

Firebase Test Lab is a service provided by Firebase to run instrumentation tests on actual devices or emulators with specific configurations. This obviates the need to run tests on an emulator on Jenkins, when you can run your tests on a variety of devices in the cloud. The test lab reports include many things such as videos of the test runs, error reports and coverage reports too. These coverage reports are stored on the gcloud bucket associated with the Firebase account of the Android project. In order to get a simple run of firebase test lab, you can execute the following command in your continuous integration environment:

gcloud firebase test android run \
--type instrumentation \
--app app/build/outputs/apk/*-debug.apk \
--test app/build/outputs/apk/*-androidTest.apk \
--directories-to-pull /sdcard \
--device model=NexusLowRes,version=25 \
--environment-variables coverage=true,coverageFile=“/sdcard/coverage.ec”

A few points to consider here:

  1. This command requires that the apks and test apk be available. That is possible using the gradle assemble command.

  2. The Android manifest for the test apk needs to enable external storage, else the coverage reports will not be generated as the coverage files are stored on the device and downloaded from there.

You can then download the coverage reports from the firebase gcloud bucket for the project by setting up gcloud and using the gsutil cp command to specify the coverage source location and the destination to copy the coverage files to.

Flank

A common problem with mobile ui tests it that it could take a significant amount of time to run all the ui tests for a mature product (nearly an hour or more). Android actually added annotations to signify small, medium, and large tests, so that you could exclude certain tests when you run so they don’t take forever. Enter Flank - an open source project built by Walmart labs with support from the Google Firebase team. Flank effectively helps scale the number of tests through test sharding to keep the total test execution time low so that all the tests can be run as part of the CI deployment process. This could also help one reduce their costs on Firebase. In a nutshell, while Firebase will run every test on a separate device and charge rounding up to the minute, Flank, stores the test run times and then aggregates to optimize the number of devices used. E.g. If you have around 300 tests and each test takes 10 seconds, the first execution will run on 300 devices and will charge for one minute for each device. The test execution times are saved by flank. Consecutive runs will take into account the test times and thus will use 50 devices (6 tests on each device) for the test runs. This helps save some setup/tear down time. Flank is well explained on this blog post and the project resides here. After the tests are run, all the details regarding the tests such as videos, coverage reports are downloaded in the root folder of the Android project in a folder called results.