GitHub Action

There’s an official GitHub Action that can be used to run instrumentation tests as part of your GitHub Actions workflow.

Quick example

A really basic example building an app apk, test apk, running tests and then finally collecting the results with the mikepenz/action-junit-report@v2 action:

name: Run tests
on: push
jobs:
  run-tests:
    runs-on: ubuntu-20.04
  steps:
    - uses: actions/checkout@v2
    - name: Build app
      run: ./gradlew assembleDebug assembleAndroidTest
    - name: Run tests
      uses: emulator-wtf/run-tests@v0
      with:
        api-token: ${{ secrets.EW_API_TOKEN }}
        app: app/build/outputs/apk/debug/app-debug.apk
        test: app/build/outputs/apk/androidTest/app-debug-androidTest.apk
        outputs-dir: build/test-results
    - name: Publish test report
      uses: mikepenz/action-junit-report@v2
      if: always() # always run even if the tests fail
      with:
        report_paths: 'build/test-results/**/*.xml'

As a best practice, do not store the emulator.wtf API token directly in your GitHub Action Workflow configuration.
Use Encrypted secrets instead.

Inputs

Variable	Description
`version`	ew-cli version to use
`api-token`	API token for emulator.wtf. We recommend using Encrypted secrets for this.
`app`	Path to application apk file
`test`	Path to test apk file
`library-test`	Path to library test apk file
`outputs-dir`	Location to store test outputs in
`outputs`	Comma-separated list to specify what to download to output-dir. Defaults to `merged_results_xml,coverage,pulled_dirs`.
`record-video`	Set to true to record a video of the test run. Defaults to false.
`devices`	Device configurations to use, in the form of `model=X,version=Y` per line
`timeout`	Timeout for the test run, number with unit (`h`, `m` or `s`). Defaults to 15m.
`use-orchestrator`	Whether to use the Android Test Orchestrator
`clear-package-data`	Whether to clear app data between every test (requires `use-orchestrator`)
`with-coverage`	Set to true to collect coverage files and save them to `outputs-dir`
`additional-apks`	Additional apks to install, one per line
`environment-variables`	Environment variables to pass to AndroidJUnitRunner, one per line in the form of `key=value`
`num-uniform-shards`	Set to a number larger than 1 to randomly split your tests into multiple shards to be executed in parallel
`num-shards`	Set to a number larger than 1 to split your tests into multiple shards based on test counts to be executed in parallel
`num-balance-shards`	Set to a number larger than 1 to split your tests into multiple shards based on test execution time to be executed in parallel
`directories-to-pull`	Directories to pull from device and store in `outputs-dir`, one per line
`side-effects`	Whether the test has any side effects, like hitting external APIs. Prevents any test caching and retries.
`num-flaky-test-attempts`	Number of times to retry flaky tests. Defaults to 0.
`file-cache`	Whether to cache files between test runs. Defaults to true.
`file-cache-ttl`	How long to cache test files for. Defaults to 1h.
`test-cache`	Whether to cache test results between test runs. Defaults to true.

Common examples

Run tests with multiple device profiles

By default emulator.wtf runs tests on a Pixel2-like emulator with API 27 (Android 8.1). If you want to run on a different version or device profile you can use the devices input to do so:

    - name: Run tests
      uses: emulator-wtf/run-tests@v0
      with:
        api-token: ${{ secrets.EW_API_TOKEN }}
        app: app/build/outputs/apk/debug/app-debug.apk
        test: app/build/outputs/apk/androidTest/app-debug-androidTest.apk
        devices: |
          model=NexusLowRes,version=23
          model=Pixel2,version=27

Run tests with orchestrator while clearing package data

You can use Android Test Orchestrator to run the tests - this will create a new app VM from scratch for each test. Slower to run, but will ensure no static state leakage between tests. Add the optional clear-package-data flag to clear app persisted state between each run. Read more about orchestrator here.

    - name: Run tests
      uses: emulator-wtf/run-tests@v0
      with:
        api-token: ${{ secrets.EW_API_TOKEN }}
        app: app/build/outputs/apk/debug/app-debug.apk
        test: app/build/outputs/apk/androidTest/app-debug-androidTest.apk
        use-orchestrator: true
        clear-package-data: true

Grab coverage data

Use the with-coverage flag to capture test run coverage data and store the results (one or more .exec or .ec files) in the path specified by outputs-dir:

    - name: Run tests
      uses: emulator-wtf/run-tests@v0
      with:
        api-token: ${{ secrets.EW_API_TOKEN }}
        app: app/build/outputs/apk/debug/app-debug.apk
        test: app/build/outputs/apk/androidTest/app-debug-androidTest.apk
        with-coverage: true
        outputs-dir: build/test-results

Run tests with shards

The following example runs tests in parallel using 10 separate shards and stores the outputs from each shard in a separate folder under build/test-results:

    - name: Run tests
      uses: emulator-wtf/run-tests@v0
      with:
        api-token: ${{ secrets.EW_API_TOKEN }}
        app: app/build/outputs/apk/debug/app-debug.apk
        test: app/build/outputs/apk/androidTest/app-debug-androidTest.apk
        num-shards: 10
        outputs-dir: build/test-results