Test execution

• Improved smart sharding by casting a wider net when there was no strict historical runtime available (e.g. due to changing --environment-variables or such)

emulator.wtf web app

• Added a workaround for an issue where double videos would be captured when running tests with orchestrator

  NOTE: upgrade to test-runtime-android 0.2.1 to fix the root cause

gradle-plugin 0.18.0

• Much improved error messages, including the name and stacktrace of the first failing test
• BETA: added an optional summary task that collects and prints out test results from all modules
• BREAKING!
  • Minimum supported Android Gradle Plugin is now 7.0.0
  • Minimum supported Gradle is now 7.0.2
  • Changed default results output folder to build/test-results/${taskName}, aligning with jvm tests
• Fixed issues when using the Gradle plugin together with Gradle isolated projects
• Fixed a bug when configuration cache was broken when using certain providers in the extension dsl
• Bumped default ew-cli to 0.11.1:
  • Fixed an issue where downloading individual test videos would bork out with a Faulty ew-cli, no QOI decoder configured? error

ew-cli 0.11.1

• Fixed a Faulty ew-cli, no QOI decoder configured? error when downloading individual test videos with --outputs individual_test_videos

test-runtime-android 0.2.1

• Fixed double videos (one short, one real) being picked up when running tests with orchestrator

Emulators

• Fixed a GPU driver issue affecting API 28-30 emulators on a subset of our worker machines. On the faulty machines we experienced some faulty GL calls resulting in quite slow tests (adding around 1 minute to total time in our internal benchmarks). If you are using one of the affected API versions with GPU acceleration you should see good improvements in p90 test times.

Test execution

• Rolled back the preinstalled orchestrator and test-services APKs to 1.4.2 due to a whitespace in test names bug in 1.5.0.

emulator.wtf web app

• Fixed emulator.wtf always rendering a 404 page when viewing results for a test run where one or more tests had blank names

Test execution

• Fixed some shards running too many tests when using --test-targets together with sharding

emulator.wtf web app

• The test videos now scale with the browser viewport size making them much more useable on laptop screens.

Test execution

• Fixed a case where tests would hang immediately after Instrumentation init on API 34 when using the wtf.emulator:test-runtime-android companion library

Test execution

• Fixed a case where tests would hang indefinitely if a single shard’s test targets (class + method names) were larger than 256 KiB.

Emulators

• (BETA!) Added API 35 emulators for all available non-ATD models

  Complete list of new combinations added:

  • model=Pixel2,version=35
  • model=Pixel7,version=35
  • model=Tablet10,version=35
  • model=NexusLowRes,version=35
  • model=Monitor,version=35

emulator.wtf web app

• When navigating to an https://emulator.wtf/o/... link we now default the user to the SSO sign-in method (GitHub, SAML) when configured. This guides your organization members to log in with the SSO regardless of how they land on the link.

Test execution

• Added a workaround for a case where tests running on API 34 would get stuck or not start at all

gradle-plugin 0.17.0

• bumped default ew-cli version to 0.11.0

GitHub action emulator-wtf/run-tests@v0.9.10

• Bumped default ew-cli to 0.11.0
• Updated subaction setup-ew-cli to v0.9.10

ew-cli 0.11.0

• made --help more helpful by splitting options into groups
• added new individual_test_videos output to download each testcase video separately
• fixed a case where test would get stuck in websocket-hostile network environments
• improved resilience against bad network conditions with a few retries
• fixed a case where some flaky test results were erroneously marked as failed

Emulators

• Added API 21, 26 and 34 (ATD) emulator images. We now have full coverage of versions between API versions 23 and 34! List of device combinations added:
  • device=Pixel2,version=21
  • device=Pixel2,version=26
  • device=Pixel7,version=21
  • device=Pixel7,version=26
  • device=Tablet10,version=21
  • device=Tablet10,version=26
  • device=NexusLowRes,version=21
  • device=NexusLowRes,version=26
  • device=Monitor,version=21
  • device=Monitor,version=26
  • device=Pixel2Atd,version=34
  • device=Pixel7Atd,version=34
  • device=Tablet10Atd,version=34
  • device=NexusLowResAtd,version=34
  • See all available emulator images here.

emulator.wtf web app

• Fixed scrubbing of per-test videos on the chrome family of browsers.

emulator.wtf web app

• The emulator.wtf web app now requires solving a captcha for certain user actions, like signing up.

Test execution

• Test runs running with balanced sharding no longer error out if there are less tests than targeted shards.

emulator.wtf web app

• We have slightly tweaked the way SAML configuration works for organizations on the Enterprise plan.

Emulators

• Two new outgoing IP addresses have been added: 65.109.54.236 and 135.181.209.62. If you rely on outgoing IP addresses we recommend automating them - we publish all the outgoing IP addresses both as JSON and as DNS records. Read more here.

emulator.wtf web app

• The results web app now has the git commit hash and link to GitHub if these details are available

Test execution

emulator.wtf now supports capturing per-test videos!

This has been the most requested feature by far. We now support capturing frame-precise test recordings of individual tests. To enable this we need some help from the instrumentation test runner to capture exact timestamps of the tests.

To enable this feature, make sure you have the record video flag set and add the following library to your Gradle build:

androidTestImplementation("wtf.emulator:test-runtime-android:0.2.0")

test-runtime-android 0.2.0

Initial public release of wtf.emulator:test-runtime-android:0.2.0.

• include this library in the test apk via androidTestImplementation to enable per-test video recordings (see above)

emulator.wtf web app

• Improved internal handling of refresh-access token pairs to improve web app performance

emulator.wtf web app

• Fixed developer role missing permissions to delete own tokens

emulator.wtf API

• Improved backend performance when spawning hundreds of test runs in quick succession

emulator.wtf web app

• Fixed an issue with GitHub SSO login

Emulators

• Disabled Google Chrome’s welcome screen, making it easier to run tests that use the browser.
• Added API 25, 28 and 32 emulators. List of device combinations added:
  • device=Pixel2,version=25
  • device=Pixel2,version=28
  • device=Pixel2,version=32
  • device=Pixel7,version=25
  • device=Pixel7,version=28
  • device=Pixel7,version=32
  • device=Tablet10,version=25
  • device=Tablet10,version=28
  • device=Tablet10,version=32
  • device=NexusLowRes,version=25
  • device=NexusLowRes,version=28
  • device=NexusLowRes,version=32
  • device=Monitor,version=25
  • device=Monitor,version=28
  • device=Monitor,version=32
  • See all available emulator images here.