Test execution

• Fixed a case where tests would hang immediately after Instrumentation init on API 34 when using the wtf.emulator:test-runtime-android companion library

Test execution

• Fixed a case where tests would hang indefinitely if a single shard’s test targets (class + method names) were larger than 256 KiB.

Emulators

• (BETA!) Added API 35 emulators for all available non-ATD models

  Complete list of new combinations added:

  • model=Pixel2,version=35
  • model=Pixel7,version=35
  • model=Tablet10,version=35
  • model=NexusLowRes,version=35
  • model=Monitor,version=35

emulator.wtf web app

• When navigating to an https://emulator.wtf/o/... link we now default the user to the SSO sign-in method (GitHub, SAML) when configured. This guides your organization members to log in with the SSO regardless of how they land on the link.

Test execution

• Added a workaround for a case where tests running on API 34 would get stuck or not start at all

gradle-plugin 0.17.0

• bumped default ew-cli version to 0.11.0

GitHub action emulator-wtf/run-tests@v0.9.10

• Bumped default ew-cli to 0.11.0
• Updated subaction setup-ew-cli to v0.9.10

ew-cli 0.11.0

• made --help more helpful by splitting options into groups
• added new individual_test_videos output to download each testcase video separately
• fixed a case where test would get stuck in websocket-hostile network environments
• improved resilience against bad network conditions with a few retries
• fixed a case where some flaky test results were erroneously marked as failed

Emulators

• Added API 21, 26 and 34 (ATD) emulator images. We now have full coverage of versions between API versions 23 and 34! List of device combinations added:
  • device=Pixel2,version=21
  • device=Pixel2,version=26
  • device=Pixel7,version=21
  • device=Pixel7,version=26
  • device=Tablet10,version=21
  • device=Tablet10,version=26
  • device=NexusLowRes,version=21
  • device=NexusLowRes,version=26
  • device=Monitor,version=21
  • device=Monitor,version=26
  • device=Pixel2Atd,version=34
  • device=Pixel7Atd,version=34
  • device=Tablet10Atd,version=34
  • device=NexusLowResAtd,version=34
  • See all available emulator images here.

emulator.wtf web app

• Fixed scrubbing of per-test videos on the chrome family of browsers.

emulator.wtf web app

• The emulator.wtf web app now requires solving a captcha for certain user actions, like signing up.

Test execution

• Test runs running with balanced sharding no longer error out if there are less tests than targeted shards.

emulator.wtf web app

• We have slightly tweaked the way SAML configuration works for organizations on the Enterprise plan.

Emulators

• Two new outgoing IP addresses have been added: 65.109.54.236 and 135.181.209.62. If you rely on outgoing IP addresses we recommend automating them - we publish all the outgoing IP addresses both as JSON and as DNS records. Read more here.

emulator.wtf web app

• The results web app now has the git commit hash and link to GitHub if these details are available

Test execution

emulator.wtf now supports capturing per-test videos!

This has been the most requested feature by far. We now support capturing frame-precise test recordings of individual tests. To enable this we need some help from the instrumentation test runner to capture exact timestamps of the tests.

To enable this feature, make sure you have the record video flag set and add the following library to your Gradle build:

androidTestImplementation("wtf.emulator:test-runtime-android:0.2.0")

test-runtime-android 0.2.0

Initial public release of wtf.emulator:test-runtime-android:0.2.0.

• include this library in the test apk via androidTestImplementation to enable per-test video recordings (see above)

emulator.wtf web app

• Improved internal handling of refresh-access token pairs to improve web app performance

emulator.wtf web app

• Fixed developer role missing permissions to delete own tokens

emulator.wtf API

• Improved backend performance when spawning hundreds of test runs in quick succession

emulator.wtf web app

• Fixed an issue with GitHub SSO login

Emulators

• Disabled Google Chrome’s welcome screen, making it easier to run tests that use the browser.
• Added API 25, 28 and 32 emulators. List of device combinations added:
  • device=Pixel2,version=25
  • device=Pixel2,version=28
  • device=Pixel2,version=32
  • device=Pixel7,version=25
  • device=Pixel7,version=28
  • device=Pixel7,version=32
  • device=Tablet10,version=25
  • device=Tablet10,version=28
  • device=Tablet10,version=32
  • device=NexusLowRes,version=25
  • device=NexusLowRes,version=28
  • device=NexusLowRes,version=32
  • device=Monitor,version=25
  • device=Monitor,version=28
  • device=Monitor,version=32
  • See all available emulator images here.

emulator.wtf web app

• Improved the performance of displaying and paging test runs in the web app

emulator.wtf web app

The test run result links now contain a few details about the run:

• display name of the test
• timestamp and total execution time
• number of devices and shards used
• total billable minutes consumed by the run
• input test run arguments in a JSON format

ew-cli 0.10.9

• Improved network failure handling in ew-cli. We now retry failed requests in a more robust way, and we also increased some of the longer network call timeouts (like file uploads). This should help in cases where you are running many tests in parallel from the same CI job.

gradle-plugin 0.16.2

• Bumped default ew-cli to 0.10.9

GitHub action emulator-wtf/run-tests@v0.9.9

• Bumped default ew-cli to 0.10.9
• Updated subaction setup-ew-cli to v0.9.9

Test execution

• Fixed a case where test runs with model=Tablet10 would capture corrupted video due to mismatched encoder parameters.

Test execution

• Fixed an escaping issue where environment variables containing the backslash \ character were not passed correctly to the test runner. You were most likely to encounter this when using regexes in your test runner arguments.
• Added a workaround for a case where a single test method was passed as a --test-target and it expanded to multiple parametric test cases. This was previously causing the test run to fail and works now without issue.

ew-cli 0.10.8

• The CLI is now better at printing some unexpected errors, for instance when misconfiguring the API token.

Emulators

• All of our emulators are now in the Atlantic/Reykjavik timezone (same as UTC). This should improve compatibility with tests that use the system timezone identifier and don’t support the Etc/UTC timezone identifier.

Test execution

• If your test manages to crash the surfaceflinger process and the whole System UI with it, we’ll trigger an automatic retry of the test run. This is a rare event that may happen with tests making heavy use of surfaces like utilizing the camera preview. The retry does not add any billable minutes to your account. NOTE: just like with flaky repeats, we do not attempt any retries when using the side-effects flag.

emulator.wtf web app

• Fixed a bug where flaky test runs were shown as failed on the test runs page in the emulator.wtf web app.