Test execution

• Added a workaround for a case where tests running on API 34 would get stuck or not start at all

gradle-plugin 0.17.0

• bumped default ew-cli version to 0.11.0

GitHub action emulator-wtf/run-tests@v0.9.10

• Bumped default ew-cli to 0.11.0
• Updated subaction setup-ew-cli to v0.9.10

ew-cli 0.11.0

• made --help more helpful by splitting options into groups
• added new individual_test_videos output to download each testcase video separately
• fixed a case where test would get stuck in websocket-hostile network environments
• improved resilience against bad network conditions with a few retries
• fixed a case where some flaky test results were erroneously marked as failed

Emulators

• Added API 21, 26 and 34 (ATD) emulator images. We now have full coverage of versions between API versions 23 and 34! List of device combinations added:
  • device=Pixel2,version=21
  • device=Pixel2,version=26
  • device=Pixel7,version=21
  • device=Pixel7,version=26
  • device=Tablet10,version=21
  • device=Tablet10,version=26
  • device=NexusLowRes,version=21
  • device=NexusLowRes,version=26
  • device=Monitor,version=21
  • device=Monitor,version=26
  • device=Pixel2Atd,version=34
  • device=Pixel7Atd,version=34
  • device=Tablet10Atd,version=34
  • device=NexusLowResAtd,version=34
  • See all available emulator images here.

emulator.wtf web app

• Fixed scrubbing of per-test videos on the chrome family of browsers.

emulator.wtf web app

• The emulator.wtf web app now requires solving a captcha for certain user actions, like signing up.

Test execution

• Test runs running with balanced sharding no longer error out if there are less tests than targeted shards.

emulator.wtf web app

• We have slightly tweaked the way SAML configuration works for organizations on the Enterprise plan.

Emulators

• Two new outgoing IP addresses have been added: 65.109.54.236 and 135.181.209.62. If you rely on outgoing IP addresses we recommend automating them - we publish all the outgoing IP addresses both as JSON and as DNS records. Read more here.

emulator.wtf web app

• The results web app now has the git commit hash and link to GitHub if these details are available

Test execution

emulator.wtf now supports capturing per-test videos!

This has been the most requested feature by far. We now support capturing frame-precise test recordings of individual tests. To enable this we need some help from the instrumentation test runner to capture exact timestamps of the tests.

To enable this feature, make sure you have the record video flag set and add the following library to your Gradle build:

androidTestImplementation("wtf.emulator:test-runtime-android:0.2.0")

test-runtime-android 0.2.0

Initial public release of wtf.emulator:test-runtime-android:0.2.0.

• include this library in the test apk via androidTestImplementation to enable per-test video recordings (see above)

emulator.wtf web app

• Improved internal handling of refresh-access token pairs to improve web app performance

emulator.wtf web app

• Fixed developer role missing permissions to delete own tokens

emulator.wtf API

• Improved backend performance when spawning hundreds of test runs in quick succession

emulator.wtf web app

• Fixed an issue with GitHub SSO login

Emulators

• Disabled Google Chrome’s welcome screen, making it easier to run tests that use the browser.
• Added API 25, 28 and 32 emulators. List of device combinations added:
  • device=Pixel2,version=25
  • device=Pixel2,version=28
  • device=Pixel2,version=32
  • device=Pixel7,version=25
  • device=Pixel7,version=28
  • device=Pixel7,version=32
  • device=Tablet10,version=25
  • device=Tablet10,version=28
  • device=Tablet10,version=32
  • device=NexusLowRes,version=25
  • device=NexusLowRes,version=28
  • device=NexusLowRes,version=32
  • device=Monitor,version=25
  • device=Monitor,version=28
  • device=Monitor,version=32
  • See all available emulator images here.

emulator.wtf web app

• Improved the performance of displaying and paging test runs in the web app

emulator.wtf web app

The test run result links now contain a few details about the run:

• display name of the test
• timestamp and total execution time
• number of devices and shards used
• total billable minutes consumed by the run
• input test run arguments in a JSON format

ew-cli 0.10.9

• Improved network failure handling in ew-cli. We now retry failed requests in a more robust way, and we also increased some of the longer network call timeouts (like file uploads). This should help in cases where you are running many tests in parallel from the same CI job.

gradle-plugin 0.16.2

• Bumped default ew-cli to 0.10.9

GitHub action emulator-wtf/run-tests@v0.9.9

• Bumped default ew-cli to 0.10.9
• Updated subaction setup-ew-cli to v0.9.9

Test execution

• Fixed a case where test runs with model=Tablet10 would capture corrupted video due to mismatched encoder parameters.

Test execution

• Fixed an escaping issue where environment variables containing the backslash \ character were not passed correctly to the test runner. You were most likely to encounter this when using regexes in your test runner arguments.
• Added a workaround for a case where a single test method was passed as a --test-target and it expanded to multiple parametric test cases. This was previously causing the test run to fail and works now without issue.

ew-cli 0.10.8

• The CLI is now better at printing some unexpected errors, for instance when misconfiguring the API token.

Emulators

• All of our emulators are now in the Atlantic/Reykjavik timezone (same as UTC). This should improve compatibility with tests that use the system timezone identifier and don’t support the Etc/UTC timezone identifier.

Test execution

• If your test manages to crash the surfaceflinger process and the whole System UI with it, we’ll trigger an automatic retry of the test run. This is a rare event that may happen with tests making heavy use of surfaces like utilizing the camera preview. The retry does not add any billable minutes to your account. NOTE: just like with flaky repeats, we do not attempt any retries when using the side-effects flag.

emulator.wtf web app

• Fixed a bug where flaky test runs were shown as failed on the test runs page in the emulator.wtf web app.

emulator.wtf web app

• The test results link now always displays the model/version combination, even if the test run was run on a single combination.

gradle-plugin 0.16.0

• The Gradle plugin now clears the output directory before running, fixing issues where you would get mixed outputs from multiple test runs when running the task repeatedly.

gradle-plugin 0.16.1

• Updated the transitive org.json library to a newer version.

GitHub action emulator-wtf/run-tests@v0.9.8

• Updated subactions to emulator-wtf/invoke@v0.9.7 and emulator-wtf/setup-ew-cli@v0.9.8 to use the node20 GitHub Actions runtime.
• Bumped default ew-cli to 0.10.7

Test execution

• Fixed an issue where sharding together with the useTestStorageService flag would run all the tests again in every shard. NOTE: there is a known issue about AndroidJUnitRunner 1.5.2 or earlier, we will now warn you if you happen to have this combination.
• Also fixed a case where --test-targets would not work as expected when used together with balanced sharding.

ew-cli 0.10.8

• We now display branch names and PR links in the test results list view. This makes it easier to understand which branch and PR a test run is associated with. The PR link is clickable and will take you to the PR on GitHub. This feature is available for test runs started after this release.

emulator.wtf web app

• The test runs page is now the default homepage in the emulator.wtf web app instead of the tokens page.

ew-cli 0.10.6

• ew-cli fails fast when trying to run tests on an apk containing only ARM binaries - our emulators are x86/x86_64.