Test execution

emulator.wtf now supports capturing per-test videos!

This has been the most requested feature by far. We now support capturing frame-precise test recordings of individual tests. To enable this we need some help from the instrumentation test runner to capture exact timestamps of the tests.

To enable this feature, make sure you have the record video flag set and add the following library to your Gradle build:

androidTestImplementation("wtf.emulator:test-runtime-android:0.2.0")

test-runtime-android 0.2.0

Initial public release of wtf.emulator:test-runtime-android:0.2.0.

• include this library in the test apk via androidTestImplementation to enable per-test video recordings (see above)

emulator.wtf web app

• Improved internal handling of refresh-access token pairs to improve web app performance

emulator.wtf web app

• Fixed developer role missing permissions to delete own tokens

emulator.wtf API

• Improved backend performance when spawning hundreds of test runs in quick succession

emulator.wtf web app

• Fixed an issue with GitHub SSO login

Emulators

• Disabled Google Chrome’s welcome screen, making it easier to run tests that use the browser.
• Added API 25, 28 and 32 emulators. List of device combinations added:
  • device=Pixel2,version=25
  • device=Pixel2,version=28
  • device=Pixel2,version=32
  • device=Pixel7,version=25
  • device=Pixel7,version=28
  • device=Pixel7,version=32
  • device=Tablet10,version=25
  • device=Tablet10,version=28
  • device=Tablet10,version=32
  • device=NexusLowRes,version=25
  • device=NexusLowRes,version=28
  • device=NexusLowRes,version=32
  • device=Monitor,version=25
  • device=Monitor,version=28
  • device=Monitor,version=32
  • See all available emulator images here.

emulator.wtf web app

• Improved the performance of displaying and paging test runs in the web app

emulator.wtf web app

The test run result links now contain a few details about the run:

• display name of the test
• timestamp and total execution time
• number of devices and shards used
• total billable minutes consumed by the run
• input test run arguments in a JSON format

ew-cli 0.10.9

• Improved network failure handling in ew-cli. We now retry failed requests in a more robust way, and we also increased some of the longer network call timeouts (like file uploads). This should help in cases where you are running many tests in parallel from the same CI job.

gradle-plugin 0.16.2

• Bumped default ew-cli to 0.10.9

GitHub action emulator-wtf/run-tests@v0.9.9

• Bumped default ew-cli to 0.10.9
• Updated subaction setup-ew-cli to v0.9.9

Test execution

• Fixed a case where test runs with model=Tablet10 would capture corrupted video due to mismatched encoder parameters.

Test execution

• Fixed an escaping issue where environment variables containing the backslash \ character were not passed correctly to the test runner. You were most likely to encounter this when using regexes in your test runner arguments.
• Added a workaround for a case where a single test method was passed as a --test-target and it expanded to multiple parametric test cases. This was previously causing the test run to fail and works now without issue.

ew-cli 0.10.8

• The CLI is now better at printing some unexpected errors, for instance when misconfiguring the API token.

Emulators

• All of our emulators are now in the Atlantic/Reykjavik timezone (same as UTC). This should improve compatibility with tests that use the system timezone identifier and don’t support the Etc/UTC timezone identifier.

Test execution

• If your test manages to crash the surfaceflinger process and the whole System UI with it, we’ll trigger an automatic retry of the test run. This is a rare event that may happen with tests making heavy use of surfaces like utilizing the camera preview. The retry does not add any billable minutes to your account. NOTE: just like with flaky repeats, we do not attempt any retries when using the side-effects flag.

emulator.wtf web app

• Fixed a bug where flaky test runs were shown as failed on the test runs page in the emulator.wtf web app.

emulator.wtf web app

• The test results link now always displays the model/version combination, even if the test run was run on a single combination.

gradle-plugin 0.16.0

• The Gradle plugin now clears the output directory before running, fixing issues where you would get mixed outputs from multiple test runs when running the task repeatedly.

gradle-plugin 0.16.1

• Updated the transitive org.json library to a newer version.

GitHub action emulator-wtf/run-tests@v0.9.8

• Updated subactions to emulator-wtf/invoke@v0.9.7 and emulator-wtf/setup-ew-cli@v0.9.8 to use the node20 GitHub Actions runtime.
• Bumped default ew-cli to 0.10.7

Test execution

• Fixed an issue where sharding together with the useTestStorageService flag would run all the tests again in every shard. NOTE: there is a known issue about AndroidJUnitRunner 1.5.2 or earlier, we will now warn you if you happen to have this combination.
• Also fixed a case where --test-targets would not work as expected when used together with balanced sharding.

ew-cli 0.10.8

• We now display branch names and PR links in the test results list view. This makes it easier to understand which branch and PR a test run is associated with. The PR link is clickable and will take you to the PR on GitHub. This feature is available for test runs started after this release.

emulator.wtf web app

• The test runs page is now the default homepage in the emulator.wtf web app instead of the tokens page.

ew-cli 0.10.6

• ew-cli fails fast when trying to run tests on an apk containing only ARM binaries - our emulators are x86/x86_64.

Test execution

• Fixed an issue where certain webpages could crash the whole emulator, leaving tests in a broken state.
• We also improved our detection of various emulator failure scenarios.

ew-cli 0.10.5

• Pulled directory sizes are now logged correctly in ew-cli output.

Emulators

• Emulators on API levels 33 and 34 can now benefit from GPU acceleration. Use the gpu=auto flag (e.g. --device model=Pixel2,version=34,gpu=auto) to enable GPU acceleration. Read our blog post on the benefits of GPU acceleration here.
• Disabled GPU acceleration for Automated Test Device images. These devices do not benefit from GPU acceleration, and it was causing issues.

ew-cli 0.10.4

• When using the --json flag or the summary output format, the consumed test run minutes are now available in the JSON outputs of test runs. The consumed test run minutes are available under the billableMinutes key in the JSON output. This is useful when estimating the cost of using emulator.wtf in the long-term. Read more on how we calculate billableMinutes here.

emulator.wtf web app

• The Usage view in the emulator.wtf web app now defaults to the past 30 days instead of showing usage from the beginning of the current calendar month.
• Fixed a bug where canceled test runs during the apk upload phase would show up as perpetually running in the emulator.wtf webapp test runs list view.

emulator.wtf web app

• The audit log can now be exported either as a CSV or a JSON file. The file contains a log of all the actions taken by users in the emulator.wtf web app. NOTE: audit log is an enterprise plan only feature.

Test execution

• Added validation of uploaded APK files to prevent corruption issues due to incomplete uploads. This issue compounded with the test file cache where we would keep the corrupted apks around and future test files would be corrupted as well.

emulator.wtf web app

• Google Workspaces is now a first-class citizen in our SSO/SAML login flow. Read the integration docs here. NOTE: enterprise plan only.

emulator.wtf web app

• Improved markdown rendering in the AI-powered test failure helper to better handle code blocks and lists.

emulator.wtf web app

• Fixed flaky tests sometimes still showing up as failed in test runs list.