Sharding

Sharding means splitting your tests across multiple same-configuration devices and running them in parallel. It’s a great way to speed up running larger test suites.

emulator.wtf currently supports three sharding algorithms, the standard uniform sharding provided by AndroidJUnitRunner , even sharding, a custom emulator.wtf algorithm using an heuristic, and explicit sharding where test targets running in shards need to be specified manually.

Sharding algorithm has a great effect on overall test runtime - the test will run as long as the longest shard does. A good algorithm will give you a linear speedup compared to no sharding, e.g. running on 10 shards should speed up your tests 10 times.

Even sharding can be used with --num-shards N where N is the desired number of shards. The underlying algorithm will spread tests evenly across all shards using an heuristic. For example, a test suite with 1000 tests will run 100 tests per shard when invoked with --num-shards 10. Even sharding has a small overhead (~5s) for discovering the list of tests in the apk.

Uniform sharding can be used with --num-uniform-shards N where N is the number of shards. This will use the standard numShards and shardIndex environment variables to split the tests randomly across the shards.

Due to it’s random nature uniform sharding can be quite non-uniform in some cases, leading to wildly different runtimes between shards and sometimes failing tests outright due to some shards not having any tests at all.

We recommend uniform sharding only for cases where strict compatibility between AndroidJUnitRunner and emulator.wtf is required, for all other cases you should prefer even sharding (--num-shards) over uniform sharding.

It’s also possible to manually control which test packages, classes and methods end up in which shard. This can be useful if you have a better split heuristic than ours or perhaps some extra layer of abstraction like Flank on top of emulator.wtf that controls sharding.

To control sharding manually add repeated --test-targets-for-shard [target] arguments to the invoke. Each repeated target will specify the packages, methods or classes to include in that shard. The target can have one of the following forms:

• package com.foo includes all test classes and methods from the com.foo package
• class com.example.Foo includes all test methods from class Foo
• class com.example.Foo#myTestMethod includes only myTestMethod test from com.example.Foo

To list multiple targets of the same type (e.g. multiple classes), use comma separated values:

• class com.example.Foo,com.example.Bar will include test from both com.example.Foo and com.example.Bar in a single shard

To mix multiple target types in a single shard like a package and a class, use semicolons:

• package com.example;class com.foo.MyTestClass will include all classes from the com.example package and all test methods from com.foo.MyTestClass in a single shard

When an output folder is specified (--outputs-dir) then every shard gets a separate output subfolder, denoted by the shard index and device model. The shards are zero-indexed: the first shard is 0, second is 1 and so on.

For example, when running a test with --num-shards 2 --device model=Pixel2 --device model=NexusLowRes --outputs-dir /tmp/foo then emulator.wtf will create 4 output folders:

• /tmp/foo/Pixel2_api27/0 for the first shard on Pixel 2
• /tmp/foo/Pixel2_api27/1 for the second shard on Pixel2
• /tmp/foo/NexusLowRes_api27/0 for the first shard on NexusLowRes
• /tmp/foo/NexusLowRes_api27/1 for the second shard on NexusLowRes

Each of these folders will have their own JUnit test result XML file, logcat output, pulled folders, etc.