Sharding

Sharding means splitting your tests across multiple same-configuration devices and running them in parallel. It’s a great way to speed up running larger test suites.

Sharding algorithms

emulator.wtf currently supports five sharding algorithms:

targeted runtime sharding, where the number of shards is dynamic and dictated by an overall runtime target (e.g. spawn N shards to run tests in 2 minutes)
balanced sharding, a custom emulator.wtf algorithm where we distribute tests based on runtime using historical test run data
even sharding, a custom emulator.wtf algorithm using a heuristic
the standard uniform sharding provided by AndroidJUnitRunner
explicit sharding where test targets running in shards need to be specified manually.

Sharding algorithm has a great effect on overall test runtime - the test will run as long as the longest shard does. A good algorithm will give you a linear speedup compared to no sharding, e.g. running on 10 shards should speed up your tests 10 times.

Targeted runtime sharding

Targeted runtime sharding can be used with --shard-target-runtime Nm where N is the number of minutes the test should run for. The underlying algorithm will spawn as many shards as needed to run the tests in the given time. For example, a test suite with a total runtime of 10 minutes will run approximately 5 shards when invoked with --shard-target-runtime 2m. This option works great if you want to blanket-configure a bunch of modules in one go and don’t want to worry about the exact number of shards as your test suite grows or shrinks.

Note: targeted runtime sharding kicks in after you have run your test at least once. In case you have no historical data, it will fall back to a default of 60 tests per shard.

Balanced sharding

Balanced sharding is similar to targeted runtime sharding in the sense that it is using historical test runtime data. Instead of targeting a specific runtime it will try to evenly spread tests across the shards instead. In case there is no historical data available, it will fall back to even sharding, detailed below.

Even sharding

Even sharding can be used with --num-shards N where N is the desired number of shards. The underlying algorithm will spread tests evenly across all shards using a heuristic. For example, a test suite with 1000 tests will run 100 tests per shard when invoked with --num-shards 10. Even sharding has a small overhead (~5s) for discovering the list of tests in the apk.

Uniform sharding

Uniform sharding can be used with --num-uniform-shards N where N is the number of shards. This will use the standard numShards and shardIndex environment variables to split the tests randomly across the shards.

Due to it’s random nature uniform sharding can be quite non-uniform in some cases, leading to wildly different runtimes between shards and sometimes failing tests outright due to some shards not having any tests at all.
We recommend uniform sharding only for cases where strict compatibility between AndroidJUnitRunner and emulator.wtf is required, for all other cases you should prefer even sharding (--num-shards) over uniform sharding.

Explicit sharding

It’s also possible to manually control which test packages, classes and methods end up in which shard. This can be useful if you have a better split heuristic than ours or perhaps some extra layer of abstraction like Flank on top of emulator.wtf that controls sharding.

To control sharding manually add repeated --test-targets-for-shard [target] arguments to the invocation. Each repeated target will specify the packages, methods or classes to include in that shard. The target can have one of the following forms:

package com.foo includes all test classes and methods from the com.foo package
class com.example.Foo includes all test methods from class Foo
class com.example.Foo#myTestMethod includes only myTestMethod test from com.example.Foo

To list multiple targets of the same type (e.g. multiple classes), use comma separated values:

class com.example.Foo,com.example.Bar will include test from both com.example.Foo and com.example.Bar in a single shard

To mix multiple target types in a single shard like a package and a class, use semicolons:

package com.example;class com.foo.MyTestClass will include all classes from the com.example package and all test methods from com.foo.MyTestClass in a single shard

Outputs

When an output folder is specified (--outputs-dir) then every shard gets a separate output subfolder, denoted by the shard index and device model. The shards are zero-indexed: the first shard is 0, second is 1 and so on.

For example, when running a test with --num-shards 2 --device model=Pixel2 --device model=NexusLowRes --outputs-dir /tmp/foo then emulator.wtf will create 4 output folders:

/tmp/foo/Pixel2_api27/0 for the first shard on Pixel 2
/tmp/foo/Pixel2_api27/1 for the second shard on Pixel2
/tmp/foo/NexusLowRes_api27/0 for the first shard on NexusLowRes
/tmp/foo/NexusLowRes_api27/1 for the second shard on NexusLowRes

Each of these folders will have their own JUnit test result XML file, logcat output, pulled folders, etc.