Sharding means splitting your tests across multiple same-configuration devices and running them in parallel. It’s a great way to speed up running larger test suites.
emulator.wtf currently supports three sharding algorithms, the standard uniform sharding provided by AndroidJUnitRunner , even sharding, a custom emulator.wtf algorithm using an heuristic, and explicit sharding where test targets running in shards need to be specified manually.
Sharding algorithm has a great effect on overall test runtime - the test will run as long as the longest shard does. A good algorithm will give you a linear speedup compared to no sharding, e.g. running on 10 shards should speed up your tests 10 times.
Even sharding can be used with
--num-shards N where
N is the desired number
of shards. The underlying algorithm will spread tests evenly across all shards using
an heuristic. For example, a test suite with 1000 tests will run 100 tests per
shard when invoked with
--num-shards 10. Even sharding has a small overhead (~5s)
for discovering the list of tests in the apk.
Uniform sharding can be used with
--num-uniform-shards N where
N is the number
of shards. This will use the standard
variables to split the tests randomly across the shards.
Due to it’s random nature uniform sharding can be quite non-uniform in some cases, leading to wildly different runtimes between shards and sometimes failing tests outright due to some shards not having any tests at all.
We recommend uniform sharding only for cases where strict compatibility between
emulator.wtfis required, for all other cases you should prefer even sharding (
--num-shards) over uniform sharding.
It’s also possible to manually control which test packages, classes and methods end up in which shard. This can be useful if you have a better split heuristic than ours or perhaps some extra layer of abstraction like Flank on top of emulator.wtf that controls sharding.
To control sharding manually add repeated
--test-targets-for-shard [target] arguments to
the invoke. Each repeated target will specify the packages, methods or classes to include
in that shard. The
target can have one of the following forms:
package com.fooincludes all test classes and methods from the
class com.example.Fooincludes all test methods from class
class com.example.Foo#myTestMethodincludes only
To list multiple targets of the same type (e.g. multiple classes), use comma separated values:
class com.example.Foo,com.example.Barwill include test from both
com.example.Barin a single shard
To mix multiple target types in a single shard like a package and a class, use semicolons:
package com.example;class com.foo.MyTestClasswill include all classes from the
com.examplepackage and all test methods from
com.foo.MyTestClassin a single shard
When an output folder is specified (
--outputs-dir) then every shard gets a
separate output subfolder, denoted by the shard index and device model. The shards
are zero-indexed: the first shard is
0, second is
1 and so on.
For example, when running a test with
--num-shards 2 --device model=Pixel2 --device model=NexusLowRes --outputs-dir /tmp/foo
then emulator.wtf will create 4 output folders:
/tmp/foo/Pixel2_api27/0for the first shard on Pixel 2
/tmp/foo/Pixel2_api27/1for the second shard on Pixel2
/tmp/foo/NexusLowRes_api27/0for the first shard on NexusLowRes
/tmp/foo/NexusLowRes_api27/1for the second shard on NexusLowRes
Each of these folders will have their own JUnit test result XML file, logcat output, pulled folders, etc.