Skip to content

Test runners

Test runner is responsible for tests run and providing test result for us.

AndroidJunitRunner — Official solution and low-level instrument. It requires a lot of effort from engineers to run tests on CI and make them stable.

It's worth to mention — tools are getting better year by year. However, some basic functionality still doesn't work from the box properly.

1. Problems with AndroidJunitRunner:

  • Overcomplicated solution with clearing
    It would be good to have only one flag which does application clearing for us.
    It exists, however to have scalability on CI and opportunity to use filters, you still have to install test-services.apk and orchestrator.apk to each device manually

  • Impossibility to scale
    As soon as you started your tests, it's impossible to connect more devices to tests run on fly

  • Impossibility to prevent flakiness
    Flakiness is one of the main problems in instrumented testing. Test runner should play role as a latest flakiness protection level, like support retries from the box or other strategies

  • Impossibility to validate flakiness
    Flakiness can be validated by running each test multiple times, and if test pass N/N, it's not flaky. It would be great to launch each test 100 times by one command

  • Poor test report
    Default test report doesn't show any useful information. As an engineer, I want to see a video of the test, logs and to make sure that test hasn't been retried. Otherwise, I'd like to see retries and each retry video and logs.

  • Impossibility to retry
    It's possible to do only via special test rule which does retry for us. However, it's up to test runner to retry each test, as instrumented process can be crashed and device less reliable than host machine.
    Also, it should be possible to define maximum retry count: Imagine, your application crashed on start. We shouldn't retry each test in that case because there is no sense to overload build agents on CI.

  • Impossibility to record a video
    It's possible to achieve and implement manually, however It would be really great to have such functionality from the box

Almost all of that problems possible to solve, but it can take weeks or even months of your time. Beside running tests, you also need to care about writing tests which is challenging as well.
It would be great to have that problems solved from the box

2. Open source test runners

All of them used AndroidJunitRunner under the hood, as it's the only possibility tun run instrumented tests.

🟩 2.1 Marathon

Powerful and the most pragmatic test runner. All you need to do it's just to connect devices to adb, and Marathon will do the whole job for you.

➕ Stand-alone or Gradle Plugin
➕ Easy data clearing (without an Orchestrator)
➕ Flexible configuration with filters
➕ Flakiness strategies
➕ Dynamic test batching (test count/test duration)
➕ Smart retries with a quotas
➕ Screenshots & video out of the box
➕ Improved test report with video & logs
➕ Automatically rebalanced test execution if connecting/disconnecting devices on fly
➕ Pull files from the device after test run, e.g. allure-kotlin
➕ Basic Allure support out of the box
➕ adb client ddmlib replacement: Adam
➕ Cross-platform (iOS support)
➕ Fragmented test execution (similar to AOSP's sharding): split large testing suites into multiple CI builds
➕ Parallel execution of parameterised tests
➕ Interactions with adb/emulator from within a test (e.g. fake fingerprint or GPS)
➕ Code coverage support
➕ Testing multi-module projects in one test run
➕ Flakiness fixing mode to verify test passing probability improvements

➖ Doesn't auto-scale devices
(Marathon will utilise more devices in runtime if some other system connects more to the adb, but marathon itself will not spawn more emulators for you)
➖ HTML report doesn't contain test retry information (but the Allure report does)
➖ For complex test executions that solve test flakiness requires an installation of TSDB (InfluxDB or Graphite)

Documentation

🟩 2.2 Avito Test Runner

Powerful test runner. Works directly with Kubernetes

➕ Easy data clearing (without an Orchestrator)
➕ Auto-scaling on fly (There is a coroutine in the background which tries to connect more devices) ➕ Retries ➕ Good test report
➕ Unit tests support

➖ Complicated adoption
➖ No stand-alone solution

This test runner has been using by Avito company for 4+ years and runs thousands tests every day. It's not as powerful as Marathon, however it doesn't have an analogue in terms of auto scaling from the box.
If you want to run your UI tests on pull requests in a large teams, this test runner is one of the best option.

Engineers from Avito are ready to help with adoption. You can contact to Dmitriy Voronin

Documentation

🟩 2.3 Fork

➕ Retries
➕ Filters
➕ Good test report with flakiness
➕ Stand-alone or Gradle plugin ➕ Performance profiling with Chimprunner

➖ Data clearing
➖ Not actively maintains

🟩 2.4 Flank

➕ Don't need to care about Device infrastructure
➕ Easy device data clearing (With an Orchestrator internally)
➕ Stand-alone or Gradle plugin
➕ Huge variety for choosing devices (Emulators/Real devices)
➕ Good test report
➕ Additional gradle plugin: Fladle

➖ Paid service

🟥 2.6 Spoon

Deprecated and not maintained anymore. Do not use it

🟥 2.7 Composer

Deprecated and not maintained anymore. Do not use it

Conclusion

There is no right and wrong choice. As you see, all test runners have something unique.
Keep in mind that nowadays Marathon is the most powerful test runner and will be pragmatic choice for any team.

From the other side, if you need to have a result right here and right now and you are ready to pay, Flank or Fladle will be pragmatic options.