WhenWise Regression Test Suite Reaches 500 Selenium tests and ~300K Test Executions

Share WhenWise’s CT process, which enables daily deployment to production.

Zhimin Zhan
5 min readMar 5, 2021


Today (2021–03–05), I reached Level 4 of AgileWay Continuous Testing Grading for my WhenWise app: 500 end-to-end/user-story-level (Selenium WebDriver) tests in the regression test suite.

This is my second time to reach Level 4. The first time was for ClinicWise on 2016–07–29.

Parallel Test Execution

A big challenge of executing automated end-to-end tests (in browsers) is the long feedback, as running a large number of UI tests will take a long time. In the case of the WhenWise regression test suite, it will take 4+ hours if running on a single machine. The only practical solution is parallel execution.

Running this suite in 9 BuildWise agents (concurrently) will only take ~35 minutes. Besides time-saving,

  • Test execution is much more stable via Auto-Retries
    Most of the 12 false-alarms (shown in the build report, top-right) were due to the chromedriver v87 not being compatible with self-updated Chrome v89 on one build agent (#11 Win10). BuildWise auto-retried the failed tests on another agent and passed!
The tests failed on Agent #11, and passed on another agent after auto-retry

Related readings: “Why Auto-Retry of Test Execution in a Test Framework is Wrong?

  • Cross-platform & Cross-browser Testing
    From the above screenshot, you can see that tests were on different platforms: Windows, macOS, and Linux.
    For some other projects, I configured agents with different browsers for cross-browser testing. For WhenWise, I just used Chrome.

Total test case executions: Nearly 300,000.



Zhimin Zhan

Test automation & CT coach, author, speaker and award-winning software developer. Help teams succeed with Agile/DevOps by implementing real Continuous Testing.