WhenWise Regression Test Suite Reaches 500 Selenium tests and ~300K Test Executions
Share WhenWise’s CT process, which enables daily deployment to production.
Today (2021–03–05), I reached Level 4 of AgileWay Continuous Testing Grading for my WhenWise app: 500 end-to-end/user-story-level (Selenium WebDriver) tests in the regression test suite.
This is my second time to reach Level 4. The first time was for ClinicWise on 2016–07–29.
Parallel Test Execution
A big challenge of executing automated end-to-end tests (in browsers) is the long feedback, as running a large number of UI tests will take a long time. In the case of the WhenWise regression test suite, it will take 4+ hours if running on a single machine. The only practical solution is parallel execution.
Running this suite in 9 BuildWise agents (concurrently) will only take ~35 minutes. Besides time-saving,
- Test execution is much more stable via Auto-Retries
Most of the 12 false-alarms (shown in the build report, top-right) were due to the chromedriver v87 not being compatible with self-updated Chrome v89 on one build agent (#11 Win10). BuildWise auto-retried the failed tests on another agent and passed!
Related readings: “Why Auto-Retry of Test Execution in a Test Framework is Wrong?”
- Cross-platform & Cross-browser Testing
From the above screenshot, you can see that tests were on different platforms: Windows, macOS, and Linux.
For some other projects, I configured agents with different browsers for cross-browser testing. For WhenWise, I just used Chrome.
Total test case executions: Nearly 300,000.
