WhenWise Regression Test Suite Reaches 500 Selenium tests and ~300K Test Executions

Share WhenWise’s CT process, which enables daily deployment to production.

Today (2021–03–05), I reached Level 4 of AgileWay Continuous Testing Grading for my WhenWise app: 500 end-to-end/user-story-level (Selenium WebDriver) tests in the regression test suite.

This is my second time to reach Level 4. The first time was for ClinicWise on 2016–07–29.

Parallel Test Execution

A big challenge of executing automated end-to-end tests (in browsers) is the long feedback, as running a large number of UI tests will take a long time. In the case of the WhenWise regression test suite, it will take 4+ hours if running on a single machine. The only practical solution is parallel execution.

Running this suite in 9 BuildWise agents (concurrently) will only take ~35 minutes. Besides time-saving,

  • Test execution is much more stable via Auto-Retries
    Most of the 12 false-alarms (shown in the build report, top-right) were due to the chromedriver v87 not being compatible with self-updated Chrome v89 on one build agent (#11 Win10). BuildWise auto-retried the failed tests on another agent and passed!
The tests failed on Agent #11, and passed on another agent after auto-retry

Related readings: “Why Auto-Retry of Test Execution in a Test Framework is Wrong?

  • Cross-platform & Cross-browser Testing
    From the above screenshot, you can see that tests were on different platforms: Windows, macOS, and Linux.
    For some other projects, I configured agents with different browsers for cross-browser testing. For WhenWise, I just used Chrome.

Total test case executions: Nearly 300,000.

No Defect Tracking at all

WhenWise is developed and maintained in my spare time, along with other apps such as TestWise, BuildWise, SupportWise, and ClinicWise. With Continuous Testing, I don’t need to do defect tracking, not at all.

Test Script Framework

Raw Selenium WebDriver (Automation) + RSpec (syntax), no fancy ‘own framework’. I don’t use test recorders, not at all (despite that I created one myself).

Related readings:

Testing and CI tools

The test scripts are pure selenium tests in plain-text, they will work with other tools. Tools are for efficiency, and I am very efficient with the help of these two tools (disclaimer: I created them. You may use these tools in free mode).


None at all. I only care whether the build (to run the whole regression test suite) passes or not. If not, №1 priority is to resolve the issues. If there are changes made, I will deploy the new build to production on that day, upon getting a green build on the BuildWise CT server. There are only a handful of cases that I could not get a green build on the day. Then I will work on those first thing the next morning.

Sample Test

Below is one of 500 WhenWise tests: “A Business signs up with validation”.

The test scripts follow the “Maintainable Automated Test Design”, using Reusable Helper Function and Page Object Model.

The style of automated test scripts has been verified in the ‘battlefields’. I have been written thousands of automated tests in this style since 2006 for clients’ and my own projects. When working in the consulting role, I was always able to implement a few key automated tests and run them in BuildWise on the first day. So don’t believe a so-called fancy ‘new test tool/framework’, raw Selenium WebDriver has been, is, and will still be (in a foreseeable future) the best (and the easiest to learn, don’t believe fake testers’ labeling ‘steep learning curve’) test automation framework for web apps.

For more ideas on test script design, refer to my “Practical Web Test Automation” book. Here I just point out two attributes:

  • Business analysts/customers can understand the test scripts, after a quick explanation
  • One test step (by line) for each user operation, important for debugging and efficient maintenance.

Test Script Stats

| | 23365 | 304 | 510 | 18381 |
| | 9036 | 159 | 1483 | 6862 |
| | 801 | 5 | 61 | 618 |
| TOTAL | 33202 | | | 25861 |

Page class count: 159. The total number of test steps: 25,861
A green build means each of the 25K+ (some repeated execution of one step) test step pass!

Build History

The CT process detected many issues/defects, then I fixed them very quickly. (Again, I never used a Defect Tracking System for my own software development). I cannot emphasize the importance of this CT process for my app development. (note: the last 6 builds took a shorter time because I added 3 more BuildWise agents)

If you are interested in my approach and prefer finding out more in a structured way, check out my books:

Award-winning software developer, test automation & CT coach, author, and speaker. Help teams succeed with Agile/DevOps by implementing real Continuous Testing.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store