Practical DDT: Randomized data scenarios in one automated test script + Continuous Testing

Introduce a practical DDT technique to have a random input data combination in one automated UI test.

This article is included in my “How to in Selenium WebDriver” series.

A reader of “Data-Driven Testing Clarified” asked me how I do DDT. The drawbacks of traditional DDT, as I put in that article, are being difficult to maintain and debug. Therefore, I will try to avoid it in practice unless it is really necessary. In this article, I will share my way to handle multi-data scenarios in one automated UI test case.

To be realistic, we cannot afford DDT most of the time

The reason is simple: execution time. We often see a long list of scenarios in manual testers’ so-called test design spreadsheet. Every row represents one data scenario and expectations.

Let’s do a quick and simple maths calculation. Supposedly on average, the execution time of a test case is 30 seconds (which is not bad for a UI test). 120 tests (i.e. scenarios) will take 1 hour. I have seen over 100 scenarios for just a single user story. In that case, how many user stories can the automated test suite realistically cover?

“If you can manage executing tens of thousands of automated test cases every day and maintain all tests well, you may do spreadsheet-driven DDT. I cannot.” — Zhimin Zhan

Clearly, the data scenarios to be included in automated tests are limited. Please note that there is no judgement to the manual testing process here, I just focus on automated UI testing.

Test Automation is about providing quick feedback (fast regression testing) to the team. Therefore, we cannot apply the manual testing mindset of listing all possible cases to E2E automated testing. This does not mean that we cannot do multi-scenarios either. I will show my approach later.

Diminishing returns on more data scenarios

Let me illustrate with an example: business sign-up for WhenWise (a service booking app created by me).