My Innovative Solution to Continuous Testing: Auto-Retry

A Continuous Testing feature that greatly reduces false alarms and increases the chance of getting a green run.

This is included in the “My Innovative Solution to Test Automation and Continuous Testing” series.

The automated tests here I refer to are Automated End-to-End (via UI) tests for web/mobile/desktop testing, which will be raw Selenium WebDriver/Appium tests for me. Most Automated Testers nowadays have heard of the “Test Retries” feature (in Cypress and Playwright).

I consider myself a pioneer in ‘Test Retries’, although some may perceive it as boasting. As with most assertions, I back them up with concrete evidence. I conceived the idea and successfully implemented it in 2009 (predating the existence of Cypress & Playwright) using a customized CruiseControl Server, the grand-daddy of CI servers. For more, refer to “My Continuous Testing Journey.”

The reasoning behind the “Auto-Retry” concept is simple: if the overall non-test-script-related flakiness rate is 1%, retry it once, and then its stability rate is increased to 99X. Therefore, only needs to be done once. Also, the retry shall not be immediate. Why? If a test step fails due to a heavy server load, retrying 0.1 seconds later is…