How to Implement Real Automated Regression Testing?

Strategy and practical advice on successful automated regression testing to enable daily production releases.

Zhimin Zhan
10 min readOct 10, 2021

--

In “Regression Testing Clarified”, I shared my thoughts and experience on Whats and Whys of automated regression testing. In this article, I will show Hows.

Table of Contents
· Strategy: Run all E2E Tests daily
· Discipline: Fixing broken builds is the highest priority
· How to Start?
· 100% coverage is impossible and unnecessary
· Specific Advice

Strategy: Run all E2E Tests daily

Set a clear goal and work towards it.

Goal: run all automated E2E UI tests every day and get a Green build by the end of the day.

Of course, there are exceptions, such as a quite bad check-in, achieving a green build every day is not always possible. However, every team member needs to understand the team’s priority: GREEN BUILD, passing all automated (unit, API, and more importantly, E2E UI) tests.

Some readers might wonder:

  • This sounds easy.

No, it is not with a large test suite. For example, there are 28000+ test steps for WhenWise’s (one of my apps) automated regression suite. A green build means each of these 28,000 test steps must pass.

I am not trying to scare you off. Focusing on the goal every day will make this challenging task easier, “Slow And Steady Win The Race”.

  • This is too vague.

I do have a list of specific advice in the next section. Here, I will highlight one important attribute of testing: feedback. Focus on the goal and everything will become natural if you adjust your approach based on feedback. During the journey (Test Automation and Continuous Testing), if you encounter challenges or difficulties, they are the feedback to you. Be open-minded and make changes accordingly to achieve the goal daily. I will show you some examples here:

(1) The execution of certain test scripts fails intermittently.
This means that your test script is not reliable. Check out this article: Working Automated Test ≠ Good Reliable Test.

--

--

Zhimin Zhan

Test automation & CT coach, author, speaker and award-winning software developer. Help teams succeed with Agile/DevOps by implementing real Continuous Testing.