Why Auto-Retry of Test Execution in a Test Framework is Wrong?

Recently, I noticed the ‘Auto-Retry’ feature in some ‘new test frameworks’, such as Jest and Cypress. It is wrong! I pioneered (and implemented) the ‘auto-retry’ back in 2007 and have been using it ever since. The difference is that my ‘Auto-Retry’ is in Continuous Testing Server, not the automation or syntax framework. BuildWise, the CT server I created with the Auto-Retry feature, received Ruby Award from Matz (the creator of Ruby, a true software legend).

“Infrastructure failures get retried” — from an excellent CI presentation “CI at Facebook”

Let’s see a sample ‘auto-retry’ syntax in the Jest test framework:

jest.retryTimes(3) //set maximum retries number

describe('Test', () => {
    test('Test_Test', async () => {
        //test code
        ...
    })
})

First of all, I am sure most will agree that adding retryTimesto a test framework does not sound ‘right’. Still, why it is bad?

1. It does not address the root problem.

As we know, automated UI test execution is prone to infrastructure issues. Common reasons that a test execution failed (not due to a real bug) are:

• the server is under the load
• dynamic operation (AJAX or JS)