Flaky Tests in Test Automation Clarified — Part 1: Flaky Tests are Common but people tend to Neglect them

Before addressing the flaky tests, we need to treat them honestly and seriously first.

In 2018, I watched the best presentation on CI/CD, “Continuous Integration at Facebook”. The presenter showed three characteristics of CI at Facebook:

The first item, “High-Signal”, means ideally the automated test failures are mostly genuine, not false positives which are test failures due to non-application reasons. A common cause for false positives is ‘flaky tests’.

In the article, I will just focus on the flaky tests in the most common form of functional testing: end-to-end web test automation.

Please note, Facebook listed “High-Signal” as the №1 issue in its CI/CD process. IMO, this is correct for real CI/CD.
When false positives are common, the team will stop trusting the test results, testing process, and the testing team. Without trust, there is really nothing then. That’s why we often see ‘fake automated testers’ sitting in the corner and the team totally don’t care about…