There is no “100% coverage” in Automated E2E Software Testing

It is simply not possible and neither necessary.

5 min readMar 28, 2022

It is well accepted that 100% test coverage (functional, not unit testing) from manual testing is not possible. However, some had unrealistic expectations of 100% coverage in E2E Test Automation. This is very wrong, this article will explain why.

“It is neither possible or necessary to achieve 100% coverage in software testing” — Zhimin Zhan

Table of Contents:
· 100% Test Coverage is simply Not Possible
· Test Execution Time?

100% Test Coverage is simply Not Possible

First of all, people saying 100% test coverage don’t understand test automation or software testing.

At a conference talk, I said, “I can easily come up with hundreds of automated test scenarios for a credit card payment user story, just on the single payment page”. Below is a partial list:

Can pay the full amount with Visa Credit Card
Incomplete credit card info (mandatory check), e.g. no CSC, CC number, Expiry Date, …
Credit card info not matching, e.g. wrong CSC, card number does not match type
Invalid expiry date, e.g. already expired, or too far in the future
Amount too small (threshold) or too large for credit card payment limit (set by the app, or/and by the credit card type)
There is not enough money left in the credit card
Enter negative amount
Partial payments (can be a lot of cases here)
Special characters in the cardholder name (e.g. Japanese or Arabic)
…
and different combinations of the above.

These are valid test scenarios. Some might argue a few of them are extreme. About 20 years ago, I did encounter one subtle failed scenario for a payment gateway: when the total length of the merchant name + credit card info exceeded a certain limit, the payment processing threw an error. I found out this strange error by luck by reading the payment…