There is no “100% coverage” in Automated E2E Software Testing
It is well accepted that 100% test coverage (functional, not unit testing) from manual testing is not possible. However, some had unrealistic expectations of 100% coverage in E2E Test Automation. This is very wrong, this article will explain why.
“It is neither possible or necessary to achieve 100% coverage in software testing” — Zhimin Zhan
Table of Contents:
· 100% Test Coverage is simply Not Possible
· Test Execution Time?
100% Test Coverage is simply Not Possible
First of all, people saying 100% test coverage don’t understand test automation or software testing.
At a conference talk, I said, “I can easily come up with hundreds of automated test scenarios for a credit card payment user story, just on the single payment page”. Below is a partial list:
- Can pay the full amount with Visa Credit Card
- Incomplete credit card info (mandatory check), e.g. no CSC, CC number, Expiry Date, …
- Credit card info not matching, e.g. wrong CSC, card number does not match type
- Invalid expiry date, e.g. already expired, or too far in the future
- Amount too small (threshold) or too large for credit card payment limit (set by the app, or/and by the credit card type)
- There is not enough money left in the credit card
- Enter negative amount
- Partial payments (can be a lot of cases here)
- Special characters in the cardholder name (e.g. Japanese or Arabic)
- and different combinations of the above.
These are valid test scenarios. Some might argue a few of them are extreme. About 20 years ago, I did encounter one subtle failed scenario for a payment gateway: when the total length of the merchant name + credit card info exceeded a certain limit, the payment processing threw an error. I found out this strange error by luck by reading the payment…