Irrational and False Excuses for Web Test Automation Failures
Real reason: Incompetence. Just admit ‘lack of the skills’, change the mindset, and seek professional help if necessary.
FACT: Web Test Automation attempts failed in most software projects.
(disagree? do an instant and objective assessment)
Over the years, I have heard of many excuses, such as:
- “It worked well on my last project”
- “Tests failed because the application changed”
- “The test scripts are fine, only test data is no longer valid”
- “Our testers do not have the capability”
- “The automation framework selected by the previous test architect was bad”
- “Automated end-2-end tests are known as fragile”
To some people, these excuses might sound reasonable. However, a real test automation engineer will conclude that those excuses are plainly wrong, very wrong. ‘Irrational’ might sound a strong word, but you may agree after reading my explanation, as these excuses are so obviously wrong and indefensible.
There is really only one reason: Incompetence.
In this article, I will list 14 common false excuses. But first, I will list some facts that might help you better understand the reasonings.
FACT: Automated UI Testing
- Brittle, prone to application changes (comparing to unit/API tests)
- Long execution time (comparing to unit/API tests)
- Testing for all web applications, technically speaking, is the same. If one tester can do test automation successfully at one job, he shall be able to replicate the success on other jobs, with virtually no changes.
FACT: Selenium WebDriver
- the only automation framework that is W3C compliant
- free (in both freedom and cost)
- the only automation framework supported by all major browser vendors
- popular, open-source and well maintained
- used and recommended by top IT companies such as Facebook, Microsoft and Google.
Now, let me expose those fake excuses or lies.
1. “It worked well on my last project”
This is a “Dog ate my homework” level lie used by naughty children. My question is “How about this project?” Show me something today, then tomorrow, and onwards. A real test automation engineer can implement a few key tests on day 1. The reason is simple: web test automation is pretty much the same for all projects.
Given the fact that web projects are 100% based on W3C standards now (previously not the case for Flash and Silverlight, which both died), it is silly that a fake professional pretends he/she “can do test automation” and IT management let them pretend for more than a week.
I take test automation consulting by days. Companies can engage my service for 1 day, and I delivered value on that day. For my recent consultation projects, if possible, I would ask a manual tester (with domain knowledge) to work with me right after greetings. If other people (engineers, managers and testers) are interested, I suggested connecting the computer to a projector: I develop real automated tests (in raw Selenium with RSpec) live against a web application which is new to me.
People attended the session were usualy impressed by my efficiency. What impressed them most is that the manual tester (and some audience) actually understood the test scripts (in Ruby) and had confidence to do modifications.
2. “Test execution failed because the application changed”
What?! Is “application changes” a surprise to you? Come on, this is normal. How could a software/test engineer even say that? If the application never changes, there is no need for regression testing.
3. “The test scripts are fine, only test data is no longer valid”
Wrong, test data is a part of test scripts.
4. “Our testers do not have the capability”
This is often used by managers and tech-leads (such as Test Architect, Chief/Principal software engineers). Yes, it was the fact. However, have you done anything to improve the situation?
Moreover, do the managers/tech-leads know what are the skills/capabilities required? I doubt it.
5. “Test Automation is expensive, a XXX tool/runtime license costs $$$$”
“For all of our end-to-end tests at Facebook we use WebDriver, WebDriver is an open-source JSON wired protocol, I encourage you all check it out if you haven’t already. ”
- Katie Coons, software engineer at Facebook, “Continuous Integration at Facebook” presentation at F8 2015.
6. “Still working on some issues of our own-framework, …”
Crazy. First of all, very very few engineers in this world can create a new working framework. This so-called “own-framework” is normally just an unnecessary layer on top of Selenium. From my experience, each of these ‘new frameworks’ failed for a silly and simple reason: this badly-designed unnecessary layer was buggy.
Why not just use raw Selenium WebDriver like Facebook, Google and Microsoft (they recommend WebDriver, not another fancy name as a ‘new framework’)? It is fair to say that the engineers in your company are not as good as those at these IT giants.
For more, please read my other article: “Please, Not Another Web Test Automation Framework, Just Use Raw Selenium WebDriver”.
7. “Selenium WebDriver has a steep learning curve”
Totally wrong! Selenium WebDriver is the easiest to learn test automation framework. Most people I mentored could work on real tests after one-day training. My 12-year-old daughter learned to write the raw Selenium WebDriver tests, so could you and your teenage child under proper mentoring.
Note: I am probably more objective than most of the test engineers, as my Selenium Recipes Book series cover all of Selenium’s five official languages.
I could develop quite good Selenium tests in Java/JS/C#/Python, but I don’t have confidence in mentoring others to achieve the productivity required for successful test automation.
Above all, scripting automated tests in Ruby is efficient, easy, creative, and fun!
8. “The reason we have to use Java/JS/C# is that we have the skills to maintain”.
Untrue. Most people will agree that, compared to on-going maintenance, creating automated tests is a lot easier. Now check the reality. After several months, the team might have been struggling with test creation by using the ‘preferred’ language. There is no point to talk about reaching the finishing line if already failed at the starting line.
The sensible way is to use Ruby, a recommended language by the Agile Testing book. Maybe with the help of a test automation coach, the team can create a handful of key tests every day.
The real reason is that a tech-lead only knows one language and fear anything unfamiliar. I have met quite a number of this kind of tech-leads. Believe it or not, I think some of them actually wanted the test automation to fail. They prefer fake test automation, as no changes will be required. If test automation succeeded, they won’t be comfortable adjusting to a new regime.
9. “Selenium WebDriver execution is not stable”, “Selenium could not test AJAX”, …
Totally untrue. Selenium WebDriver is the most stable web automation framework. Remember, web test automation depends on browsers, and all major browser vendors only support the WebDriver standard. You may use ‘unstable’ to describe any other framework such as UTP and Cypress, but not Selenium WebDriver.
Testing AJAX with raw Selenium Webdriver, if done properly, is far easier than the others. See this article: “Test AJAX Properly and Efficiently with Selenium WebDriver”.
10. “The automation framework selected by the previous test architect was bad”
A typical office politics. Though it might be true, a test architect is only allowed to use this excuse for one day. Web test automation skills and practices are fully transferable. A real test automation engineer, like me and my teenage daughter, can implement a few key tests on the first day.
11. “Automated end-2-end tests are known as fragile”
Yes, that’s why you are hired to implement it. It is still professional for engineers to reject test automation because they are unable to accomplish the task. However, if someone did take the task (and getting paid) and failed it, using this excuse is quite low.
UI Test Automation can be done, see my article: “WhenWise Regression Test Suite Reaches 500 Selenium tests and ~300K Test Executions”. From my experience, after having done one successful web test automation properly, I can always replicate the success in other projects.
Why not change to Ruby? Ruby is the best scripting language recommended by the classic ‘Agile Testing’ book. The real reason: Test Lead does not know Ruby, and lacks a learning altitude, which is more concerning.
13. “The automated UI test scripts are hard to read, difficult to maintain by nature”
Not necessarily. Writing tests in RSpec (Ruby) will greatly enhance the readability of the test scripts. I could maintain 617 and 500 selenium tests for my two apps ClinicWise and WhenWise, respectively. Of course, efforts are required but maintenance is surely possible. Otherwise, why is your company trying to do it anyway?
14. “The DevOps team is no good, unable to run our tests reliably in Jenkins”
I will write another article on silly and false excuses for CI/CD/CT failures. Here I would like to focus on test automation. My question is: “How reliable are your tests?” If your team members run the same test script at different times, are you confident to get 10 successful test executions?
In my opinion, if the average test pass rate is <95%, don’t bother running your test suite in the CI/CD server. Stabilize your tests first!
Read this article: “Working Automated Test ≠ Good Reliable Test”.
Below are recent test execution stats for WhenWise, one of my own web apps. With ~500 Selenium tests, 99.5% pass rate was in last month and 97.9% over the last 3 years.