Estimating Test Automation Story Points is a Total Waste of Time
95+% of test automation efforts are on test maintenance & regression testing
Many fake “agile project managers/agile coaches” are fixated with Velocity Charts. The so-called “velocity” is based on the rate of completing the story points estimated by the team. As a result, “story estimation sessions” are a part of “Agile Ceremonies” in some projects, along with“stand-up meetings”, “retrospectives” and “sprint planning”. (see the end of the article for Kent Beck’s, the father of Agile, view on these ‘agile ceremonies’)
I do not estimate points/time for developing automated tests for a user story. The reason is simple: it does not make sense.
The below is a slide from a presentation I gave in 2015.
It has three charts:
- A typical Scrum Sprint: 10 working days (2 weeks)
The planning sessions are on the first day, and the showcase & retrospective are on the last day. Software development activities (coding, testing) occurred on the 8 days in the middle.
- Assume the team is trying to do Agile and a completed user story is verified by one or more automated end-to-end tests. From Sprint 2 onwards, the testing activities are consist of :
- Prepare the automated tests for this sprint
- Develop the new automated tests
- Regression, make sure the automated tests created in the previous sprints are still valid (common sense, right?). Besides test execution, this includes test maintenance such as updating test scripts/data along with application changes, stabilizing test execution, and refining/refactoring test scripts.
- The regression effort, as you can see, will accumulate quickly along with the development. Maybe on Sprint 3 (or maybe even earlier), the regression testing effort will exceed all other test automation activities.
When I showed the above charts one by one, most of the audience showed signs of agreement with me. However, the natural conclusion is quite different from their previous thoughts, that is, the maintenance efforts of previous tests will far exceed that of developing new automated tests of the current sprint. Then I said, “so, it is meaningless to estimate the efforts on automated-test creation for new stories, as the majority of test automation engineer efforts are on regression and maintaining existing tests”. From my memory, no single disagreement from the audience.
Two simple facts of UI test automation:
1.UI test execution takes longer time (much longer than unit/api tests)
2. one simple application change may affect an unknown number of automated tests.
I could sense that the most of audiences were quite shocked by the finding, and the reasoning is so strong that it is hard to debate. One of them asked: “I concur, but it will be hard to convince managers”. Hence, I shared a story.
Once I worked in a small software company. A co-founder (without coding/testing background) was particularly fixated with story-points. He printed out a stack of “Story Estimation Cards” like the below, to be used in the ‘story points estimation’ sessions.
The Fibonacci sequence is one popular scoring scale for estimating agile story points. This sounds like ‘an urban myth’ to me. Watch the presentation or read the book in the resourse section, to see why this is a joke.
Here is a part of the conversation when the scrum master asked me to provide estimations for test automation efforts.
“Z (my nickname), what is the story point of test automation for the user story #123”.
“2”, I replied.
“OK” (noted in JIRA)
“Z, your estimate for story #125?”
“2”, I replied.
My estimate is always “2” (or whatever numbering scheme they are comfortable with). Of course, very soon, understandably, the scrum master was not happy.
“Z, be serious, the test automation estimate for #126?”
“2, my answer will always be 2”, I replied. “Test automation is different from manual testing. Over 95% of my time as a test automation engineer will be spent on regression testing/maintenance, i.e., not on this sprint. So the estimation has absolutely no meaning to me”.
Then I explained with charts (showed the above). Like most of the audience who attended my training and presentation, the team members agreed mostly and uncomfortably. It is like some things in our life, once it is pointed out, it is painfully obvious. For the estimation session, the scrum master accepted 2 as the story points for test automation (for each user story).
Some might wonder why so many did not figure it out before? The answer is simple: most agile coaches/scrum masters/tech leads have never worked on a real Agile project. They knew test automation were an essential part of Agile, but had never experienced a real test automation process that overcame the hard-to-maintain phase. Commonly, test automation was for a showoff at the start of a project, then given up with all sorts of execuses. Executing all automated end-to-end tests as regression testing multiple times a day is new concept to them. One of the colleagues told me: “running automated regression testing daily is how test automation shall be done, why haven’t I thought about this on other projects?
This is by no means belittling my colleagues (we got along quite well), this is the failure of IT education and training. I was forunate enough to receive proper training from a world-classs hands-on agile technical coach in 2005, after working as Senior Java Programmer (Contractor) for 8 years. That was the turning point of my IT career.
Not surprisingly, I was told that I was no longer required to attend the “story estimate sessions.” I was happy that I had more time for testing. The “scrum master” would be happy too, without my presence, he could continue doing his version of ‘Agile Planning’.
I would suggest others do it more diplomatically, my approach was too direct.
So, how does the management know which user stories are hard to automate?
The short answer is “No, there is no need”. For automating different web apps, for a competent test automation engineer, there is not much difference. The reason: automation scripts interact with web applications (driving the browser) the same way. Rather, the capability of test automation frameworks and engineers is the main factor. For examples,
- The test automation framework (like Cypress in the early versions) did not support iFrame.
- The test automation framework (like Cypress) does not support two windows. (see Why Cypress Sucks for Real Test Automation? (Part 2: Limitations)
- The test automation engineer never had the experience of developing/maintaining over 50 tests. (Level 1 of AgileWay Continuous Testing Grading)
- The execution of automated tests not reliable. (see Working Automated Test ≠ Good Reliable Test)
- The test scripts are difficult to read and maintain. (see Maintainable Automated Test Design)
- Unable to handle AJAX or dynamic content efficiently and reliably. (see Test AJAX Properly and Efficiently with Selenium WebDriver, and Avoid ‘Automated Waiting’)
- Flaky execution in CI servers such as Jenkins and TeamCity. (See my upcoming My Continuous Testing Journey article)
I think readers should get the picture. Any of these above issues, if not addressed, will fail your test automation. Comparing to these, developing a new automated test is a piece of cake. There is really no need to estimate the efforts (for automated test creation).
I usually implement a couple of core client’s end-to-end regression tests on my first day of consulting. What impressed my clients most was not my efficiency, is that their manual testers actually understood the test scripts and worked on it, yes, on the first day.
How about story estimation for coding? As a matter of fact, it is a waste of time as well. I probably will write another article sharing my experience in developing my own apps, without story points. In the meantime, please check out the following resources. Allen Holub's keynote “#NoEstimates” presentation is great, its key takeaways are:
- “Estimation is waste.”
- “Estimation is destructive.”
- “Counting stories is enough for projections.”
- “Build the most important thing first.”
One of my dauther’s upcoming Uni courses is on Agile/Scrum. I forwarded the “#NoEstimates” YouTube link to her, and wrote in the email: “to see the quality of a lecturer who, with real agile experience, taught at University of California, Berkerly”.
- “#NoEstimates” keynote presentation by Allen Holub
One of the best Agile presentations I watched in recent years.
- Book: “NoEstimates: How To Measure Project Progress Without Estimating”
“I was in South Africa, at Agile Africa, and somebody came up to me and said “Well, we want to do software development, but we just can’t stand all this ceremony and Agile stuff. We just want to write some programs.” And tears came into my eyes…like…how can it be that we who set out to refocus development on essentials and get rid of stuff that didn’t matter, how can it be that we’re right back where we were 20 years ago? Like how can it be that ‘this is too much ceremony’? … No, this is wrong. I don’t know what to do about it. ”
– Kent Beck, Father of Agile, in an inverview (2019–10–15)