AgileWay Continuous Testing Grading

A simple grading system to rank your Agile/DevOps instantly!

Zhimin Zhan

Feb 1·5 min read

(This article is an excerpt of my book “Practical Continuous Testing”)

DEFINITIONS

• AgileWay

Means “do things in an agile way”, also the company I founded in 2006 when I discovered the revolution way (daily releases enabled by automated end-2-end testing) to do software development.

• Continuous Testing (CT)

CT, simply speaking, is to run all automated end-2-end functional tests (at user-story level) as regression testing multiple times a day, to help software teams deliver high-quality software to production frequently.

• Grading

Like the belt system in Taekwondo, to determine the level of competence and pursue advancements. Using a grading system in IT is not new, such as Google’s internal Test Certified Program (6 levels, see “How Google Tests Software” book, page 55), which weighs more on the technical level such as test coverage (and is a more complex rating system compared to mine).

The purposes of AgileWay Continuous Testing Grading:

1. help identify fake Agile/DevOps practices and personnel, instantly
2. set directions for advancement to real Agile/DevOps

FACT: most so-called “Agile” projects are fake. I will borrow some (absolute) authorities here:

• “Agile now means, we do half of Scrum poorly and use Jira” — Andy Hunt (Co-author of Agile Manifesto)
• “Agile is Dead (Long Live Agility)” — Dave Thomas (co-author of Agile Manifesto)

I was in South Africa, at Agile Africa, and somebody came up to me and said “Well, we want to do software development, but we just can’t stand all this ceremony and Agile stuff. We just want to write some programs.” And tears came into my eyes…like…how can it be that we who set out to refocus development on essentials and get rid of stuff that didn’t matter, how can it be that we’re right back where we were 20 years ago? Like how can it be that “this is too much ceremony”? … No, this is wrong. I don’t know what to do about it.
— Kent Beck, Father of Agile, said in this interview (2019–10–15)

Clearly, from the above top agile experts in the world, ‘Agile’ is right, however, many people do it wrongly: trying to fool others for money under the slogan of Agile. In short, fake “agile coaches” and fake “Scrum masters” are ruining Agile. Now, DevOps is getting popular, some ‘creative’ people come up with “TestOps”, “GitOps”, …, etc. Sigh!

Test Automation is the cornerstone of Agile and DevOps. If a software team agrees on that, please focus on getting automated regression testing done properly to enable frequent releases, the stuff actually matters!

Once a good CT process is in place for just a few weeks, the team’s productivity and morale will get a significant boost (one project manager once told me 3X based on his measurements). CT makes frequent releases possible, which would change the whole process. People will work better if they see their work being used and appreciated. Sadly, for many so-called “agile” projects, the team spent more time on JIRA/Confluence than their own product!

CT is the foundation of both Agile and DevOps. In my opinion, this is the quickest and easiest way to determine a bad/fake Agile or DevOps project instantly.

Here it is, AgileWay Continuous Testing Grading. Six levels in total: 0–6. The assessment criteria is simple: the number of automated end-2end functional (user-story level) tests, working ones of course.

Objective

To be able to assess the level of CT in a software project, instantly.

Assumptions

• Each automated functional test is at the user-story scenario level
• All tests are executed for each CT run, not splitting into sub-suites
• No human intervention at all during the course of test execution
• The total execution time of the regression suite is under 1 hour
• When software changes, automated regression testing is performed

If any of the above is not met, Level 0.

My estimated achievement rates of each level

For some like to use a term to describe the IT professionals at these levels:

• Level 0: Layman
• Level 1: Novice
• Level 2: Advanced Beginner
• Level 3: Professional
• Level 4: Expert
• Level 5: World Class
• Level 6: Legendary

For most, Level 4: Expert is the highest that most can realistically achieve. Beyond that, you need support of a massive infrastructure.

There is no scientific base for this, just like the Taekwondo’s belt system, purely based on my experience, observation, and intuition (according to Andy Hunt’s Pragmatic Thinking and Learning book: the experts rely on intuition), for over 15 years. Discard it if you don’t like it.

I understand that some will disagree with my estimates, shown above, especially on the achievement rates. I suggest you get a pencil and fill the table below.

My guess would be, like most of the people who have done these exercises in my training (or mentoring) sessions, Level 0 for all.

My Score

I know your next question to me will be: “Which level are you in then?” My answer is Level 4. The below are the UI test counts (as of 2020–12–15) for my three apps, which I push new builds to the production as long as the regression tests pass in our CT server (BuildWise).

(For experienced CT engineers, the total number of test executions might seem low. This is because I develop and maintain these 3 apps along with several others mostly in my spare time)

I helped implement CT for a number of clients, which I won’t share here. What I can tell you is that a large percentage of the projects reached ~Level 3.