My Core Test Automation Stack Largely Unchanged for 17 years
If our testing target (Web and Mobile) has not changed for decades, why should QA Engineers change (for the sake of change)?
I embarked on my test automation journey in 2005; as of 2024, it has been nearly two decades. I don’t mean to boast, but I’ve accomplished quite a bit in the realm of end-to-end test automation:
- Authored 12 books on E2E Test Automation and Continuous Testing
- Invited to speak at international software testing conferences, including the premier StarWest.
- Implemented successful E2E test automation (web, API, desktop) in a number of software projects. Rescued some too.
- TestWise IDE I created received acknowledgements worldwide.
- My BuildWise CT Server won the 2018 Ruby International Award in Japan.
- I developed (and have been maintaining) several highly acclaimed apps in my spare time, thanks to my E2E Test Automation and Continuous Testing process.
- Regarded as one of the world’s top bloggers on software testing.
Table of Contents:
· 1. Pre-WebDriver (2005–2010)
∘ Tech. Stack (Pre-WebDriver):
· 2. Web Test Automation Stack
∘ Tech. Stack:
∘ Example: Selenium Suite for WhenWise web app
∘ Example: Selenium Suite for ClinicWise web app
· 3. Desktop Test Automation Stack
∘ Tech. Stack:
∘ Example: Appium Suite for TestWise Desktop App
· 4. API Testing
∘ Tech. Stack:
∘ Example: API Testing Ruby Recipes
· 4. Performance Testing
∘ Tech. Stack:
∘ Example: WhenWise Performance Testing
· 5. Load Testing
∘ Tech. Stack:
∘ Example: WhenWise Load Testing
· 6. Mobile Test Automation
∘ Tech Stack:
∘ Example: My Daughter’s book “Practical Mobile Test Automation with Appium”
· 7. I am more open-minded than most.
· Summary
1. Pre-WebDriver (2005–2010)
I started with JWebUnit (Java), then quickly switched to real E2E (in browser) testing with Watir.
Tech. Stack (Pre-WebDriver):
- Automation Framework: Watir
- Test Syntax Framework: RSpec
- Test Scripting Language: Ruby
- Testing IDE: TestWise
- Continuous Testing Server: Cruise Control (with cutomization)
2. Web Test Automation Stack
Tech. Stack:
- Automation Framework: raw Selenium WebDriver
- Test Syntax Framework: RSpec
- Test Scripting Language: Ruby
- Testing IDE: TestWise
- Continuous Testing Server: BuildWise
Example: Selenium Suite for WhenWise web app
368K test executions for ~6 years.
Example: Selenium Suite for ClinicWise web app
I have another even bigger suite for the ClinicWise app, with 611 selenium tests (and 618,327 test executions). Below is an execution history report for ~8 years.
For more, check out my articles:
3. Desktop Test Automation Stack
My Desktop Test Automation started in 2009, tried autoit3 with limited success. I changed the tech stack after seeing Microsoft deprecated its own Coded UI Tests and recommended using Appium + WinAppDriver for Desktop app automation in 2018.
Tech. Stack:
- Automation Framework: raw Appium + WinAppDriver
- Test Syntax Framework: RSpec
- Test Scripting Language: Ruby
- Testing IDE: TestWise
- Continuous Testing Server: BuildWise
Example: Appium Suite for TestWise Desktop App
73K+ test executions over 4.5 years.
4. API Testing
While I haven’t explicitly worked with an official job title, API Test Automation Engineer, I have used API testing at most jobs in the form of either API testing or using APIs to assist Web Test Automation.
Tech. Stack:
- Automation Framework: N/A. Just use Ruby to invoke HTTP
- Test Syntax Framework: RSpec
- Test Scripting Language: Ruby
- Testing IDE: TestWise
- Continuous Testing Server: BuildWise
Example: API Testing Ruby Recipes
For more on my API Testing approach:
- My Universal, High-Efficient and Free Approach to API Testing
- A Story of 100X API Testing I Implemented in a Matter of Days
- My daughter’s API testing series: SOAP Web Services, RESTful Services, GraphQL
4. Performance Testing
Traditionally, performance testing is conducted by a separate team towards the very end of software development. Using the BuildWise CT server, I implemented “Continuous Performance Testing”, which runs performance tests early and frequently.
Tech. Stack:
- Automation Framework: Ruby Mechanize or Selenium WebDriver
- Test Syntax Framework: RSpec
- Test Scripting Language: Ruby
- Testing IDE: TestWise
- Continuous Testing Server: BuildWise
Example: WhenWise Performance Testing
For more, check out:
5. Load Testing
Like Performance Testing, Load Testing is not my specialty. In 2016, while working as a contract test automation consultant (functional), I solved a key technical load testing challenge that bothered this large tech company’s performance testing team for a year. The solution is simple: Using 10 BuildWise Agents (in VMs, which this company has the infrastructure), and I completed the proof of concept within a couple of days, the task was accomplished the task within weeks.
Since then, I have added more load-testing features in BuildWise and used them with my clients and my own projects with great outcomes. In 2022, I published the book Practical Performance and Load Testing.
Tech. Stack:
- Automation Framework: Ruby Mechanize or Selenium WebDriver
- Test Syntax Framework: RSpec
- Test Scripting Language: Ruby
- Testing IDE: TestWise
- Continuous Testing Server: BuildWise with Multiple Agents
Example: WhenWise Load Testing
6. Mobile Test Automation
My primary End-to-end test automation is on testing web and desktop apps. I have tried some Appium to drive iOS apps and could put up an impressive demonstration or presentation. However, long-time readers know I don’t consider that real test automation.
It is not me not trying to do mobile test automation. The primary reason is that I don’t have mobile apps, i.e., no practical needs. I did attempt to join mobile app teams (as a contractor), twice. The first one turned out to be a simple wrapper of the web app; therefore, web testing worked fine. The other did not happen due to the impact of the COVID-19 pandemic.
Anyway, I believe my core tech stack also works for Mobile Test Automation, as my daughter is writing a mobile test automation book with this tech stack.
Tech Stack:
- Automation Framework: Appium + XCUIDriver/ExpressoDriver
- Test Syntax Framework: RSpec
- Test Scripting Language: Ruby
- Testing IDE: TestWise
- Continuous Testing Server: BuildWise
Example: My Daughter’s book “Practical Mobile Test Automation with Appium”
The use of Appium should not be controversial, as it’s the de-facto choice in the mobile automation space. Some readers might think: “With the above test automation types, you provided proof. But without proof of execution history for a sizeable suite, I have doubts”. If you have doubts, stay tuned for my daughter’s book.
7. I am more open-minded than most.
People have different opinions. I don’t expect everyone to agree and adopt my tech stacks. Frankly, I don’t care.
Being objective is a core attribute of a Software Tester. I am more open-minded than most test automation engineers. Again, I say this with proof.
Scripting language
I am probably more objective than most, as My Selenium Recipes Book series covers all of Selenium WebDriver’s five official programming languages.
Cucumber
Many readers know that I am against using Cucumber or any Genkin BDD framework in E2E Test Automation from my articles:
- Why Gherkin (Cucumber, SpecFlow,…) Always Failed with UI Test Automation?
- A Practical Advice on Rejecting Gherkin for Test Automation
Yet, I have done extensive work to add supporting Cucumber in my TestWise IDE.
My BuildWise CT server supports Cucumber from its first version.
Playwright
TestWise 7 added support for Playwright.
Summary
As you can see, my core tech stack for E2E test automation is unchanged. Moreover, there is a lot of synergy among different types of E2E testing (Web, Desktop, API, and Mobile): all scripting in Ruby using TestWise IDE, running regularly, in a continuous manner, in BuildWise.
Related reading:
- My eBooks:
- Practical Web Test Automation with Selenium WebDriver
- Practical Continuous Testing: make Agile/DevOps real
- Practical Performance and Load Testing - Benefits of Real E2E Test Automation and Continuous Testing series: Executives, Managers, Business Analysts, Developers, Testers and Customers.
- A Practical Guide to Self-Learning E2E Test Automation and Being Better Than the Industry Average in One Week, for $1 to $11.
- Why Ruby is the Best Scripting Language for End-to-End Test Automation?
- Programming Language Advice to My Daughter, Part 1: Ruby
- Why JavaScript Is Not a Suitable Language for Real Web Test Automation?
- Too Many Failed JavaScript Test Automation Frameworks!
