My Test Automation Journey

Java/C# Programming ⇒ Test Automation + Continuous Testing; Now I do coding, testing, and CT daily for my own apps and coach test automation and CT for clients.

This article is a reflection of my 15-year journey in test automation since 2005. Prior to that, I had worked mostly as a Java Programmer (contractor) for 8 years.

  • (2005–2006) JWebUnit (HTMLUnit) using Intellij IDEA
  • (2006–2011) Watir with RSpec
  • (2011–2012) RWebSpec on top of Watir and Selenium WebDriver
  • (2011–2013) Load Testing with LoadWise

In 2012, I switched to work fulltime in test automation and Continuous Testing. My love for programming was not changed, just do it differently: coding for my own products in my spare time. Thanks to my CT process, my productivy in software development is an order of magnitude better. Please read “Reflections of Software I Created over the Last 14 Years in My Spare Time”.

  • (2012-present) raw Selenium WebDriver with RSpec
  • (2012-present) API/WebServices Testing with Ruby
  • (2019-present) Appium with WinAppDriver for testing Win desktop apps
  • (2020-present) Performance/Load testing using browser-based Selenium test to run in BuildWise CT server

1. JWebUnit on top of HTMLUnit (2005–2006)

At that time, though most IT people heard of automated functional testing, it was considered a luxury as the testing tools such as Mercury Interactive’s QTP (yes, later acquired by HP) were super expensive.

In 2005, I was extremely fortunate to have a chance to pair program with a world-class programmer (Martin Fowler often mentioned him) for 6 weeks. He showed me TDD (which I was aware of) and writing automated functional tests in Java, which was a new concept to me. I liked it instantly.

The framework we used was JWebUnit, a syntax layer on top of HtmlUnit.

Realization:

  1. Automated functional testing greatly enhances programmers’ productivity
  2. Automated functional testing increases job satisfaction and is fun
  3. Automated functional testing helps software design skills because I need to think more from different perspectives

Sample test (JWebUnit):

    @Test
public void testLogin() {
beginAt("/home");
clickLink("login");
assertTitleEquals("Login");
setTextField("username", "test");
setTextField("password", "test123");
submit();
assertTitleEquals("Welcome, test!");
}

2. Watir (2006–2011)

Once I ran a JWebUnit test in front of a manual tester. I pointed to the green tick in IntelliJ IDEA and said: “See, it passed”. This poor tester was confused, “How?”. I scrolled down and showed him the test output. He was half-convinced, “I want to see the test execution”. Hence, I made one of the biggest realizations of my IT career: “Seeing is believing applies to automated test execution”.

Fortunately, I found Watir and did exactly what I needed to do. Furthermore, after forcing myself to learn Ruby, I fell in love with this beautiful language quickly.

After selecting Watir as the automation framework, I naturally looked for a JUnit equivalent in Ruby as a veteran Java programmer, Test::Unit . It worked. Shortly after, I found RSpec, a BDD framework with more human-readable syntax, while still in 100% Ruby (this is very different from Gherkin frameworks). I liked RSpec. In fact, I have been using RSpec mostly ever since.

Realization:

  1. Automated test execution needs to be visible
  2. Automated test scripts shall be in a scripting language
  3. Need to think outside of the programmer’s perspective
  4. Java IDE such as Intellij IDEA is too complex for non-programmers to use
  5. Ruby is a great scripting language for writing automated tests.
  6. Need to run all tests in CI server with auto-retry features (will be covered in ‘My Continuous Testing Journey’)

Sample test (Watir):

it "Login OK" do
browser.goto "https://travel.agileway.net"
browser.text_field(name: "username").set(user_name)
browser.text_field(name: "password").set(password)
browser.button(value: "Sign in").click
try_for(3) { expect(browser.text).to include("Welcome agileway") }
end

Tool:

  • I created iTest2, a testing IDE that supports developing/debugging Watir tests. One of its unique features is “Functional Test Refactoring”.

Other testing tools (tried/reviewed):

  • IBM Rational Functional Tester, QTP (neither is good)
  • Selenium v1 (good support with Firefox, but with limitations; JavaScript is not a good language for writing automated tests)

Publications:

Acknowledgment:

Speaker at software testing conferences:

  • ANZTB 2010, Melbourne
  • Tianjin Software Testing Conference (TIST) 2010, China
  • Tianjin Software Testing Conference (TIST) 2011, China

Participated:

  • CITON (Continuous Integration and Testing Conference) 2009, Brisbane

3. RWebSpec on Watir and Selenium WebDriver (2011–2012)

Selenium WebDriver, a.k.a. Selenium 2, was released in 2011. This was big news for me. While I have been using Watir testing IE6 fine at that time, I would like to use Mozilla Firefox which was my preferred browser for its speed. With Selenium 2, finally, I could run automated tests in Firefox.

Selenium 2 comes with Firefox support, and there was no separate driver required then. Selenium with IEDriver was unreliable though. Also, the idea of cross-browser testing became popular. I created RWebSpec which will use Watir for IE and Selenium 2 for Firefox, in the same test script.

Sample test (RWebSpec):

before(:all) do    
open_browser :base_url => "http://testwisely.com/demo",
:browser => :firefox
end
it "Login" do
page_title.should == "AgileTravel"
enter_text("username", "agileway")
enter_text("password", "testwise")
click_button("Sign in")
expect(page_text).to include("Welcome agileway")
end

Other testing tools tried/reviewed:

  • Selenium WebDriver in Java, using Intellij IDEA

Publications:

4. Load Testing with LoadWise (2011–2013)

I have used JMeter for Load Testing before. It worked but was quite complicated. Therefore, I created my own performance/load testing tool: LoadWise. It did attract some commercial interests.

However, LoadWise, using the traditional HTTP-request simulations approach, does not support AJAX and more web apps were adopting AJAX. I deprecated LoadWise in 2013.

5. Selenium WebDriver (2012–Present)

With Selenium WebDriver became more reliable with IE (and Chrome dominated the browser market), I deprecated RWebSpec to use raw Selenium WebDriver + RSpec. I was quite pleased after converting all existing RWebSpec tests to raw Selenium. The scripts are actually more intuitive.

Over the last 8 years, I used raw Selenium + RSpec to test my own apps and help others.

Sample test (Selenium WebDriver + RSpec):

it "[1] Can sign in OK" do
driver.get("https://travel.agileway.net")
driver.find_element(:id, "username").send_keys("agileway")
driver.find_element(:id, "password").send_keys("testwise")
driver.find_element(:name, "commit").click
try_for(3) { expect(driver.page_source).to include("Welcome")}
driver.find_element(:link_text, "Sign off").click
end

Tool:

  • Renamed iTest2 to TestWise, and added support for Selenium WebDriver

Realization

  • Test syntax framework shall be in 100% scripting language like RSpec, while Cucumber is not.
  • Presenting at international software testing conferences, in my opinion, is not effective in terms of influence. I met a number of attendants who liked my presentation, but few of them dared to try test automation at work (lack of confidence). So I decided to spend more time on writing books that come with good exercises, creating video demos, etc. rather than speaking at conferences.
  • I worked on many projects as a contractor/consultant. I didn’t care about job titles, just wanted to work on different projects for testing challenges. After a few years, I was able to answer and solve other people’s testing problems quickly, usually under a minute.
  • While test automation was being adopted by more projects, every single project I visited failed to adopt the automation properly. Some large companies failed a few attempts.
  • With my increased efficiency on test automation, my productivity as a programmer also grows significantly. From 2012, I created ClinicWise, SiteWise, TestWise v4, SupportWise, BuildWise, and WhenWise apps, in my spare time.

Other testing tools (tried/reviewed)

  • UFT, Coded UI Test, Ranorex, Sahi Pro, Cypress. (none of them was good)

Speaker at software testing conferences

Publications:

6. API/WebServices Testing with Ruby

While I focused mostly on web test automation, quite often, I was required to perform API/WebServices testing. As a veteran programmer, writing API test scripts is quite easy for me. However, after I worked on web test automation with many manual testers, I learned to look beyond the correct API tests: easy-to-understand (for non-programmers) and easy-to-maintain.

A common mistake with API Testing is to use tools such as SoapUI/Postman. Every project I knew of that uses these kinds of tools failed for the reasons below:

  • Not flexible (fact: programming provides the ultimate flexibility)
  • Hard to maintain
  • Inefficient. I know some might find it hard to believe that using a good scripting language such as Ruby can be far more efficient than GUI tools.
  • Difficult (or impossible) to run the tests in a CT server. Many projects did not even realize how to manage executions of a large suite. Therefore, tests went out of date quickly. As time went on, it was getting harder and harder to maintain. Eventually, there was only one outcome: the tools were dumped.

My approach with API (Soap/Rest/Microservices/GraphQL) has always been the same (for over 12 years): raw Ruby with RSpec and execute them in BuildWise.

At one project, I completed an assigned 2-month’s API testing (SoapUI) workload under one day (including intital start time), using raw Ruby.

With quick training, most manual testers can work on the scripts following my examples. For more, please read my book (listed below).

Other testing tools (tried/reviewed)

  • SoapUI, ReadyAPI, Postman (none of them was good)

Publications:

7. Appium with WinAppDriver for testing win apps (2019-present)

I created a small framework RFormSpec on top of AutoIT3 for automating windows native apps. AutoIT3 has some limitations, it worked well for some scenarios. I developed a dozen of RFormSpec tests for TestWise, my desktop app. However, it wasn’t easy.

In 2019, I saw a recommendation from Microsoft: “Selenium for web apps, Appium for desktop apps”.

After quick work on proof of concept, I liked it. Therefore, I switched the automated tests of TestWise (my desktop app) to Appium + Desktop. To see a test execution of Appium+WinAppDriver, check out this article “Demo of Automated Testing for Windows Desktop App: TestWise tests TestWise”.

As of 2021–03–26, the regression suite of TestWise has 276 automated tests running in the BuildWise CT server. This reaches Level 3 of AgileWay Continuous Testing Grading.

This CT process enables me to release TestWise daily if required.

Sample test (Appium + RSpec):

it "Plus two numbers in Calculator" do 
desired_caps = {
caps: {
platformName: 'Windows',
deviceName: ENV["DEVICE_NAME"] || 'MacBook',
app: "Microsoft.WindowsCalculator_8wekyb3d8bbwe!App"
}
}
driver = Appium::Driver.new(desired_caps, true).start_driver
driver.find_element(:name, "One").click
driver.find_element(:name, "Plus").click
driver.find_element(:name, "Three").click
driver.find_element(:name, "Equals").click
result = driver.find_element(:accessibility_id, "CalculatorResults").text
expect(driver).to eq("Display is 4")
driver.quit
end

2020–03–25, TestWise 6 is released. TestWise 6 is a total rewrite (the work started in 2018–07) using C++ language, and runs on Windows, macOS and Linux.
If you are interested, see TestWise in action in this article: “Step by Step showing how to learn to write raw Selenium WebDriver test scripts in minutes”.

8. Continuous Performance/Load testing using browser-based Selenium testa to run in BuildWise CT server (2020-present)

Load testing is moving from Protocol-Simulation to Browser-based as the modern web apps are dynamic and used AJAX. Since last year, I have combined my past experience of developing LoadWise with Web Test Automation and Continuous Testing and come up with a new way to conduct load testing.

  1. Engineer develops a load test script as a functional UI test first, in raw Selenium WebDriver +RSpec. The engineer can run the testing tool such as TestWise, and view (or in load testing perspective, preview) test execution to ensure correctness.
  2. Add benchmark (timings) of operations
  3. Make the test script load-testing aware of virtual users
  4. Run a set of load test scripts in the BuildWise CT server which will distribute tests to run them in multiple build agents in parallel.
  5. The test reports (and history) are viewable on BuildWise’s web interface.

Below is a report of the ‘User Login’ load test with 5 virtual users on BuildWise.

Currently, I am working on a new book: “Practical Performance and Load Testing”. I usually make the book available (on Leanpub) after the whole book is completed. If some readers want to access it now, please contact me.

Award-winning software developer, test automation & CT coach, author, and speaker. Help teams succeed with Agile/DevOps by implementing real Continuous Testing.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store