Upgrade WhenWise to Rails 8. Following 3 Rounds of Full Automated E2E (UI) Regression Testing, Deploy to Production (All in ~3 hours)

Thanks to Comprehensive E2E Automation Regression Testing.

5 min readNov 12, 2024

This article was originally published on SubStack, which is now my main blogging platform. For readers interested in learning E2E test automation, I invite you to subscribe to my Substack newsletter. You can start with the free plan and explore valuable insights and content.

Four days ago, Rails 8 was released.

https://rubyonrails.org/category/releases

I didn’t upgrade WhenWise immediately because I’ve been exploring the new M4 Mac Mini lately and used the opportunity to optimize my Continuous Testing lab.

At 7 AM on November 12, 2024, I started the upgrade to Rails 8 (from version 7). Experienced software engineers understand that a major part of such framework upgrades is E2E regression testing.

Here, I mean real End-to-End (UI) Testing. Customers can see and understand it.

How did I accomplish the entire upgrade in just 3 hours? Thanks to end-to-end (UI) test automation and continuous testing practices I’ve applied to all my web applications since 2012.

1. Upgrade

Refer to the Rails 8 upgrade guide and carry out the following steps:

Update Gemfile (library), e.g. `gem “rails”, “~> 8.0.0`

bundle update rails

Update the app

rails app:update

Then follow the prompt to update various configuration files. Some customized settings (e.g. STMP, Fake Google/Facebook authentication) needed to be kept and added back to the new default Rails configurations.

Also, I bumped the WhenWise version to 2.0.0.

2. Quick Local Check

I launched a local server instance (‘DEV’) with the updated code and ran into a minor issue.

This can be resolved by changing one environment configuration entry (in all my 10+ environments).

config.action_controller.raise_on_missing_callback_actions = false

After restarting the DEV server, I could sign in and perform a few simple operations without problems.

It seemed going to be a smooth upgrade.

I then executed several key workflow automated tests in TestWise IDE.

One end-to-end test passed in TestWise.

All three of them passed.

At 07:23, I committed the changes locally and pushed them to the parent repository.

3. Regression Testing

I ran the deployment script (a simple Ruby script) to deploy the latest build to my 10 test server instances, all now operating on the new M4 Mac Mini.

For this regression testing, only 7 instances were utilized.

I triggered a run on the BuildWise Server, and the main work commenced (I am not just sitting and waiting)

4. Ready for Regression Testing

This time, I pushed the limits by using up to 7 build agents on the newly configured App Server. Experienced Test Automation Engineers understand:

Infrastructure changes can impact server performance, making UI tests more sensitive.
Conducting a major version upgrade (Rails 7 to 8).
Using three more agents than usual increased the load, especially since all 7 instances were on the same server machine.

Run 1: Start time 7:47 AM (took 38 minutes), with 5 test failures.

BuildWise’s auto-retry feature helped prevent multiple false alarms (18 in total). The initial run resulted in only 5 test failures, which was better than expected. During the CT run, I monitored the test executions across different build agents (I do not run tests in headless mode).

For every test failure (while the run is still undergoing on BuildWise), I reran the test again the DEV server on TestWise. Then, analyse and update test scripts (or even the app) accordingly. This is challenging work under significant pressure, which many senior test automation engineers find difficult to manage. (Tip: seek out a mentor to help guide your growth in this area 😉).

The application itself performed well, and the failure analysis (based on error traces and screenshots from BuildWise) indicated timing-related issues. The solution is “stabilizing tests”.

Run 2: Start time 8:34 AM (took 36 minutes), with 3 test failures.

The same process.

Run 3: Start time 9:23 AM (took 29 minutes), SUCCESS.

This is the first time, from memory, that the total regression testing duration is under 30 minutes, after the number of automated E2E tests has surpassed 500.

Great, it is time to push it to production.