Why Cypress Sucks for Real Test Automation?
Note: The sequel to this article “Why Cypress Sucks for Real Test Automation? (Part 2: Limitations)” is now available.
Cypress, a proprietary JavaScript end-to-end testing tool, seems to me that it is an improved Selenium v1 (also based on JavaScript), which was a big hype in 2005 (until it was merged with WebDriver in 2009). I believe many testers had not used Selenium v1 before, what I could tell you is that being replaced by Selenium WebDriver was a good thing.
As a test automation & continuous testing coach, I visited many projects. I have never seen a single success using Cypress, not even Level 2(of AgileWay Continuous Testing grading in ‘Practical Continuous Testing’ book), that is:
- A regression suite of 50+ user-story level automated UI tests
- The test suite runs in a CI (or CT) server multiple times a day as regression testing. (otherwise, there was no point doing it)
- while coping with frequent application changes (reality, right?)
- At least one green build (passing all tests, or with explainable causes) a day.
- On a green build, the team has high degrees of confidence to release to the production. (that’s the ultimate purpose of doing test automation, isn’t it?)
How do you score? As you can imagine, Level 3 (200+ tests) will be much much harder. All successful test automation implementations I have seen (up to Level 4, 500+ automated UI tests run in CT daily) were all using Selenium WebDriver/Watir + RSpec. I heard of Facebook's test suite is huge, if you are interested in large-scale test automation, check out this F8 presentation by a Facebook engineer. By the way, Facebook uses Selenium WebDriver.
“For all of our end-to-end tests at Facebook we use WebDriver, WebDriver is an open-source JSON wired protocol, I encourage you all check it out if you haven’t already. ”- Katie Coons, software engineer at Facebook, “Continuous Integration at Facebook” presentation at F8 2015.
In test automation history, there were full of false hypes. Just name a few:
- HP QTP -> Micro Focus UFT -> Micro Focus LeanFT (to support Selenium WebDriver). QTP/UFT has been a long history known as ‘the leader’ in test automation. Since 2010, I started questioning QTP (at my conference talks and consulting). The simple reason was, every QTP project I saw was a complete failure. Micro Focus now is a “Selenium level sponsor” of Selenium WebDriver.
- Phantom JS (for headless). Between 2010–2015, Phantom JS was talked highly by test architects. I tried, it puzzled me how could that possibly work? I wrote down my thoughts in my book “Selenium WebDriver Recipes in Ruby.” In 2017, Phantom JS was deprecated, according to Vitaly Slobodin, the main maintainer: “(Headless) Chrome is faster and more stable than PhantomJS. And it doesn’t eat memory like crazy. I don’t see any future in developing PhantomJS.” It turned out that those ‘we tested with Phantom JS’ were just lies. (if the test automation framework not reliable, ….)
Why Selenium WebDriver is the automation framework for testing web apps? I guess the best answer to that is this announcement from Microsoft in December 2018.
Microsoft acknowledged its tool (Coded UI Test) was no good, (by the way, I met a few hyped about Coded UI Test as well), comparing to Selenium (or Appium for desktop app testing). Microsoft finally deprecated CodedUI and recommended Selenium instead. In fact, some Microsoft engineers have started recommending Selenium (over its own) since 2016. If you are interested in the reasons behind it, read “Why we are deprecating Coded UI Test” part of this article by a Microsoft Engineer.
I assume that most will agree on that Selenium Works. It is a standard, more importantly, the only framework supported by all major browser vendors. Some may argue that it is difficult to learn, Wrong! Selenium WebDriver is the easiest to learn, far easier than others. I mean: raw Selenium WebDriver, not with some added custom layer (often marketed as ‘own test framework’, which always ended badly). I mentored many manual testers. Most of them could write real Selenium tests for work on the next day. Of course, mastering one skill takes time, but learning can be quick. (If students have the confidence to practice and grow, the teaching/learning is successful.)
Let me repeat, under proper guidance, you can learn Test Automation with Selenium WebDriver within one day. My daughter started writing real and useful Selenium tests at the age of 12, so could your child, see “Step by Step showing how to learn to write raw Selenium WebDriver test scripts in minutes.” The reason: Selenium WebDriver syntax follows a pattern: “find an element and drive it.” This post is about why I think Cypress sucks, not how great Selenium WebDriver is. For more, you may refer to my two books, “Practical Web Test Automation” and “Selenium WebDriver Recipes in Ruby”, both were listed in the best selenium books of all time by Book Authority.
I understand why cypress is appealing to some who do not understand real test automation and scripting. I agree, Cypress looks good for demonstration purposes, but not for real automation. The so-called ‘nice features’ of cypress are either actually not useful when addressing the real test automation challenges, or there is a far better way with Selenium scripting. Let me illustrate:
- Nice debugging features. Cypress and Puppeteer are built based on Chrome DevTools Protocol (CDP), a protocol to enable a debugger inside Chromium-based browsers. CDP has some issues and limitations. If you are interested, read this article by Simon Stewart, creator of Selenium WebDriver. Anyway, Selenium 4 will support CDP, and most likely better than what you can do with Cypress. Even with Selenium 3 (without CDP), testing IDEs such as TestWise offer very good debugging capabilities (see below or this article for more).
- Record. The recording is actually bad for test automation, and above all, it is not necessary. Strangely, the recording has been criticized for over 20 years, and it keeps coming back in another form. I once developed a recorder (TestWise recorder) for fun, so I do have a good understanding of it. The harm of using a recorder far outweighs its benefits. My suggestion: don’t use the recording. As a matter of fact, with a good testing IDE and proper training, recorders are hardly needed. (I developed and maintained thousands of automated tests for a number of projects over 15 years, I only remembered only one case that a recorder was helpful). Cypress, in my opinion, is somewhere between traditional record-n-playback and (better) raw scripting. While many people choosing to use Cypress Studio or cypress recorder is not Cypress’s fault (same goes Selenium IDE for Selenium WebDriver), but the nature of how Cypress works is encouraging testers to choose a convenient locator (rather the most reliable one), this leads to maintenance problems.
“Record/Playback scripts are notoriously costly from a maintenance perspective.” — ‘Agile Testing’ p315, 2009
- Wait for elements. This can be easily implemented in scripting. Testers who attended my selenium training knows this:
tfTab. Done. By raw scripting, you have full flexibility to control the wait, much better than cypress. See the article ‘Test AJAX Properly and Efficiently with Selenium WebDriver’ (with video). - Productive, which is false. Every Cypress tester I met was very poor on productivity. Selenium WebDriver, on the other hand, with the right tool such as TestWise, can be super-productive. I repeatedly was 10x or more productive than a Cypress tester when working on the same set of tests. Check out another article: “Step by Step Showing Why Selenium WebDriver is the Easiest-to-Learn Web Test Automation Framework”, a video is also included.
Executing selected test steps against the current browser is possible with Selenium. As you can see in the above screenshot, the user may freely add/change test steps to run against an attached browser. The animated GIF shows the “run steps against the current browser” action.
Disclaimer: I am NOT trying to promote my tool TestWise here. The test script does attach the browser using the
--remote-debugging-portoption with ChromeDriver. You can view the test helper script here. In other words, what I show is 100% free and open. TestWise is just one tool utilize this, no dependnecy there. You may add this support to your tool, for example, an extension to Visual Studio Code. Freedom is Good!
- Easy to learn, which is false. Cypress is much harder to learn, compared to Selenium WebDriver. If Selenium WebDriver is taught well, a 12-year old (verified by my daughter) can learn basic Selenium to be useful in minutes. Check out this article.
- JavaScript, which is wrong. Node.js, thanks to the v8 engine, is very fast. For UI testing, the raw speed of a language matters a little. However, the JavaScript language is not a good testing language. JavaScript programmers would strongly disagree, isn’t this a problem? We are talking about functional tests here. The audiences of functional tests are mainly business analysts/customers/manual testers. Ruby is a far better choice, and it has long been proven. (Watir = Web Application Testing in Ruby, please note, the authors put the language in the framework name).
I like scripting Selenium WebDriver tests in Ruby. It is efficient, creative, and fun! I dislike the limitations of proprietary test frameworks. For people who are interested to know some limitations of Cypress, read this article by Alister Scott.
In the true spirit of test automation, whether an automation framework good or bad, is judged by the result, the cold hard binary result: 0 or 1. It is good to see new test frameworks or tools, competition is good. In that sense, I welcome and actually tried Cypress, PhantomJS, Selenium WebDriver in all 5 languages, and others. Once I was asked to score available test frameworks, which I didn’t expect, as Selenium WebDriver shall be a no-brainer. Later I came up with a set of criteria (such as reliability, flexibility, maintainability, cost, ease of CT integration, …, etc. like judging for the car of the year) and gave my score for each framework.
Selenium -WebDriver: 99 (out of 100), Cypress — 58, Ranorex — 29, UFT — 42
Update (2021–01–29): Some Cypress testers claim Cypress is fast. I tried a simple example, did not notice its advantage comparing to Selenium. But I didn’t want to waste more time on Cypress, so I moved on (sorry, I heard so many lies from so-called ‘Cypress Automated Testers’). Giovanni Rago did a thorough test comparing the performance of tools under different scenarios: Selenium vs Playwright vs Puppeteer speed comparison. This test result shows: not much difference in terms of speed for 4 frameworks, however, Cypress was the slowest on most categories.
Maybe one day, cypress will be more open, actually works, and with fewer limitations. So far, I was not convinced at all. If your cypress implementation enables your project to release the production daily, then you are successful, ignore anyone who tells you otherwise. If not, try Selenium WebDrver with RSpec, and embrace freedom, productivity, and fun!
Lastly, I share an execution of 608 selenium tests in the Continuous Testing server yesterday.
And of course, production deployment after the green build. The test suite has over 30,000 test steps (every step matches a user operation). To get a green build, each of every 30366 test step must pass.
+--------+-------+---------+---------+-------+
| TEST | LINES | SUITES | CASES | LOC |
| | 27508 | 260 | 612 | 21308 |
+--------+-------+---------+---------+-------+
| PAGE | LINES | CLASSES | METHODS | LOC |
| | 11697 | 260 | 2065 | 8707 |
+--------+-------+---------+---------+-------+
| HELPER | LINES | COUNT | METHODS | LOC |
| | 459 | 1 | 45 | 351 |
+--------+-------+---------+---------+-------+
| TOTAL | 39664 | | | 30366 |
+--------+-------+---------+---------+-------+For this app alone, I have been conducting Continuous Testing for nearly 7 years ( development, testing, and customer support all in spare time), over 600,000 test executions.
