Why Cypress Sucks for Real Test Automation? (Part 2: Limitations)

Many limitations of Cypress makes it not suitable for serious test automation

Zhimin Zhan

Feb 15·8 min read

I have received some positive feedback on my “Why Cypress Sucks for Real Test Automation” (Part 1) article. A few readers wanted to clarify it a bit more. Yes, the article was more focused on why there is really no need to use Cypress, as there is a much better tool (in almost every way) Selenium WebDriver. In this follow-up article, I will focus on the limitations of Cypress.

1. Technical limitations of Cypress

For the specific technical limitations of Cypress, please read an excellent article “My Thoughts on Cypress.io” by Alister Scott on 2019–07–16. I will just highlight the comments section here:

“There are technical limitations to the way Cypress loads and runs the tests, which are covered in https://www.cypress.io/how-it-works Some of the limitations will be removed (iframes, shadow dom support) as we continue, and some limits will probably be there for many years.”
- Gleb Bahmutov, distinguished engineer at cypress

and the reply from Alister Scott:

I did hear this line “if it’s on the web, Cypress can test it” which I still have a problem with as my recent apps I have worked on have both used iFrames can Cypress can’t test them — including WordPress.com which is a super high volume web application.
- Alister Scott, Excellance Wrangler at Automattic (WordPress)

Gleb and Alister exchanged their opinion professionally. A top test automation engineer like Alister had problems testing iframes with Cypress, and the distinguished engineer at cypress confirmed it. For the wrong claim (“if it’s on the web, Cypress can test it”) of Cypress, it is from Cypress’s marketing team, don’t take it seriously. One sure thing is, even Cypress die-hard fan would agree: Cypress has technical limitations.

Frames have been in HTML for as long as I could remember. For > 50% of projects (yes, iframes are being less used in modern web sites, yet still…), I had to drive iframes with Selenium WebDriver, which was totally fine. About 2 years ago, a so-called ‘Cypress tester’ I met mentioned that he was unable to test frames or upload files with Cypress. I thought it was absurd! He simply lacked a basic understanding of test automation. That is, if unable to testing one operation on a web page, then it pretty much means the application cannot be tested. This is simply the nature of UI test automation.

To qualify for a web test automation framework, it shall not have serious technical limitations. This is common sense, isn’t it? After all, people will use it for verification. I never had ‘limited’ feeling with Selenium WebDriver.

I know some would argue: “I can do with cypress-iframe now (in 2020)”. Yes, it might be the case (frankly, I didn’t want to waste time to verify that after finding so many limitations with Cypress). Then, how about other limitations, such as not supporting multi-tabs, cannot visit two domains of different origin, only inside the browser, unable to drive two browsers, …, etc. For each of the above limitations, I had a case to use it in recent projects. With Selenium WebDriver, no problems at all.

Gleb Bahmutov, who is probably the man with the most Cypress knowledge on earth, acknowledged that “some limits will probably be there for many years”. Will your project have ‘many years’ to wait? Unless you want only a fancy demo, do not use it for real test automation.

2. Cypress is actually slower

In Cypress’ documentation, it claims:

This is incorrect, at least no longer correct. Cypress test execution speed may be OK initially (suitable for a demo) but will get slower. Cypress is based on Chrome DevTools Protocol (CDP) which is a protocol developed to enable a debugger inside Chromium-based browsers. According to this article by Simon Stewart, the creator of Selenium WebDriver.

Because of what it (CDP) was designed to do, it is a chatty protocol, and not a user-facing API that allows you to introspect into the browser. Worse, introduce a network hop between the test and the browser, this chattiness leads to slower tests as the network latency increases. — Simon Stewart, 2020–11–06

My personal experience confirmed this. According to this recent benchmark comparison by Giovanni Rago: Cypress was the slowest in most categories. For a very small test suite, Cypress is about the same or slightly slower than Selenium WebDriver. I expect Cypress will be considerably slower on a medium or large test suite. But test automation attempts with Cypress have probably failed long before reaching that size, as it is inefficient and difficult to maintain.

3. Cypress tests are hard to maintain

In test automation, the view of an inexperienced test automation engineer can be quite different from an expert. I have had extensive experience in maintaining a large scale of automated UI tests (referred to as an ‘expert’ on some software conferences’ brochures). I pioneered “Functional Test Refactoring” which was presented at STARWEST 2013, one of the longest-running and most respected conferences on software testing.

I will post a series of articles in the near future on test maintenance. Here I will just be brief with some key facts and questions:

• Fact: The efforts to maintain automated UI tests is much larger than test creation
• Fact: Real test automation means to keep your automated tests up with application changes that happen frequently
• Fact: The audience of the automated test scripts are the whole team, not just programmers
• Question: How do you score on “AgileWay CT Grading” (based on the test count, valid and frequently run ones)? My score is Level 3. For all Cypress projects that I knew of, Level 0. The №1 reason: ‘Cypress testers’ inability to maintain them.

4. Cypress can only use Node.JS, which is not suitable for test automation

JavaScript is good for web programming, not ideal for test automation. I will write another article on why we should avoid JavaScript in automated UI test scripts. FYI, I have been using JavaScript since the last century and had authored the book: “Selenium WebDriver Recipes in Node.js”.

Here I will just highlight three issues with Node.js for automated testing:

(1) Confusing JavaScript Promise

JS Promise is an async programming model, a complex concept. It can usually cause confusion. Above all, it is totally unnecessary for automated testing. A Node.JS programmer might say “I can use Promise without problems at all…”. Wrong! “It is not about you”. Automated functional test scripts shall be understandable to all the team members, including business analysts.

Here is a JS Promise example from cypress-examples by Gleb):

Try to find a business analyst to understand and use test scripts like the above.

(2) Unnecessary NPM package.json

The file package-lock.json is for a Node.JS project to manage dependencies. This makes sense for an app, but not much for automated tests. I have seen many people, myself included, often got this error when trying to run a Mocha test.

% mocha spec/01_login_spec.jsError: Cannot find module 'selenium-webdriver'

The library was installed, there was just no package-lock.json in the current directory. While the solution is simple: npm install, it is still quite troublesome.

I know some smart JS programmers might argue: ‘I have a solution’. Again, the audience of automated functional tests are for the whole team. The simpler and more intuitive, the better.

In Ruby, there is a similar concept for library dependency management: Gemfile.lock, but it is optional and won’t affect test execution as long as your dependent libraries are installed on the machine. Easy!

(3) Large node_modules folder

npm install will create a node_modules folder, a cache for the external modules. For a minimal Selenium with Mocha test, its node_modules folder contains 3759 files/folders, 23MB on disk.

A common mistake is that engineers checked the node_modules folder into a Git repository!

With Ruby, there is no such mess at all.

5. Cypress wrongly added “retry-ability”, which shall be a CT feature

Cypress introduced Test Retries in v5 in August 2020, which is another step in the wrong direction. I came up with the idea of auto-retries in 2007 and have been used it ever since, but in CT server, not the test framework. Please read this article: “Why Auto-Retry of Test Execution in a Test Framework is Wrong?”

6. Complex and Inconsistent Syntax

“Good syntax” is one of the main reasons that Cypress testers promote it. In my opinion, it is the opposite. Cypress syntax is inferior. To be fair, I will use the example in the Cypress Documation to compare against Selenium WebDriver.

Here is the Selenium + RSpec version:

I like Selenium’s version better, for the following reasons:

• One line for one user operation, which will be very helpful for debugging test failures
• It follows an intuitive pattern: “Find a control, then drive it”
• Selenium does not have built-in assertion syntax. It leaves it to the test syntax framework. Therefore, the user has maximum flexibility. By the way, RSpec deprecated should assertion syntax (still works) for expect instead, and I prefer expect .
• There is no ambiguity in Selenium. Cypress is syntactic ambiguity. For example, Is cy.contains('type') for the text, button, link, or all of the visible text on the page? Or what happens if there are two links big type and type , which one will Cypress click?
• The scope definition do to end (Ruby) is better than () => { and }) (Node.js)

The above example is just Cypress in the simplest form. With a proprietary syntax framework, a common pattern is that complex and inconsistent new syntax with the updates. Here is a ‘Get retry attempts’ example.

it('does something differently on retry', { retries: 3 }, () => {     
   // cy.state('runnable') returns the current test object  
   // we can grab the current attempt and   
   // the total allowed attempts from its properties   
   const attempt = cy.state('runnable')._currentRetry   
   const retries = cy.state('runnable')._retries   
   // use the "attempt" and "retries" values somehow 
}))

7. Not Future-Proof

For web test automation, with the only exception of Selenium WebDriver, all others are NOT future-proof. The simple reason is that only Selenium WebDriver is a W3C standard.

“CDP is designed as a debugging protocol, it can change with no notice between versions. That’s why Puppeteer and Cypress are tied to specific versions of browsers, and that causes a dilemma for you as a test author: how do you test on multiple versions of a browser?” — Simon Stewart, 2020–11–16

There are more Cypress/Puppeteer limitations I can go on. However, I will stop here, as the article is getting quite long now. I hope this article, together with Part 1, will convince you to switch to Selenium WebDriver which is far better in every way. If you tried Selenium WebDriver before and failed, you shall seek help from a good mentor.