Playwright vs Selenium Speed Comparison

Selenium WebDriver is slightly faster than Playwright.

10 min readJan 8, 2023

A repost of my daughter’s article with permission. I added a few notes.

In the previous article, I compared the syntax of Playwright and Selenium WebDriver. One feedback I have heard is that “Selenium is slower”, I disagree. Both are fast in terms of test execution, from my experience.

Today, I did a quick benchmark to verify the timing difference.

Table of Contents:
· Test Setup
· Test Script
· Benchmark Results (raw)
· Refactored Versions
∘ Benchmark Results (POM)
· Findings
∘ Why I did not include Cypress?
· Zhimin’s notes
∘ The language factor is minor, in terms of test execution speed

Test Setup

Test Machine: iMac (2015 model, macOS 12.6.2)
I chose a relatively slow machine purposefully.
Target website: https://travel.agileway.net
This a simple test site created by my father for training purposes.
Execution method
I developed both versions in TestWise IDE. To remove any factors, I timed and ran them both from the command line.
Framework versions
- Selenium WebDriver: 4.7.1 (Ruby 3.0.2)
- Playwright 1.18.1 (Node v18.10)
Browser versions
- Chrome: 108.0.5359.124 (Official Build)
- Playwright’s Chromium: 99.0.4812.0 (Developer Build)
Test Scenarios
24 test steps cover:
- Launching and Closing the Chrome/Chromium browser
- Various control types, such as Link, Text Field, Checkbox, Radio, Select List, and Button
- Different locators, such as ID, Name, Tag, Attribute, CSS and XPath.
- Assertion

Zhimin: Some framework/tool vendors release dubious benchmark test results, with specially optimized web pages. That is fraud. The site used for this benchmark testing just contains several simple generic web pages, good for testing raw execution speed.

Test Script

TestWise supports Mocha too, i.e. JavaScript frameworks, such as Playwright. But using Ruby is much more fun!

Raw Selenium WebDriver (Ruby) in RSpec syntax framework

  it "End-to-End Selenium Raw" do
    driver.find_element(:id, "username").send_keys("agileway")
    driver.find_element(:id, "password").send_keys("testwise")
    driver.find_element(:id, "remember_me").click
    driver.find_element(:id, "username").submit
    expect(driver.find_element(:tag_name, 'body').text).to include("Signed in")

    driver.find_element(:xpath, "//input[@name='tripType' and @value='oneway']").click
    Selenium::WebDriver::Support::Select.new(driver.find_element(:name, "fromPort")).select_by(:text, "Sydney")
    Selenium::WebDriver::Support::Select.new(driver.find_element(:name, "toPort")).select_by(:text, "New York")
    Selenium::WebDriver::Support::Select.new(driver.find_element(:id, "departDay")).select_by(:text, "02")
    Selenium::WebDriver::Support::Select.new(driver.find_element(:id, "departMonth")).select_by(:text, "May 2016")
    driver.find_element(:xpath, "//input[@value='Continue']").click

    # now on passenger page
    driver.find_element(:name, "passengerFirstName").send_keys("Bob")
    driver.find_element(:name, "passengerLastName").send_keys("Tester")
    driver.find_element(:name, "passengerLastName").submit

    # on payment page
    driver.find_element(:xpath, "//input[@name='card_type' and @value='master']").click
    driver.find_element(:name, "holder_name").send_keys("Bob the Tester")
    driver.find_element(:name, "card_number").send_keys("4242424242424242")
    Selenium::WebDriver::Support::Select.new(driver.find_element(:name, "expiry_month")).select_by(:text, "04")
    Selenium::WebDriver::Support::Select.new(driver.find_element(:name, "expiry_year")).select_by(:text, "2016")
    driver.find_element(:xpath, "//input[@value='Pay now']").click
  end

Raw Playwright in Mocha test syntax framework


  it('End to End Playwright', async function() {
    this.timeout(5000)
    await driver.locator('text=Login').click()
    await driver.fill("#username", "agileway")
    await driver.fill("#password", "testwise")
    await driver.click("#remember_me")
    await driver.click("input:has-text('Sign in')")
    await driver.textContent("body").then(function(body_text) {
      //console.log(body_text)
      assert(body_text.contains("Signed in"))
    });
    
    const trip_radio = await driver.$$("input[name=tripType]");
    await trip_radio[1].check();
    await driver.selectOption("select[name='fromPort']", "New York");
    await driver.selectOption("select[name='toPort']", "Sydney");
    await driver.selectOption("select[name='departDay']", "02");
    await driver.selectOption("#departMonth", "052021");
    await driver.click("input:has-text('Continue')");
    
    await driver.fill("input[name='passengerFirstName']", "Bob");    
    await driver.fill("input[name='passengerLastName']", "Tester");
    await driver.click("input:has-text('Next')");

    const card_type_radio = await driver.$$("input[name='card_type']");
    await card_type_radio[1].check();
    await driver.fill("input[name='holder_name']", "Bob the Tester");
    await driver.fill("input[name='card_number']", "4242424242424242");
    await driver.selectOption("select[name='expiry_month']", "04");
    await driver.selectOption("select[name='expiry_year']", "2016");
    await driver.click("input:has-text('Pay now')");
  });

Benchmark Results (raw)

Zhimin: Please note, all the timings are in normal execution mode, not headless (which will be a further ~8% faster, according to my benchmark test).

1. Raw Selenium WebDriver, 2.21

Raw Playwright: The best of 3 runs, the other two: 2.23, 2.50

2. Playwright, 2.34

Raw Playwright: The best of 3 runs, the other two: 2.43, 2.44

Zhimin: besides the comparison (see below). I will point out that both are quite fast driving the web pages in a browser: completing 24 steps (including launching and quit the browser) ~2 seconds.

Timing of the same test on Mac Mini M1 (2020 model), is ~100% faster than my old iMac 2015 model.

I am not sure I want test execution any more faster, otherwise, I won’t be able to see the app. I detected many defects by watching test execution.
By the way, because of Selenium test execution is so fast on new hardware, it is possible to perform load testing using Selenium, using a parallel testing lab. In 2018, I implemented the №1 load testing challenge for a large company with 24 build agents. The performance team tried for 2 years and failed, I accomplished that, using selenium in CT server with parallel testing, within one week. Apple’s M3 chip (TSMC’s 3nm process) is expected to be a further big improvement. If you are interested, check out my book: Practical Performance and Load Testing and article: Poor Software Load Testing Can Cause Many People Misery.
My point here is: there is no value to debate the test execution speed, such as the set-up for this benchmark test, if reaching the top limit of useful range (Selenium and Playwright both did) in functional testing perspective. Still, we want to shorten the overall execution time, the only practical way is via parallel execution in a proper way (many JS frameworks got that wrong, it shall never be the automation’s framework’s responsiblity, its CT server’s). Check out my article, My Innovative Solution to Continuous Testing: Parallel Automated End-to-End Test Execution.

Refactored Versions

After getting the test steps right, as a habit, I immediately refactor based on the Maintainable Automated Test Design, using Page Object Models, a well-known design pattern in test automation. Because the raw linear test steps are hard to read and maintain.

Selenium WebDriver using POM

  it "End-to-End Selenium POM" do
    login_page = LoginPage.new(driver)
    login_page.enter_username("agileway")
    login_page.enter_password("testwise")
    login_page.check_remember_me
    login_page.click_sign_in
    expect(driver.find_element(:tag_name, 'body').text).to include("Signed in")

    flight_page = FlightPage.new(driver)
    flight_page.select_trip_type("oneway")
    flight_page.select_depart_from("Sydney")
    flight_page.select_arrive_at("New York")
    flight_page.select_depart_day("02")
    flight_page.select_depart_month("May 2016")
    flight_page.click_continue

    passenger_page = PassengerPage.new(driver)
    passenger_page.enter_first_name("Bob")
    passenger_page.enter_last_name("Tester")
    passenger_page.click_next

    payment_page = PaymentPage.new(driver)
    payment_page.select_card_type("master")
    payment_page.enter_holder_name("Bob the Tester")
    payment_page.enter_card_number("4242424242424242")
    payment_page.enter_expiry_month("04")
    payment_page.enter_expiry_year("2016")
    payment_page.click_pay_now
  end

Playwright using POM

it('E2E Playwright POM', async function() {
    this.timeout(5000);
    
    let login_page = new LoginPage(driver);
    await login_page.enterUsername("agileway");
    await login_page.enterPassword("testwise");
    await login_page.checkRememberMe();
    await login_page.clickSignIn();
    await driver.textContent("body").then(function(body_text) {
      assert(body_text.contains("Signed in"));
    });
    
    let flight_page = new FlightPage(driver);
    await flight_page.selectTripType("oneway");
    await flight_page.selectDepartFrom("New York");
    await flight_page.selectArriveAt("Sydney");
    await flight_page.selectDepartDay("02");
    await flight_page.selectDepartMonth("052021");
    await flight_page.clickContinue();

    let passenger_page = new PassengerPage(driver);
    await passenger_page.enterFirstName("Bob");
    await passenger_page.enterLastName("Tester");
    await passenger_page.clickNext();
    
    let payment_page = new PaymentPage(driver);
    await payment_page.selectCardType("master");
    await payment_page.enterHolderName("Bob the Tester");
    await payment_page.enterCardNumber("4242424242424242");
    await payment_page.enterExpiryMonth("04");
    await payment_page.enterExpiryYear("2016");
    await payment_page.clickPayNow();
});

Astute readers will conclude that putting the differences in language conventions aside, these two versions are quite similar. Yes, that’s the beauty of Page Object Models.

Benchmark Results (POM)

1. Raw Selenium WebDriver using POM design, 2.23

Raw Selenium POM: The best of 3 runs, the other two: 2.24, 2.49

2. Playwright using POM design, 2.34

Raw Playwright: The best of 3 runs, the other two: 2.35, 2.35

Zhimin: as expected, there is virtually no loss in speed for POM tests. So, always refactor!

Findings

For this simple test scenario, Selenium WebDriver is about 5% faster than Playwright for both the raw and Page Object Model versions. The difference is minor, after taking out variances (network latency at the time and web page content), I would say they are on par, in terms of test execution speed.

This finding was similar to the one by Giovanni Rago two years ago (2021–01), except Playwright was slightly faster than Selenium then.

With Selenium 4 and the ongoing refinement of ChromeDriver, I have noticed a speed improvement in Selenium test execution. So, I am not too surprised by the result: Selenium is faster than Playwright.

Why I did not include Cypress?

I just did a google search of “Playwright vs Cypress”, and the results page shows, if there is any judgement, favour Playwright.

Speed-wise, Playwright is faster than Cypress, according to this post.

Giovanni Rago did a thorough test comparing the performance of tools under different scenarios: Cypress vs Selenium vs Playwright vs Puppeteer speed comparison. The test result: not much difference in terms of speed for the four frameworks, however, Cypress was the slowest in most categories.

So I did not bother timing Cypress; even though they claimed to be much, much faster on their web page.

Zhimin’s notes

Update 2023–01–15: Check out my article, Execution Speed of Automated End-to-End (via UI) Testing Clarified.

The language factor is minor, in terms of test execution speed

Please notice the languages used in the test scripts.

Selenium: Ruby
Hired’s 2023 State of Software Engineers Report: “Ruby on Rails and Ruby are the most in-demand skills.”
Playwright: JavaScript

I have heard many times that people slam Ruby for being slow. Oh well, Selenium Ruby is faster than Playwright JavaScript for this benchmark test.

Yes, for the raw execution speed of a computer program, JavaScript is much faster than Ruby. But people cannot use that in the context of test automation, as the majority of execution time is spent on the execution engine (usually native), network latency, and mostly the web page content. The script language speed difference is very minor, and can be ignored.

Therefore, choosing a concise, easy-to-read, and fun scripting language for test automation, such as Ruby.

FAQ

1. Why the test execution (Selenium, Playwright, or Cypress) is a lot slower for my website, compared to your benchmark?

The majority of the time (in executing an automated web test) is on the website itself, such as rendering pages (HTML, CSS and JavaScript) and server processing time (e.g. submitting a payment).

The test site used in this benchmark is very basic.

2. In some cases, I found Cypress is close and maybe a little faster than Selenium.

Maybe for special cases, remember, Cypress is written in JavaScript. Selenium is a generic solution.

3. I ran your test script, Selenium indeed is very fast. Why do some people claim “Selenium is slow”?

Lies exist for a reason or reasons. In the commercial world, one reason is Money. Those commercial test automation tool vendors knew their stuff could not compete with free, open, standard-based Selenium WebDriver in any aspect, what would they do? Checkout out my other article, Evil Mudslingings against Selenium WebDriver.

It is unfortunate that so many fake automated testers (most in percentage, not convinced, try to answer “One Simple Test Automation Scenario Interview Question that Most Candidates Failed”) did NOT have test automation knowledge, and they ruined the reputation of “Test Automation Engineer”, along with it, some of Selenium. There are plenty of obvious mistakes, such as:

Using Java/C# compiled language. Wrong! Should use a scripting language such as Ruby.
Make good Selenium syntax worse by adding a ‘better syntax’ tier, such as the failed Protractor.
Using Gherkin syntax frameworks such as Cucumber and SpecFlow. Wrong! Too much maintenance overhead, unnecessarily!
Using wrong tools such as Visual Studio Code. Not good, that’s a tool for programmers. Should use TestWise or JetBrains Aqua.
Run tests in CI servers such as Jenkins. Wrong, CI servers are for executing unit/integration tests. Instead, should use CT servers such as BuildWise or Facebook’s sandcastle.
Adding auto-retry into the framework. Wrong! That’s the responsibility of the CT Server.
…

“Selenium WebDriver is still better than any other web test automation framework in every aspect: feature complete, standard-conforming, speed, reliable, easy-to-learn, flexible, free (in both freedom and price), support, …, etc.”— Zhimin Zhan

4. This is not a real benchmark test, you need to run a longer time with a more complex website.

This is not meant to be a research experiment (I was a research scientist). The purpose is to prove Selenium is fast (please reread my conclusion part) for this quick exercise. I know some Playwright testers won’t be happy, but at least in this basic scenario, Selenium is objectively faster, accept that. As I said, Selenium is fast enough, if talking about real test automation, all things considered, speed is not that important (if running in a CT server with parallel execution, I will show proof shortly).

However, many fake automated testers used “Selenium is slow”, a false statement, to defame Selenium. A few years ago, many Cypress testers claimed, “Cypress is much faster (than Selenium)” (this is still shamelessly on its website).

“Every JavaScript tester, using PhantomJS, Protractor, TestCafe, Puppeteer, Cypress and Playwright, I met was fake. None could develop and maintain over 50 tests (Level 2 of AgileWay Continuous Testing Grading) that runs daily and remain valid” — Zhimin Zhan

Before my daughter started in an intern role (late last year), I told her the above observation. It turned out that, at work, the senior test automation engineer, who was promoting Playwright, was a fake. Check out this story, An IT Graduate’s frustration with a Fake ‘Senior Test Automation Engineer’.

Back to the question, why not a longer benchmark test? There is no need, I haven’t yet witnessed a single test automation success using Playwright. The best one so far is my daughter’s implementation in her first intern role: running 29 Playwright tests daily in the BuildWise CT server, remaining valid. That’s because she did (develop & maintain) in Selenium RSpec (in Ruby) first, then converted to Playwright. By the way, she did two demonstrations in the department, people there were shocked and had never seen such high productivity in test automation. And, it was from an intern. But in my view, her productivity is certainly not good enough for serious test automation, she still needs to learn and practice, a lot.

In real test automation with a large test suite (will show shortly), there is a much higher requirement for a real test automation engineer. That’s why real test automation engineers are extremely rare, even the LinkedIn co-founder needed to lure one (check out Wired’s The Software Revolution Behind LinkedIn’s Gushing Profits).

Below is a recent execution of my WhenWise regression test suite: 559 raw Selenium tests. After getting the green run, I push the build to production. I have been doing this for all my web apps since 2012, and some of my clients’ since 2008 (before 2011 when Selenium WebDriver was released, I used Watir).

Run in BuildWise CT server on 5 build agents.

Related reading: