ChatGPT is Useless for Real Test Automation

Avoid yet another hype

Zhimin Zhan
9 min readJan 1, 2023

--

Update (2023–02–023): Two Simple Reasons that ChatGPT (AI) Won’t Work in End-to-End Test Automation.

Update (2023–6–09): ChatGPT is out for over 6 months. Have you implemented real test automation with AI?

Non-Medium-Members: you can read this article free on Vocal.

Coding Jag (a software testing newsletter) #120 featured three articles on ChatGPT:

I am aware of this because my daughter’s article, Playwright vs Selenium WebDriver Syntax Comparison by Example, is also featured in Coding Jag #120.

Some of you might have heard of “ChatGPT” and AI in Test automation in general. I will give my conclusion first: “ChatGPT is totally useless for Real Test Automation”. I will illustrate with ChatGPT examples.

Table of Contents:
·
A Simple User Login Test
My Assessment
How about a more complex test scenario?
·
Argument 1: “It generates skeleton”. Yes, in the wrong way.
·
A much better way to generate a test skeleton.
·
Argument 2: “It generates test data”. Yes, again, there is a better way.
·
Argument 3: “By using ChatGPT in more sessions, you can train it to do better”

A Simple User Login Test

A user login test in Test Automation is regarded as the “HelloWorld” in coding. Logically, we expect AI Testing to score perfectly on this simplest and well-trained test scenario. Let’s see how well ChatGPT performs.

1. My Request:

2. ChatGPT’s Answer (part 1: test design in steps)

3. ChatGPT Answer (part 2: test script)

--

--

Zhimin Zhan

Test automation & CT coach, author, speaker and award-winning software developer. Help teams succeed with Agile/DevOps by implementing real Continuous Testing.