ChatGPT is Useless for Real Test Automation
Update (2023–02–023): Two Simple Reasons that ChatGPT (AI) Won’t Work in End-to-End Test Automation.
Non-Medium-Members: you can read this article free on Vocal.
Coding Jag (a software testing newsletter) #120 featured three articles on ChatGPT:
- ChatGPT for testers
- Using ChatGPT for Test Automation
- ChatGPT for iMessage, Slack, user feedback, and beyond
I am aware of this because my daughter’s article, Playwright vs Selenium WebDriver Syntax Comparison by Example, is also featured in Coding Jag #120.
Some of you might have heard of “ChatGPT” and AI in Test automation in general. I will give my conclusion first: “ChatGPT is totally useless for Real Test Automation”. I will illustrate with ChatGPT examples.
Table of Contents:
· A Simple User Login Test
∘ My Assessment
∘ How about a more complex test scenario?
· Argument 1: “It generates skeleton”. Yes, in the wrong way.
· A much better way to generate a test skeleton.
· Argument 2: “It generates test data”. Yes, again, there is a better way.
· Argument 3: “By using ChatGPT in more sessions, you can train it to do better”
A Simple User Login Test
A user login test in Test Automation is regarded as the “HelloWorld” in coding. Logically, we expect AI Testing to score perfectly on this simplest and well-trained test scenario. Let’s see how well ChatGPT performs.
1. My Request:
2. ChatGPT’s Answer (part 1: test design in steps)
3. ChatGPT Answer (part 2: test script)