Test Automation Process
A test automation effort runs through eight phases, with four quality gates in between. The big difference from a performance engagement: it does not end with a sign-off. Once the suite is trusted, phases 6 to 8 keep looping for as long as the product is alive.
Process at a glance
Your part as a tester
The tester owns phases 4 to 7 and feeds the lead's work in the others.
| Phase | Who leads | Your part as a tester | You hand over |
|---|---|---|---|
| 1 Scope and candidates | Test lead + PO | Mark which manual test cases you run every release; flag flows that are unstable or hard to automate | Candidate list with notes |
| 2 Approach and framework | Automation lead | Try the tool on one real screen; say what the app makes hard (iframes, captcha, OTP, no test IDs) | Proof-of-concept result, blockers |
| 3 Environment and test data | Tester + DevOps | Ask for one test account per role, a data reset, stubs for payment and email, and CI runner access | Environment checklist, test accounts |
| 4 Script development | Tester | Record, clean up locators, add assertions, move steps into page objects, raise a merge request for peer review | Reviewed tests in the repo |
| 5 Stabilisation | Tester | Run each new test many times locally and in CI; fix waits, data clashes and order dependence | Green run history (evidence for G3) |
| 6 Run in pipeline | Tester + CI | Make sure the right suite runs on the right trigger; keep reports and traces as CI artifacts | Run reports, traces |
| 7 Triage and defects | Tester + dev | Read every red run; label it; raise defects with the trace attached | Defect log, triage notes |
| 8 Maintain and report | Automation lead | Update tests when screens change; retire dead ones; supply pass rate, flaky rate and runtime numbers | Maintenance log, metrics |
Phases in detail
Each phase ends with a named output. A gate blocks the next phase until its owner approves.
| # | Phase | Key activities | Inputs | Outputs | Owner | Exit gate |
|---|---|---|---|---|---|---|
| 1 | Scope and candidates | List regression test cases; score each on frequency, business risk, stability and pass condition; pick the first batch; agree what stays manual | Manual test cases, release history, defect history | Automation candidate list, out-of-scope list | Test lead + PO | G1 Scope agreed |
| 2 | Approach and framework | Pick tool and language; decide structure (page objects, fixtures, data files); naming and tagging rules; proof of concept on one real flow; set entry/exit criteria (pass rate, flaky limit, maximum runtime) | Candidate list, app tech stack, team skills | Automation plan, framework skeleton, coding guide | Automation lead | G2 Plan approved |
| 3 | Environment and test data | Stable test environment; accounts per role; seeded data that resets; stubs for third parties; CI runner and secrets | Automation plan | Environment checklist, test data, CI access | Tester + DevOps | Checklist complete |
| 4 | Script development | Record a flow; replace brittle locators; add one clear assertion per pass condition; move steps into page objects; pull data out of the script; peer review by merge request | Candidate flows, framework, test data | Reviewed tests, linked to test case IDs | Tester | Merge request approved |
| 5 | Stabilisation | Repeat each new test locally; run in CI across several pipelines; fix flakes at the cause; run in parallel to catch shared data | Merged tests | Green run history, known-issue list | Tester | G3 Suite ready: N green CI runs |
| 6 | Run in pipeline | Smoke after deploy, regression on merge request or nightly; store the report, traces and screenshots on failure | Trusted suite, CI config | Run reports, traces | Tester + CI | Run complete |
| 7 | Triage and defects | Label each failure; raise product defects with trace and steps; fix test bugs; quarantine flaky tests with a ticket; rerun after fixes (loops back to phase 6) | Failed runs | Defect log, triage notes, quarantine list | Tester + dev | G4 Release regression green, or failures accepted |
| 8 | Maintain and report | Update tests for screen changes; add the next candidate batch; delete tests that no longer pay; report metrics; lessons learned each release | Run history, change list, defect log | Metrics report, updated suite, next candidates | Automation lead | Ongoing (loops back to phase 6) |
The Playwright guides cover the hands-on part of several phases: Understand What to Automate (phase 1), Configuration and Running (phase 2), Authentication and State (phase 3), Recording with Codegen, Locators and Assertions and Test Structure and Page Objects (phase 4), Debug (phases 5 and 7) and Running in CI (phase 6).
From recording to a trusted suite
A new test only joins the regression suite once it has passed many times in a row in CI. A suite with even a few random failures teaches the team to ignore red, and then it stops catching real bugs.
- Record: gives you a first draft of the steps fast. It is never the finished test.
- Clean up: swap brittle CSS paths for role, label or test-ID locators; add the assertion that proves the pass condition; move data out of the script.
- Repeat locally: run the same test several times in a row (in Playwright,
--repeat-each=5). This shows timing problems that one green run hides. - Peer review: another tester reads the merge request for clear names, one flow per test, and no fixed waits.
- CI × N runs: the CI machine is slower and runs tests in parallel, so it finds problems your laptop does not. After N green runs in a row, the test gets the regression tag.
When a test fails
Every red run gets one of four labels before anyone acts on it. Open the trace or step record first: it shows each step, the screen at that moment and the network calls.
| Label | How you know | What you do |
|---|---|---|
| Product bug | The app does the wrong thing; you can repeat it by hand | Raise a defect with the trace, screenshot and steps |
| Test bug | The app is right; the locator, assertion or data in the test is wrong | Fix the test in a merge request |
| Environment or data | Server down, account locked, data already used | Tell DevOps or reset the data; rerun |
| Flaky | Passes on rerun with no code change | Quarantine with a ticket; fix the cause within the sprint |
Tips
- Start with the smoke suite. Five tests that run after every deploy show value in the first week and get the team used to watching results.
- Put the test case ID in the test name. For example
TC-102 place order as guest. Coverage reports and defect links then trace back to the manual case. - Ask developers for test IDs early. A
data-testidon key buttons and fields, agreed in phase 2, saves more upkeep than any locator trick. - Never fix flakiness with a fixed wait. A
waitForTimeout(3000)hides the problem and makes the suite slower. Wait for what the user waits for: a visible element or a finished request. - Retries are a signal, not a cure. If CI retries a test and it passes on the second try, the report marks it flaky. Count those; a rising flaky rate means the suite is losing trust.
- Track three numbers each release. Pass rate, flaky rate and suite runtime. When runtime grows past what the team will wait for, split the suite or shard it.
- Set N in the automation plan. Five green CI runs in a row is a common starting point. Agree it at G2 so nobody argues about it later.
- Delete tests that no longer pay. A test for a retired feature, or one that needs fixing every sprint without ever catching a bug, costs more than it saves.