Running in CI

A UI suite only earns its keep when it runs on every change without anyone remembering to start it. This page wires the sample project into GitHub Actions, keeps the report and traces as artifacts, and shows how to split a slow suite across machines. Concepts carry over to GitLab CI and Azure Pipelines. This is the automation step of the JMeter track.

Note: The workflow on this page is the one generated by npm init playwright@latest and matches the official Playwright docs. I have not yet run it against this repository's own CI, so treat the sharding section in particular as not yet validated here.

The Basic Workflow

npm init playwright@latest with the GitHub Actions option writes .github/workflows/playwright.yml:

name: Playwright Tests
on:
  push:
    branches: [ main, master ]
  pull_request:
    branches: [ main, master ]
jobs:
  test:
    timeout-minutes: 60
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - uses: actions/setup-node@v4
      with:
        node-version: lts/*
    - name: Install dependencies
      run: npm ci
    - name: Install Playwright Browsers
      run: npx playwright install --with-deps
    - name: Run Playwright tests
      run: npx playwright test
    - uses: actions/upload-artifact@v4
      if: ${{ !cancelled() }}
      with:
        name: playwright-report
        path: playwright-report/
        retention-days: 30

What each step does:

Step	Why
npm ci	Installs the exact versions from package-lock.json, including @playwright/test
npx playwright install --with-deps	Downloads the browsers for this Playwright version and the Linux system libraries they need
npx playwright test	Runs with CI=1 set by GitHub, so the config picks up retries: 2, workers: 1, forbidOnly: true
upload-artifact	Keeps the HTML report even when tests fail (!cancelled()), for 30 days

If the sample project lived at the repo root this would work as is. Because it lives in playwright/sample/, add a defaults block or working-directory on the run steps:

    defaults:
      run:
        working-directory: playwright/sample

and point the artifact path at playwright/sample/playwright-report/.

Tip: Only install the browsers you use: npx playwright install chromium --with-deps. It cuts a minute or two off every run.

---

Reading Results from CI

1. Open the workflow run on GitHub
2. Scroll to Artifacts and download playwright-report
3. Unzip it and serve it (the report needs a web server for traces to load):

   npx playwright show-report path/to/unzipped/playwright-report

4. Click a failed test, then the trace icon, to open the trace viewer exactly as in Section 7

Because the config records trace: 'on-first-retry', every CI failure that was retried comes with a trace. That is usually enough to fix it without reproducing locally.

---

Environment and Secrets

• Base URL: pass it as an env var so the same workflow can target staging or a preview environment:

      - name: Run Playwright tests
        run: npx playwright test
        env:
          BASE_URL: ${{ vars.STAGING_URL }}

• Credentials: GitHub Secrets, never the repo. Read them in the setup test with process.env.TEST_USER and process.env.TEST_PASSWORD.
• The saved auth state is created fresh by the setup project on every run. Nothing to store.

---

Sharding a Slow Suite

When the suite takes longer than you are willing to wait, split it across parallel jobs. Each job runs a slice and writes a blob report, then one job merges them into a single HTML report.

The config already switches to blob in CI:

reporter: process.env.CI ? 'blob' : 'html',

Workflow:

jobs:
  test:
    timeout-minutes: 60
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shardIndex: [1, 2, 3, 4]
        shardTotal: [4]
    steps:
    - uses: actions/checkout@v4
    - uses: actions/setup-node@v4
      with:
        node-version: lts/*
    - run: npm ci
    - run: npx playwright install --with-deps
    - name: Run Playwright tests
      run: npx playwright test --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}
    - name: Upload blob report
      if: ${{ !cancelled() }}
      uses: actions/upload-artifact@v4
      with:
        name: blob-report-${{ matrix.shardIndex }}
        path: blob-report
        retention-days: 1

  merge-reports:
    if: ${{ !cancelled() }}
    needs: [test]
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - uses: actions/setup-node@v4
      with:
        node-version: lts/*
    - run: npm ci
    - name: Download blob reports
      uses: actions/download-artifact@v4
      with:
        path: all-blob-reports
        pattern: blob-report-*
        merge-multiple: true
    - name: Merge into HTML report
      run: npx playwright merge-reports --reporter html ./all-blob-reports
    - name: Upload HTML report
      uses: actions/upload-artifact@v4
      with:
        name: html-report--attempt-${{ github.run_attempt }}
        path: playwright-report
        retention-days: 14

fail-fast: false keeps the other shards running when one fails, so you see every failure in one run. The merge step is the same command you can run locally:

npx playwright merge-reports --reporter html ./all-blob-reports

---

Flaky Test Policy

Retries make CI green. They also hide tests that are slowly rotting. Decide the policy up front:

• The HTML report marks a test flaky when it failed and then passed on retry. Read that list every week.
• To make flakiness fail the build: failOnFlakyTests: true in the config, or --fail-on-flaky-tests on the command line. Good for a small, stable suite. Too strict for a new one.
• A test that is flaky twice gets a ticket, not another retry.

---

Other CI Systems

The steps are the same everywhere: Node, npm ci, npx playwright install --with-deps, npx playwright test, keep playwright-report/. Playwright also publishes a Docker image (mcr.microsoft.com/playwright) with browsers and system dependencies preinstalled, which is the fastest option on GitLab CI and Azure Pipelines. Match the image tag to your Playwright version.

---

Tips

• Run on pull requests, not just main. The point is to catch the break before it merges.

• Keep the retention short for blob reports and long for the merged HTML. Blobs are intermediate, the HTML is the record.

• Pin the Playwright version in package.json and upgrade on purpose. A browser update on CI that does not match your local one is a classic "works on my machine".

• Lint in CI too. npx tsc --noEmit catches broken imports and a lot of missing await mistakes before any browser starts.