Quickstart: a verdict on every PR

CertusQA Gate is a GitHub Action that turns the Playwright JSON report you already produce into a pre-deploy verdict you can prove: one of four states, a Proof Artifact per failure, a summary on the run page, and outputs you can gate on. It starts in report-only mode. It records; it blocks nothing until you say so.

Install

Two changes. First, make Playwright write its JSON report. In playwright.config:

reporter: [['list'], ['json', { outputFile: 'test-results/results.json' }]],

Then add one step after your tests, with if: always() so the gate sees the failing run too:

# .github/workflows/e2e.yml
- name: Playwright
  run: npx playwright test
  continue-on-error: true

- name: CertusQA Gate
  id: gate
  if: always()
  uses: certusqa/gate-action@v1
  with:
    results: test-results/results.json
    mode: report-only   # flip to enforce once you have read a few verdicts

- name: Verdict
  if: always()
  run: echo "CertusQA Gate says ${{ steps.gate.outputs.verdict }}"

That is the whole install. No account, no token, no network call: the action reads one file and writes a directory. The full workflow is in the repository under examples/quickstart.yml.

Optional: record runs on the platform

Add api-key: ${{ secrets.CERTUSQA_API_KEY }} and the action signs the run and uploads it after the evidence is written. History, trends and the judge's classification then live on the platform, and the step outputs run-id and run-url. The secret never leaves the runner except in the request to the platform, which stores only a hash; the request itself is signed with a key derived from it. What is sent is the same gate.json and Proof Artifacts you get locally, with the GitHub run, commit and PR number as the reference. Get a key by creating a free account: 50 runs a month, no card, the key is shown once on the "Connect your CI" page together with this step.

The four verdicts

The last row is the one most gates get wrong. A missing report means the tests did not run. A gate that passes then is not a gate, so this one never treats a missing or corrupt report as green, and the counts it reports in that case are empty rather than zero.

What you get

Every run writes one directory, uploads it as a workflow artifact, and appends a summary to the run page.

certusqa-gate/
├── gate.json            verdict, counts, failures, media, mode
├── summary.md           the same summary that lands on the run page
└── proof/
    └── pa_unjudged_8f1b8b0e.json   one per failed or flaky test

These are real files from the action's own test run, on a fixture with one pass, one failure, one flaky test and one skip: gate.json, summary.md, a Proof Artifact for the failure, and one for the flaky test.

Proof Artifacts, honestly unjudged

Each artifact has the same shape as the ones the CertusQA engine emits: classification, rootCause, remediation, evidence. From the free action they carry failKind: "unjudged" and confidence: "none". The action records the failure, its attempts, its sanitised error and its media. Deciding whether it is a regression, a flake or a defect in the test is the judge's job, and the judge is the hosted part of CertusQA.

Error text is sanitised before it is written: colour codes are stripped, Playwright's Received block, which is page text, is dropped, and the message is capped. Nothing is sent anywhere.

Inputs and outputs

Outputs: verdict, report-status (parsed, absent or unparseable), passed, failed, flaky, skipped, output-dir, and with an api-key, run-id and run-url. Use them in a later step, a required check, or a deploy job's if:.

Switching to enforce

Run in report-only for a week. Read the verdicts on the run page. When they match what you would have decided, change one line:

    mode: enforce
    strict: false   # true to block on REVIEW_ARTIFACTS as well

The order of the action's three steps is the reason this is safe: it computes the verdict and writes the evidence first, uploads it second, and enforces last. By the time the job goes red, the evidence is already saved.

What it is not