Quickstart: a verdict on every PR
CertusQA Gate is a GitHub Action that turns the Playwright JSON report you already produce into a pre-deploy verdict you can prove: one of four states, a Proof Artifact per failure, a summary on the run page, and outputs you can gate on. It starts in report-only mode. It records; it blocks nothing until you say so.
Install
Two changes. First, make Playwright write its JSON report. In playwright.config:
reporter: [['list'], ['json', { outputFile: 'test-results/results.json' }]],
Then add one step after your tests, with if: always() so the gate sees the failing run too:
# .github/workflows/e2e.yml
- name: Playwright
run: npx playwright test
continue-on-error: true
- name: CertusQA Gate
id: gate
if: always()
uses: certusqa/gate-action@v1
with:
results: test-results/results.json
mode: report-only # flip to enforce once you have read a few verdicts
- name: Verdict
if: always()
run: echo "CertusQA Gate says ${{ steps.gate.outputs.verdict }}"
That is the whole install. No account, no token, no network call: the action reads one file and writes a directory. The full workflow is in the repository under examples/quickstart.yml.
Optional: record runs on the platform
Add api-key: ${{ secrets.CERTUSQA_API_KEY }} and the action signs the run and uploads it after the evidence is written. History, trends and the judge's classification then live on the platform, and the step outputs run-id and run-url. The secret never leaves the runner except in the request to the platform, which stores only a hash; the request itself is signed with a key derived from it. What is sent is the same gate.json and Proof Artifacts you get locally, with the GitHub run, commit and PR number as the reference. Get a key by creating a free account: 50 runs a month, no card, the key is shown once on the "Connect your CI" page together with this step.
The four verdicts
| Verdict | When | In enforce mode |
|---|---|---|
| CLEAR_TO_DEPLOY | Every test passed first time, and none passed by writing a missing screenshot baseline. | Passes. |
| REVIEW_ARTIFACTS | A test passed only on retry, or a passing test wrote a screenshot baseline instead of comparing one. Not an automatic pass; a human decides. | Passes, unless strict: true. |
| BLOCK_DEPLOY | At least one test failed after its retries. | Fails the job. |
| INSUFFICIENT_EVIDENCE | The report is missing, unreadable, or contains no tests. | Fails the job. |
The last row is the one most gates get wrong. A missing report means the tests did not run. A gate that passes then is not a gate, so this one never treats a missing or corrupt report as green, and the counts it reports in that case are empty rather than zero.
What a verdict also tells you
- It never mistakes a recording for a failure. Only a screenshot, video or trace that a failed or flaky test references counts as failure media. A green run that records video stays
CLEAR_TO_DEPLOY. - It flags green runs that compared nothing. A passing test that wrote a missing screenshot baseline instead of comparing one is named in the summary, and the run is
REVIEW_ARTIFACTS. - It says what it did not run. Projects that are configured but were not run are named beside the counts, and each skipped test is listed with the reason its author gave.
- A shuffled run can be reproduced. If you ran with
--shuffle, the summary prints the seed to re-run with. - It says when a report left the passes out. Playwright's
onlyFailuresreporter option makes a green run look empty. The verdict isINSUFFICIENT_EVIDENCE, and the summary says why.
What you get
Every run writes one directory, uploads it as a workflow artifact, and appends a summary to the run page.
certusqa-gate/
├── gate.json verdict, counts, failures, media, mode, run scope
├── summary.md the same summary that lands on the run page
└── proof/
└── pa_unjudged_f12e8e14.json one per failed or flaky test
These are real files: the output of v1.1.0 on a Playwright 1.65.1 run of a five-test demo suite, with two passes, one deliberate failure, one deliberately flaky test and one skip, and a second browser configured but not run: gate.json, summary.md, a Proof Artifact for the failure, and one for the flaky test.
Proof Artifacts, honestly unjudged
Each artifact has the same shape as the ones the CertusQA engine emits: classification, rootCause, remediation, evidence. From the free action they carry failKind: "unjudged" and confidence: "none". The action records the failure, its attempts, its sanitised error and its media. Deciding whether it is a regression, a flake or a defect in the test is the judge's job, and the judge is the hosted part of CertusQA.
Error text is sanitised before it is written: colour codes are stripped, Playwright's Received block, which is page text, is dropped, and the message is capped. Nothing is sent anywhere.
Inputs and outputs
| Input | Default | Meaning |
|---|---|---|
results | test-results/results.json | Playwright JSON reporter output. |
media-dir | test-results | Scanned for png, webp, webm and zip media. Only files a failed or flaky test references count as failure media. |
mode | report-only | report-only never fails the job; enforce fails on BLOCK_DEPLOY and INSUFFICIENT_EVIDENCE. |
strict | false | In enforce mode, also fail on REVIEW_ARTIFACTS. |
output-dir | certusqa-gate | Where the evidence is written. |
artifact-name | certusqa-gate | Workflow artifact name; an empty string skips the upload. |
working-directory | . | Base for the paths above. |
api-key | empty | Optional platform key from a secret. Set: the run is uploaded after the evidence is written. Unset: no network call. |
api-url | https://app.certusqa.com/api/v1/runs | Only for a self-hosted or test platform. |
Outputs: verdict, report-status (parsed, absent or unparseable), passed, failed, flaky, skipped, output-dir, and with an api-key, run-id and run-url. Use them in a later step, a required check, or a deploy job's if:.
Switching to enforce
Run in report-only for a week. Read the verdicts on the run page. When they match what you would have decided, change one line:
mode: enforce
strict: false # true to block on REVIEW_ARTIFACTS as well
The order of the action's three steps is the reason this is safe: it computes the verdict and writes the evidence first, uploads it second, and enforces last. By the time the job goes red, the evidence is already saved.
What it is not
- It does not run your tests. It reads the report Playwright wrote.
- It does not decide why a test failed. That classification is the CertusQA judge's, and it arrives with the hosted console.
- It does not phone home unless you ask it to. Without
api-keythere is no network call and no account. With it, exactly one signed POST ofgate.jsonandproof/*.jsonto the platform, nothing else, and a refused upload is a warning, never a failed step. The whole action is three small dependency-free files you can read. - It does not repair anything. Self-healing locators are part of the managed service, where a person is in the loop.