Problems that always pass visual regression tests run with the GitHub action #261
Description
Any test that intentionally fails in a local environment is always passed when it is run as a GitHub action. I checked the screenshot of the cypress dashboard to see what happens when I run the GitHub action.
The first image is a baseline screenshot.
The second image is a snapshot of the ui intended to fail. An extra button has been added.
As I assume, the test should not pass if the snapshot taken in the second GitHub action test differs from the baseline in the first GitHub action test after the baseline is generated by the first GitHub action test.
In fact, local tests do not pass the above tests. The settings are as follows.
// commands.js
import { addMatchImageSnapshotCommand } from "cypress-image-snapshot/command";
addMatchImageSnapshotCommand({
failureThreshold: 0.001,
failureThresholdType: "percent",
customDiffConfig: { threshold: 0.1 },
capture: "viewport",
});
However, if you try to test with GitHub action, all of them succeed.
(Run Starting)
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Cypress: 11.1.0 │
│ Browser: Chrome 107 (headless) │
│ Node Version: v16.13.0 (/home/runner/runners/2.299.1/externals/node16/bin/node) │
│ Specs: 1 found (cameraTranslateInDirection.cy.ts) │
│ Searched: cypress/e2e/**/*.cy.{js,jsx,ts,tsx} │
│ Params: Tag: false, Group: false, Parallel: false │
│ Run URL: https://cloud.cypress.io/projects/9iq3e5/runs/3 │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
────────────────────────────────────────────────────────────────────────────────────────────────────
Running: cameraTranslateInDirection.cy.ts (1 of 1)
Estimated: 9 seconds
translateInDirection
✓ translate to N (2162ms)
✓ translate to NE (687ms)
✓ translate to E (655ms)
✓ translate to SE (621ms)
✓ translate to S (636ms)
✓ translate to SW (641ms)
✓ translate to W (706ms)
✓ translate to NW (725ms)
✓ translate to backward (976ms)
✓ translate to forward (1121ms)
10 passing (9s)
(Results)
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Tests: 10 │
│ Passing: 10 │
│ Failing: 0 │
│ Pending: 0 │
│ Skipped: 0 │
│ Screenshots: 10 │
│ Video: true │
│ Duration: 8 seconds │
│ Estimated: 9 seconds │
│ Spec Ran: cameraTranslateInDirection.cy.ts │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
The reasons I expect are as follows
- The threshold setting for the github action exists independently, and the default threshold value is more generous than the threshold value I set in the project(cypress.config).
- The test always passes because the baseline of the test to be attempted with the github action is initialized every time. ⇒ If so, how do you generate baseline and where is it stored?
I hope the results of the local test and the test run by GitHub action are the same. I'd really appreciate your help
name: CI
on: [push]
jobs:
test-integration:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/setup-node@v2
with:
node-version: "14"
cache: "yarn"
- uses: cypress-io/github-action@v4
with:
browser: chrome
record: true
env:
CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
// cypress.config.js
import { defineConfig } from "cypress";
import { addMatchImageSnapshotPlugin } from "cypress-image-snapshot/plugin";
export default defineConfig({
projectId: '******',
e2e: {
setupNodeEvents(on, config) {
addMatchImageSnapshotPlugin(on, config);
},
},
});