Mutation never killed

[Julien-Marpault]

Hello, I'm wondering if I'm totally dumb because I can't kill any mutation.

I have a project on .Net 9, with the last release of Stryker.

Here's a sample case:

As you can see the third mutation is the removal of the call to repository.

I have two tests failing if I remove manually the call to the repo, but Stryker still consider mutants have survived.

Is anyone there having an idea about this issue ?

Thanks.

[rouke-broersma]

Are the tests correctly identified as being related to the mutant? We match unit tests with mutants and if we make a mistake here, it could be that stryker does not recognize the unit test and does not execute it. You can try running stryker with coverage analysis turned off and see if the mutant is killed then: https://stryker-mutator.io/docs/stryker-net/configuration/#coverage-analysis-string

If so, we may have a detection issue.

[Julien-Marpault]

Unfortunately it doesn't change anything:

As you can see, removing the call to the repo make test fail.
But Stryker still report survived mutations.
I'm using XUnit V3, and Microsoft Testing Platform for this project.

config file:

{
  "stryker-config": {
    "concurrency": 12,
    "mutation-level": "Standard",
    "language-version": "latest",
    "additional-timeout": 5000,
    "mutate": [
      "**/*"
    ],
    "solution": null,
    "configuration": null,
    "target-framework": null,
    "project": "",
    "coverage-analysis": "off",
    "disable-bail": false,
    "disable-mix-mutants": false,
    "thresholds": {
      "high": 80,
      "low": 60,
      "break": 0
    },
    "verbosity": "info",
    "reporters": [
      "Progress",
      "Html"
    ],
    "since": {
      "enabled": false,
      "ignore-changes-in": [],
      "target": "master"
    },
    "baseline": {
      "enabled": false,
      "provider": "disk",
      "azure-fileshare-url": "",
      "fallback-version": "master"
    },
    "dashboard-url": "https://dashboard.stryker-mutator.io",
    "test-projects": [],
    "test-case-filter": "",
    "ignore-mutations": [],
    "ignore-methods": [],
    "report-file-name": "mutation-report",
    "break-on-initial-test-failure": false
  }
}

Command line:

dotnet stryker --project "src/BookStarter.Application/BookStarter.Application.csproj"  --test-project "tests/BookStarter.Application.Tests/BookStarter.Application.Tests.csproj" --output "StrykerOutput"

[dupdob]

hi @Julien-Marpault

Here is the overall strategy when you face surprising results such as here:

1. check that at least some mutants have been killed. If none has been killed there is a general issue with the testing part. Your screen cap only show survivors, so the problem may be here
2. check the log for error messages. try and solve any reported error and/or open a discussion/issue if no test have been executed
3. run again with coverage analysis disabled (like you did). If it solves the problem, you ran into an issue with coverage analysis (this rare nowadays)
4. check if some of the unit tests are failing during the initial test run. If some tests are failing before mutation, their result will be ignored. A possible cause is Stryker not running those tests in the proper context (e.g. incorrect working dir), leading to test files not being properly loaded or some other dependency failing

Conclusion: at this stage, either Stryker is not able to properly run the tests, leading to every mutant being a survivor, but it should appear in the log or some tests are failing, and this will appear in the log as well.

Note: this project looks familiar to me, is it some kind of sample project/app? I am merely asking to see if we could try and reproduce the issue.