Report which tests failed on new PRs

[alexstyl]

This is a follow up work to #137.

We now have unit tests running when new PRs are opening.

However, right now there is no information displayed about which tests are failing. See #140 for an example of failing PR.

The goal of this task is to:

• investigate how we can display information about which tests failed, so we give context to the contributor about what went wrong.
• implement the solution by updating the pr_tests workflow.

PS: When running the tests locally (using ./gradlew jvmTest), there is an test report generated. Can we use that somehow?

[alexstyl]

cc @Uma1r as you might be interested in this one

[Uma1r]

I tried doing this here, take a look at this PR I created for testing: Uma1r#6

I didn't know much about this before yesterday :D so there are a couple of nuances here (and a bug in existing workflow)

When PR's are coming from forks then the automatic comment thing and publishing of test results within comments doesn't work because the forked PR has read-only github token access only, yesterday we only tested same-repo PRs.

When a workflow runs on a pull request from a fork, GitHub limits the GITHUB_TOKEN to read only permissions for security reasons (more info on this here). This means the workflow inside that PR cannot:

• Post comments on the PR
• Create or update Check runs
• Or modify PR state (e.g. close, label, or reopen)

Because of this restriction, actions like publishing unit test results directly to the PR (via EnricoMi/publish-unit-test-result-action) don’t work on forked PRs

---

But there is a way to handle this though, we'll have to use two workflows:

• The main CI workflow (pr_checks) runs tests and uploads the results as artifacts.

• A separate (e.g. "Publish Test Results") workflow (running in the base repo) downloads those artifacts and publishes the test results as a comment with full permissions.

Another thing, I also recommend against auto-closing PR's now, if a PR is closed by a bot (or yourself) in the base repo then our current message conveys the user to test locally, push fixes and re-open this PR, this unfortunately is not possible as the forked PR author doesn't have the rights to re-open a closed PR in the base repo.

p.s. let me know if what I have here is what you want then I'll create a PR.

[alexstyl]

Thanks for the investigation @Uma1r

I had a closer look at Github actions and it seems like it's clear when a check fails.

Could you kindly remove the auto-closing of the PR (as your suggested) and remove any messages sent by the CI to the PR?

From the looks of it, the checks that fail are clear on the PR itself when you don't close it and it should be clear enough to show which test fails when you click on the check.

PS: I have 'assigned' it to you since you are looking into it, so that others looking at the Issues board know that task is already taken. No rush on finishing it or anything (it's the weekend anyway)

[alexstyl]

Thanks again @Uma1r for the contribution. Now that the PR is merged it's good to close this one.

Nice one.