GitHub Apps: How to do acceptance testing

For testing GitHub apps, we've created a private repository called github-app-testing in the @coderly account.

The github-app-testing repo contains stub issues, pull requests, and comments that allow us to do acceptance testing. The repo is private solely because we assert certain things in tests that could break were the repository public.

In your .env.example you should see a number of GITHUB_TEST_ environment variables. If you're a volunteer, you won't be able to run these acceptance tests locally or in CircleCI within your fork. We unfortunately cannot expose the PEM file or secret keys.

These acceptance tests will run on Circle by specifying to include tasks with the tag @acceptance: true.

If you're not a volunteer, you can run these acceptance tests locally with mix test.acceptance. You'll need to acquire the credentials from the Code Corps Testing GitHub App, Base64 encode the PEM file, and place all the credentials in your .env. These credentials will also need to be available on Circle for the tests to run properly. If you're seeing acceptance tests fail, it's likely that the data has changed in the stub github-app-testing repository or you're missing credentials.

Keep in mind that these acceptance requests do make HTTP requests. These requests should not be mocked or stubbed as it's testing full acceptance of our integration with the GitHub API.