Self-tests for Custom Tests #162
Description
This issue is to keep track of what the self-tests of the Custom Tests covers, and to keep track of what scenarios we still need to cover.
Current Test coverage
Environment
| Behavior | Covered by Tests |
|---|---|
| Env up with default settings | ✔️ |
| Env up with PHP and WP versions passed as parameters | ✔️ |
| Env up with PHP and WP versions in config file | ✔️ |
| Env up with PHP and WP versions in config file overridden by parameters | ✔️ |
| Env up with PHP and WP versions in config file overridden by config file override | ❌ |
| Env up with Object Cache | ✔️ |
| Env up with Additional Volumes | ✔️ |
Env up with WordPress stable version |
✔️ |
Env up with WordPress rc version |
❌ |
Env up with WordPress WPOrg tag version |
✔️ |
Env up with WooCommerce stable version |
✔️ |
Env up with WooCommerce rc version |
✔️ |
Env up with WooCommerce git tag version |
❌ |
| Env up with additional PHP Extensions | ✔️ |
| Env up with additional theme(s) | ✔️ |
Tagging
| Behavior | Covered by Tests |
|---|---|
| Upload test tag | ✔️ |
| List test tags | ✔️ |
| List test tags of a specific extension | ✔️ |
| Delete test tag | ✔️ |
| Upload, delete and list test tags of extension you don't own | ❌ |
E2E Test
| Behavior | Covered by Tests |
|---|---|
| Scaffold a basic E2E test | ✔️ |
| Run a scaffolded E2E test locally | ✔️ |
| Tag a scaffolded test and run it from the tag | ✔️ |
| Run a test and assert it fails if SUT failed to activate | ✔️ |
| Run an E2E test with multiple tags | ✔️ |
| Run an E2E test with theme as SUT | ✔️ |
| Run an E2E test with plugin as SUT and theme with test or bootstrap actions | ❌ |
| Run an E2E test with snapshot test for visual regression | ✔️ |
Compatibility Test
| Behavior | Covered by Tests |
|---|---|
| Run an E2E test and activate additional plugin sourced from slug | ✔️ |
| Run an E2E test with bootstrap additional plugin sourced from slug | ✔️ |
| Run an E2E test with testing additional plugin sourced from slug | ✔️ |
| Run an E2E test with multiple tags and additional plugin | ✔️ |
| Run an E2E test with same additional plugin having multiple tags | ✔️ |
| Run a scaffolded E2E test with additional plugin sourced from local directory/zip file/URL (activate/bootstrap/test actions), both from runtime parameters, config files and config file override | ❌ |
| Run a test with multiple plugins, some of them with tests, some with local directory and others from tags | ❌ |