Enable merge queue

[sarahchen6]

What Does This Do

Enable a merge queue, and include the following configurations:

• Merge using the squash method
• Allow skipping merge queue checks for master, but require a reason for skipping these checks
  • This configuration is included as an extra precaution in case we need to override the merge queue, but it should be avoided because it defeats the purpose of a merge queue and would skip any checks specific to merge queues
• Require the PR to have at least one label
• Require the PR's commits to be signed
• Require the PR to have all necessary reviews

Motivation

By enabling a merge queue, not only can we ensure that PRs pass CI with the latest version of master, including all PRs merged before it, but also we can run certain checks (e.g. benchmarks) only when merging to master. Merge queues by default run with the "speculative" workflow type. More details here, but TLDR; each PR needs to pass with the latest version of master + any PRs that sit before it in the merge queue before being merged.

This is a separate discussion from the merge queue itself, but as an example of what it can be used for -- benchmarks are the longest-running job on CI and currently run for every push. Certain PRs like documentation changes that do not warrant benchmarking runs or commits that are not yet ready to be benchmarked still need to wait for these jobs to pass in order for the CI run to complete. This delays feedback and can slow down the development cycle, even moreso if we want to increase the benchmark runtime (e.g. to improve stability). Instead we can run benchmarks when manually triggered and merging to master through a merge queue. The check-big-regressions PR-gate would prevent large regressions from being merged to master, but it would be the responsibility of the developer to run benchmarks when relevant prior to putting their PR in the merge queue.

Additional Notes

Contributor Checklist

• Format the title according the contribution guidelines
• Assign the type: and (comp: or inst:) labels in addition to any useful labels
• Don't use close, fix or any linking keywords when referencing an issue.
  Use solves instead, and assign the PR milestone to the issue
• Update the CODEOWNERS file on source file addition, move, or deletion
• Update the public documentation in case of new configuration flag or behavior

Jira ticket: [PROJ-IDENT]

[PerfectSlayer]

Instead we can run benchmarks when manually triggered and merging to master through a merge queue. The check-big-regressions PR-gate would prevent large regressions from being merged to master, but it would be the responsibility of the developer to run benchmarks when relevant prior to putting their PR in the merge queue.

Would there be a way for benchmarks to excluded on purpose rather than opt-in? I guess big regression would catch most of the issues, but I would think that casual offenders introduce regression because their don't know about it, so they would not opt into benchmarks in the first place?
That's a discussion we can have with the guild :)

[sarahchen6]

Would there be a way for benchmarks to excluded on purpose rather than opt-in?

@PerfectSlayer Yes we can also do that! Perhaps we could run a longer, more stable benchmark in the merge queue and still have some benchmarks run on every push 🤔

That's a discussion we can have with the guild :)

Agreed! I'll wait for a Guild discussion before proceeding any further