Benchmark System

[joocer]

To help make assessments of the effectiveness of changes to parts of the engine, a benchmark tool should be written.

This should take two branches/tags from GitHub, install Opteryx from the branch, and then execute like queries multiple times and report on the performance characteristics.

There should also be a variation that just runs one branch hundreds of times with different feature flags set.

The outputs should be detailed logs of :

• the normalised query (spaces removed etc)
• the AST
• the initial plan
• the optimized plan
• the execution statistics
• a hash of the result (to confirm the outputs are identical)

Once benchmarked, there should be a requirement to maintain a minimum performance and any breaches of this should be noted in the change log.