Can nightly charts somehow separate test-only changes from actual Lucene code changes?

[mikemccand]

Nightly charts like the KNN indexing chart look depressing. But when you look closer, you realize that many of the big plummets in indexing throughput are due to benchy changes like switching to harder or higher dimensionality vectors, increasing topK, Lucene increasing default beamWidth, etc.

Could we somehow alter the chart to normalize away such changes?

They are similar to how financial charts normalize away stock splits, so you aren't visually deceived into thinking your stock actually dropped or jumped 50% on some days.

If we took such an approach, maybe with a checkbox to switch between "real" and "normalized", only the most recent data point would have trustworthy actual GB/hour, and all values from the past might be a normalized/relative value.

Maybe the annotations that are test-only could be a different color or font?

As we add annotations, we could set an additional optional boolean when it is a test only change. But it might be hard to categorize ... e.g. if Lucene changes its defaults, like recent TieredMergePolicy defaults changes, which would that be?

[xande]

Mike, I agree that the current nightly benchmark charts can be visually misleading when performance changes are due to benchmark methodology changes rather than actual regressions. Here are some ideas:

Enhanced Annotations: We could distinguish between code changes and benchmark methodology changes by using different colors or styling for annotations. This would make explanations for significant drops immediately visible.

Normalization Feature: As you suggested, a "normalized view" toggle that attempts to factor out known benchmark methodology changes. This would provide a more accurate visualization of actual performance trends over time, similar to how financial charts normalize for stock splits.

Categorization System: Develop a classification system for annotations where changes are tagged as either "benchmark methodology change" or "actual code change" to help with normalization.

Data Trimming: Provide a view that only shows periods with consistent benchmark methodologies, though this might significantly reduce the visible history.