Segment Traces Graph Fails to Visualize Indexing Stalls During High Merge Activity

[nipunbatra8]

Problem Description

When multiple merge operations occur simultaneously, indexing stalls are observed (as evidenced in infostream logs). However, the segment traces graph does not adequately visualize these stalls, making it difficult to diagnose performance issues.

Current Behavior

• Infostream logs show clear messages about indexing stalls during high merge activity
• Segment traces graph shows merge operations but doesn't highlight or indicate the resulting indexing stalls

Expected Behavior

• Segment traces graph should clearly indicate when indexing stalls occur
• Visual correlation between merge operations and indexing stalls should be apparent
• Ideally, the graph would show a warning or different color when indexing stalls are detected

Additional Information

• This issue is particularly problematic when diagnosing performance in high-throughput environments and when/where indexing is stalling

Possible Solutions

• Add a new visualization layer to the segment traces graph that shows indexing stall periods
• Implement visual indicators (color changes, warning icons) when stalls are detected

Steps to Reproduce

1. Create an environment with high indexing throughput (e.g. NRTPerfTest)
2. Trigger multiple merge operations simultaneously
3. Observe infostream logs showing indexing stalls
4. Compare with segment traces graph visualization

In this aggregates chart we can see indexing stalls around 17:07 to 17:08 while merging catches up.

And in the infostream logs we see:

MS 0 [2025-08-05T17:08:09.891095001Z; Index #0]: too many merges; stalling...

MS 0 [2025-08-05T17:08:30.718250666Z; Index #0]: stalled for 20827 ms MS 0 [2025-08-05T17:08:30.718491803Z; Index #0]: no more merges pending; now return MS 0 [2025-08-05T17:08:30.718526844Z; Index #8]: stalled for 95683 ms MS 0 [2025-08-05T17:08:30.718586996Z; Index #8]: no more merges pending; now return MS 0 [2025-08-05T17:08:30.718637011Z; Index #15]: stalled for 109869 ms MS 0 [2025-08-05T17:08:30.718693439Z; Index #15]: no more merges pending; now return MS 0 [2025-08-05T17:08:30.718734473Z; Index #14]: stalled for 67663 ms MS 0 [2025-08-05T17:08:30.718786621Z; Index #14]: no more merges pending; now return MS 0 [2025-08-05T17:08:30.718832174Z; Index #13]: stalled for 102739 ms MS 0 [2025-08-05T17:08:30.718886612Z; Index #13]: no more merges pending; now return MS 0 [2025-08-05T17:08:30.718927964Z; Index #11]: stalled for 99563 ms MS 0 [2025-08-05T17:08:30.718978590Z; Index #11]: no more merges pending; now return MS 0 [2025-08-05T17:08:30.719015795Z; Index #3]: stalled for 59151 ms MS 0 [2025-08-05T17:08:30.719069728Z; Index #3]: no more merges pending; now return MS 0 [2025-08-05T17:08:30.719113028Z; ReopenThread]: stalled for 111227 ms MS 0 [2025-08-05T17:08:30.719168959Z; ReopenThread]: no more merges pending; now return MS 0 [2025-08-05T17:08:30.719211242Z; Index #9]: stalled for 36180 ms

However, no clear indication in the segment traces graph of the indexing stalls

[mikemccand]

+1 this is really important! Else we are flying blind when we look at segment traces that we might otherwise feel tempted to interpret as healthy!

Merge stalling is an act of desperation on Lucene's part, when merging falls too far behind, to apply backpressure to the indexing threads producing all the segments in the first place. CMS hijacks the incoming segment-producers and forcefully stalls them (wait before returning control back to caller).

Maybe it could render in the UI as another vertical band of color (like flush, and merge-on-commit), some horrible scary color. And we'd need to fix infostream_to_segments to record these start/stop stall events...