Search performance issue with MMapDirectory under load #1151
Description
Is there an existing issue for this?
- I have searched the existing issues
Describe the bug
I have discovered that MMapDirectory has serious performance problems under parallel load when searching (with no background writes), across all platforms. The demo repo to reproduce this issue is at https://github.com/paulirwin/LuceneAdventureWorks/
This demo builds a simple index from the AdventureWorks2022 database data, and does a load test against it with 100k requests, using Parallel.For to search in parallel as fast as possible.
Of note: MMapDirectory is the default Directory implementation returned by FSDirectory.Open on Windows and Linux 64-bit platforms.
| OS | Architecture | Runtime | SimpleFS | NIOFS | MMap | Compared to SimpleFS |
|---|---|---|---|---|---|---|
| macOS 15 | arm64 | .NET 9.0 | 4.07s | 5.46s | 3m25s | 51x slower |
| Ubuntu 24.04 | x64 | .NET 9.0 | 2.21s | 2.43s | 35.57s | 16x slower |
| Windows 11 | x64 | .NET 9.0 | 4.13s | 11.66s | 2m33s | 37x slower |
| Windows 11 | x64 | .NET Framework 4.8.1 | 7.83s | 2m21s^ | 2m21s | 18x slower |
^ This was an outlier for NIOFS, and yes it was identical to MMap. Also note that hardware between these machines is not exactly comparable but all are 6-8 performance core machines with at either 32GB or 64GB RAM.
Given that MMapDirectory is the default Directory implementation returned by FSDirectory on Windows and Linux 64-bit (which is the majority of Windows and Linux use nowadays), we definitely need to resolve this, as MMapDirectory is supposed to be a fast implementation.
Expected Behavior
MMapDirectory performs well on all platforms.
Steps To Reproduce
See demo repo.
Exceptions (if any)
No response
Lucene.NET Version
4.8.0-beta00017
.NET Version
No response
Operating System
No response
Anything else?
No response