feat(loader-runner): Allow limiting worker pool size for parallel loaders #12277
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This is related to feat: support experimental loader parallelism ( #9807 ). Currently worker threads are spawned on demand and, by default, use all available CPU threads (effectively
cpus() - 1, fallback to1)Problem
Solution
the number of spawned workers, defaults to (number of cpus - 1) or fallback to 1 when require('os').cpus() is undefined.paralleloption in order to enable the multi-process parallel and lets you control how many concurrent workers are used.As recomended I extended the parallel setting to accept
booleann | { maxWorkers?: number }.maxWorkers. Then, we use that number as the requested thread count for Tinypool (applied tominThreads/maxThreads). Depends on the user to use a valid integer. It should be an integer > 0 (and ideally not larger than your CPU count).parallel: truedefaults to (number of cpus - 1) or fallback to 1 (as usual)This lets users tune Tinypool concurrency per-loader, helping avoid OOMs on constrained machines or very large builds. Any feedback, concerns, or alternative approaches welcome — I’m happy to adjust the implementation. Thanks in advance for the review!
Related links
Checklist