Skip to content

Conversation

@Pablinho
Copy link
Contributor

@Pablinho Pablinho commented Nov 24, 2025

Summary

This is related to feat: support experimental loader parallelism ( #9807 ). Currently worker threads are spawned on demand and, by default, use all available CPU threads (effectively cpus() - 1, fallback to 1)

Problem

  • For very large projects that build multiple targets (for example, browser and node in dev mode), enabling parallel loaders (for project still using babel-loader for example) can cause OOMs or build failures with little diagnostic output. I’ve seen high CPU usage just before the OOM. The root cause appears to be the total number of worker threads tinypool is using.

Solution

  • Tools like thread-loader already let you control worker count via a worker option: the number of spawned workers, defaults to (number of cpus - 1) or fallback to 1 when require('os').cpus() is undefined.
  • Similarly, the terser-webpack-plugin provides a numberic parallel option in order to enable the multi-process parallel and lets you control how many concurrent workers are used.

As recomended I extended the parallel setting to accept booleann | { maxWorkers?: number }.

  • If parallel is an object, user can set maxWorkers. Then, we use that number as the requested thread count for Tinypool (applied to minThreads/maxThreads). Depends on the user to use a valid integer. It should be an integer > 0 (and ideally not larger than your CPU count).
  • If parallel: true defaults to (number of cpus - 1) or fallback to 1 (as usual)

This lets users tune Tinypool concurrency per-loader, helping avoid OOMs on constrained machines or very large builds. Any feedback, concerns, or alternative approaches welcome — I’m happy to adjust the implementation. Thanks in advance for the review!

Related links

Checklist

  • Tests updated (or not required).
  • Documentation updated (or not required).

@Pablinho Pablinho requested a review from hardfist as a code owner November 24, 2025 20:08
@netlify
Copy link

netlify bot commented Nov 24, 2025

Deploy Preview for rspack ready!

Built without sensitive environment variables

Name Link
🔨 Latest commit e209af4
🔍 Latest deploy log https://app.netlify.com/projects/rspack/deploys/6928b59dfafab000081269c6
😎 Deploy Preview https://deploy-preview-12277--rspack.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@github-actions github-actions bot added the release: feature release: feature related release(mr only) label Nov 24, 2025
@CLAassistant
Copy link

CLAassistant commented Nov 24, 2025

CLA assistant check
All committers have signed the CLA.

@codspeed-hq
Copy link

codspeed-hq bot commented Nov 26, 2025

CodSpeed Performance Report

Merging #12277 will not alter performance

Comparing Pablinho:make-tinypool-number-threads-configurable (e209af4) with main (e2b3889)

Summary

✅ 17 untouched

@Pablinho Pablinho changed the title feat(loader-runner): Make tinypool thread count configurable feat(loader-runner): Make parallel loader thread count configurable Nov 27, 2025
@Pablinho Pablinho changed the title feat(loader-runner): Make parallel loader thread count configurable feat(loader-runner): Allow limiting worker pool size for parallel loaders Nov 27, 2025
Copy link
Member

@chenjiahan chenjiahan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thank you!

@chenjiahan chenjiahan enabled auto-merge (squash) November 28, 2025 05:58
@chenjiahan chenjiahan merged commit b78207d into web-infra-dev:main Nov 28, 2025
53 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release: feature release: feature related release(mr only)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants