ESLint multi-thread linting creates new chanllenge to `eslint-plugin-import` and `eslint-plugin-import-x`

[SukkaW]

cc @nzakas @mdjermanovic @ljharb @JounQin

Context: eslint/eslint#19794

TL; DR With multi-thread ESLint workers, the existing cache mechanism for one of the biggest performance overheads of both eslint-plugin-import and eslint-plugin-import-x will now be isolated, which will result in those slow jobs being done repeatedly in different workers.

---

One of the biggest performance overheads of both eslint-plugin-import and eslint-plugin-import-x is that they will look for the ImportDeclaration (and ImportExpression), using user-specified eslint-import-resolver-* packages to resolve the real location of the importee, read and parse (grabbing a parser from the rule context, or use the user-specified one) it, then walk through that AST to get its ImportDeclaration, ImportExpression, Export*Specifier, etc.. These all happen in a utility class called ExportMap (eslint-plugin-import's implementation, eslint-plugin-import-x's forked implementation). The results (which are ExportMap instance objects) are cached by a Map object at the module scope.

In the worst-case scenario, let's say a.js and b.js, picked up by different ESLint workers, both import the same node_modules/huge-barrel-library/entrypoint.js, the node_modules/huge-barrel-library/entrypoint.js will be parsed and processed twice in different workers (previously, with only one thread, it would be cached and reused).

We would all agree that ESLint should continue to be multi-threaded; however, this is not something we should overlook.

[ljharb]

Indeed; I've repeatedly asked for a "pre-lint" phase for awhile, that would run once instead of N times, and would be shown in TIMING separately.

[nzakas]

Please provide some benchmarks so we can see what kind of a problem we're dealing with.

[SukkaW]

@nzakas I will try to create some example projects to demonstrate this.

---

I just found this library sock-daemon made by @isaacs that can automatically spin up (and kill) a "server" when needed. It is possible to make the entire ExportMap (parser, analyzer, etc.) live inside the server, and the rule will call methods (e.g., client.getExports(modulePath, ruleContext, options), client.getReExports, etc.).

Note that the sock-daemon library can only be invoked asynchronously (by its nature), so we need to wrap those methods in synckit to get synchronous versions.