Support for Configurable Directories for Generated and Loaded Content in rclnodejs via Environment Variables for Monorepo Reuse

[uinz]

Currently, the rclnodejs project generates and stores its output within the node_modules/rclnodejs directory in the repository. This approach is limiting in monorepo projects, as the generated content is tightly coupled to a specific directory structure, making it challenging to reuse a single rclnodejs dependency across multiple sub-projects. This restricts its adaptability for diverse platforms or applications, particularly in monorepo scenarios where dependency sharing is critical.
To improve flexibility, enable reuse of a single rclnodejs instance in monorepo projects, and better accommodate the needs of different platforms and applications, I propose adding support for configurable directories for both generating and loading content via environment variables. This would allow users to specify custom directories while preserving the current behavior as the default.
Proposed Changes:

1. Configurable Output Directory for Generated Content via Environment Variables:
   • Add support for specifying the output directory for generated files through an environment variable.
   • This would allow users to define a custom location for generated content, facilitating organization and management in monorepo setups.
   • Default behavior: Continue generating content in node_modules/rclnodejs to maintain backward compatibility.
2. Configurable Loading Directory for Generated Content via Environment Variables:
   • Add support for specifying the directory from which generated content is loaded through an environment variable .
   • This would enable users to load content from a custom directory, supporting use cases where monorepo sub-projects need to source content from different locations.
   • Default behavior: Continue loading content from node_modules/rclnodejs to ensure existing functionality remains unchanged.
   Benefits:
   • Enhanced ability to reuse a single rclnodejs dependency across monorepo sub-projects.
   • Increased flexibility for projects targeting multiple platforms or applications.
   • Better organization of generated content for complex workflows.
   • Backward compatibility with the current directory structure to avoid breaking existing setups.
   Additional Notes:
   • Configuration should be implemented via environment variables for simplicity and cross-platform compatibility.
   • Documentation should be updated to guide users on setting these environment variables, particularly for monorepo scenarios.

export RCLNODEJS_GENERATED_DIR="./custom-dir"

[minggangw]

@uinz Thanks for your feedback, indeed, it's more flexible that the generated JavaScript files location can be configurable. I will take it later, thanks!

[minggangw]

Hi @uinz I did some investigation and found:

1. If putting the generated folder externally (out of rclnodejs\), you have to set NODE_PATH to tell node where it can load the dependency, like:

export NODE_PATH=/path/to/your_rclnodejs/node_modules

to load ref-napi correctly.

2. Meanwhile, the generated JavaScript messges needs the files from rosidl_gen\ and C++ bindings both, so the bindings package expects to find a package.json in the root of the module tree to resolve native bindings, which doesn't exist if you put the generated messages externally. For example:

If requiring /path/to/generated/std_msgs/std_msgs__msg__UInt8.js, a package.json is needed under /path/to/generated. So it's not easy to just change the location without changing anything else. Any ideas?

[uinz]

@minggangw

Yes, this is a bit tricky at the moment.

I don't think setting NODE_PATH is a good idea.

Do you think including index.js as part of the generated code is a good idea?

Think of the rclnodejs package as a base library + codegen tools.

---

Longer term, I hope rclnodejs can better support bundling (e.g., ncc/rolldown) to significantly reduce the size of the project's deb files (of course, this requires additional processing of the .node file).

This presents some difficulties with the current design based on path.join dynamic require.

[uinz]

Maybe like this, using package.json#exports to define the module

{
  // package.json
  "name": "rclnodejs",
  "description": "ROS2.0 JavaScript client with Node.js",
  "exports": {
    "./rclnodejs.node": "./build/Release/rclnodejs.node",
    "./parser": {
      "import":"./rosidl_parser/index.mjs",
      "require":"./rosidl_parser/index.cjs",
      "types":"./rosidl_parser/index.d.ts"
    }
    // ...
  }
}

[minggangw]

Looking through the nodejs doc, it says that exports is an alternative to the "main", not sure if it's feasible for our case, i can try it later, thanks for your suggetions!

[uinz]

@minggangw

To be more precise, I recommend looking at the relationship between protobufjs and protobufjs-cli

pb lib	rclnodejs lib	desc
protobufjs	rclnodejs	provides the runtime
protobufjs-cli	rclnodejs-cli	provides the generated code

The exports I mentioned are not simply a replacement for main, but a more static modular approach to replace require(path.join(__dirname, '...')

[uinz]

If this suggestion is feasible, I recommend abandoning the use of binding and defining .node files through exports.

Furthermore, the .node file processing model can refer to esbuild, pre-building content for different platforms and pulling in pre-compiled content through the install hook.

// esbuild#package.json
{
  "name": "esbuild",
  "version": "0.20.2",
  "description": "An extremely fast JavaScript and CSS bundler and minifier.",
  "repository": {
    "type": "git",
    "url": "git+https://github.com/evanw/esbuild.git"
  },
  "scripts": {
    "postinstall": "node install.js"
    // ^^^^^^^^^^^^^^^^^^^^
  },
  "main": "lib/main.js",
  "types": "lib/main.d.ts",
  "engines": {
    "node": ">=12"
  },
  "bin": {
    "esbuild": "bin/esbuild"
    // ^^^^^^^^^^^^^^^^^^^^
  },
  "optionalDependencies": {
    // ^^^^^^^^^^^^^^^^^^^^
    "@esbuild/aix-ppc64": "0.20.2",
    "@esbuild/android-arm": "0.20.2",
    "@esbuild/android-arm64": "0.20.2",
    "@esbuild/android-x64": "0.20.2",
    "@esbuild/darwin-arm64": "0.20.2",
    "@esbuild/darwin-x64": "0.20.2",
    "@esbuild/freebsd-arm64": "0.20.2",
    "@esbuild/freebsd-x64": "0.20.2",
    "@esbuild/linux-arm": "0.20.2",
    "@esbuild/linux-arm64": "0.20.2",
    "@esbuild/linux-ia32": "0.20.2",
    "@esbuild/linux-loong64": "0.20.2",
    "@esbuild/linux-mips64el": "0.20.2",
    "@esbuild/linux-ppc64": "0.20.2",
    "@esbuild/linux-riscv64": "0.20.2",
    "@esbuild/linux-s390x": "0.20.2",
    "@esbuild/linux-x64": "0.20.2",
    "@esbuild/netbsd-x64": "0.20.2",
    "@esbuild/openbsd-x64": "0.20.2",
    "@esbuild/sunos-x64": "0.20.2",
    "@esbuild/win32-arm64": "0.20.2",
    "@esbuild/win32-ia32": "0.20.2",
    "@esbuild/win32-x64": "0.20.2"
  },
  "license": "MIT"
}

[minggangw]

Hi @uinz thanks for your deep investigation, I submitted a draft PR #1223 that implements reading/writing from/to GENERATED_MSG_PATH if set, do you have time to try with what you recommend becasue i believe you have a better understanding than me, appericate it! Talking about the prebuild binary, actually #865 is just for it :)

[uinz]

Hi @uinz thanks for your deep investigation, I submitted a draft PR #1223 that implements reading/writing from/to GENERATED_MSG_PATH if set, do you have time to try with what you recommend becasue i believe you have a better understanding than me, appericate it! Talking about the prebuild binary, actually #865 is just for it :)

Thank you very much, I will try it later ;)

[uinz]

Sorry, I've been busy with work recently and don't have time to realize my idea.

[minggangw]

Never mind, please take your time and you could join anytime :)