"Something like haxeshim"

[Simn]

We discuss this at every meetup but never opened an issue about it, so let's do that. The idea is to have something official that works similar to haxeshim. From what I gather, there are two main tasks for such a tool:

• deal with dependencies: figure out the dependencies, e.g. from -lib arguments, and prepare them for the actual compiler to consume
• allow self-updating the compiler: have a separate executable so that the actual compiler executable can easily self-update

I don't know much more about haxeshim, so I'll need some input here from various people: @back2dos @nadako @Gama11

I'm willing to work on this, but I'll need a clear understanding of what exactly we want to do.

[Gama11]

I think you would have to replace all hardcoded haxelib invocations in the compiler - that's not just haxelib path, but also haxelib run hxcpp/hxcs/hxjava. IIRC @back2dos suggested something like haxe -lib hxcpp --run Main could work?

[nadako]

I think these two things can be approached separately:

• locking down compiler version. this should be handled by some frontend executable that reads some file (e.g. `.haxerc) and determines which actual haxe executable to run, as well as setting up necessary environment for it

• locking down library versions. this can be either handled by the same tool (i.e. resolve -lib into -cp+-D and pass it to the real compiler like haxeshim does), or it can be done inside the compiler itself (we can already just look for haxe_libraries or whatever and use it instead of the current haxelib path calling).

these are the things that require work in the compiler itself, however the end user also needs a tool that downloads and places things in the correct places. so either lix should become an official tool, or haxelib should provide the same functionality. or both.

also we need to design some format for locking versions. the current haxeshim approach (.haxerc+haxe_libraries) works fine, but I'm still not sure if we should instead just have a single haxe.json that contains all the info (also potentially more things for the future, e.g. project-specific define descriptions and whatnot)

[RealyUniqueName]

I think we should go another way:

• compiler should not know about libraries and (probably) hxml
• have a separate tool, which gathers classpaths from libraries, handles hxml, composes a final list of compiler arguments and then invokes the compiler. This tool could also become a replacement for haxelib at some point.

[ncannasse]

Could we start by listing the actual problems to solve before discussing the actual solutions ? we want to make sure solutions are fixing actual problems ;)

[ncannasse]

Problem P1 : being able to install side by side several haxe compiler versions and be able to specifiy at compile time which version to use.

[kevinresol]

• The compiler should be self-sufficient. i.e. Remove haxelib dependency (hence neko)
• Lock down library versions per project

[back2dos]

Could we start by listing the actual problems to solve before discussing the actual solutions ?

Fully reproducible builds. That is the main problem. The rest is icing on the cake.

[ncannasse]

For fully reproducible builds we need to be able to :

• P1 select compiler version
• P2 pin the haxelib libraries

P1 can be handled with haxelib : if the haxe compiler is a library, then we can resolve its path, install and publish versions, etc. It then only requires a parameter for the "shim" to select the compiler.

P2 can already be acheived by explicitely setting the version in hxml : -lib name:version, but that's not very practical as one will often forget about it, and you might have several hxml that would require to be synchronized wrt version upgrade.

So for P2 it would require something similar to npm package-lock.json that store the installed version for a set of libraries (+ dependencies). haxelib-lock.json would be read/written by haxelib. No need to have the libs in local repository for that, that's a different problem.

Do I forget anything ?

[nadako]

That's more or less what haxeshim does: it's a "frontend" that runs a correct haxe binary with correct env vars based on selected version. and the haxe_libraries folder serves the same purpose as the "haxelib-lock.json" - it contains full information about libraries: some metadata for installing this exact version and compiler arguments (so the compiler binary don't have to depend on haxelib and run haxelib path itself).

While it would be awesome to have compiler as a haxe library, this would require more thought and work because: a) we need a way to install binaries for different architectures and b) we want an easy way to install nightlies. And I don't see anyone working on haxelib anytime soon. That said, this is not a blocker for implementing the haxeshim-like thing, it doesn't care how libraries/compilers are installed, it simply looks for them in the folders specified by the convention.

Anyway, we can discuss the actual format of the lock files and how to publish haxe as haxelib and whatnot, but in any case we need to implement this "frontend" executable that runs the specified version. I would (again) suggest we start with this first and continue thinking about libraries after that.

[grepsuzette]

Also we need to design some format for locking versions. the current haxeshim approach (.haxerc+haxe_libraries) works fine, but I'm still not sure if we should instead just have a single haxe.json that contains all the info (also potentially more things for the future, e.g. project-specific define descriptions and whatnot)

@nadako Having dependencies as mere files e.g. haxe_libraries/hx-yaml.hxml (as opposed to a big file with all dependencies) is very convenient when you want to copy one from another related project using the terminal, as opposed to having to: edit a file, find the line, ignore all surrounding information, do whatever you want, save, close. (With a locate haxe_libraries/hx-yaml.hxml you could even list all projects having that lib as a dependency). I of course see things from my end, but there are clear advantages with this approach, not even talking about how since it would use json, a cat haxe.json would be hard to read compared to ls haxe_libraries, yet it's something that one would have to use very often.