Macro generated tests not recognized and parent folder marked as "failed"

[NobbZ]

We have a quite huge testsuite, which generates several tests from given metadata.

The testfiles are marked with a "not ran" or "skipped" symbol (I guess, see screenshot) while the parent folder is marked as failed.

The testfiles do not contain much more than a use and a copule of module attributes, from which a test matrix is built and then the describe and test implementations are generated via macro magic.

Perhaps a mix test --trace can be used to get a more accurate list of tests or at least the parent is not marked as failed?

[jfpedroza]

I experimented with getting an accurate list of tests, but we need to compile the file for that. It's quite slow, at least compared to treesitter. In my work project, many test files took 1 sec or more (just the compile part). Try mix test --profile-require=time to see the times. It's usually not noticeable because ExUnit runs tests while compiling others, but you would notice if it's a single file. And it's not possible to create tests while running them, so the files would need to be compiled twice.

In my experiment, I was starting a VM per file so that contributed a lot to the slowness, but having a single VM to discover tests requires a lot more work. In the end, I didn't implement it, but it's possible.

As for the parent folder, the adapter only sets the status of each individual test, while Neotest proper is the one that calculates the status for files and folders. Probably better to change that part there.