Add support for LazyTest?

[seancorfield]

Currently, Cognitect's test-runner supports any testing library that is compatible with clojure.test, such as Expectations.

LazyTest is a new testing library that is built from scratch to provide a more expressive style than either clojure.test or Expectations, and it addresses a lot of the criticisms that Sierra had about the implementation of clojure.test. However, since it doesn't use the clojure.test "runner" functions, tooling has to choose to explicitly add support for it.

I have a working fork of test-runner that relies on requiring-resolve (and therefore Clojure 1.10), to dynamically support LazyTest at runtime, if it is on the classpath. It matches most of the test-runner CLI options to equivalent ones in LazyTest, and will use LazyTest "runner" functions to run any LazyTest tests it finds, in addition to using the clojure.test "runner" functions to run any standard tests it finds.

It essentially makes two passes over the tests: the first for clojure.test tests, and the second for LazyTest tests.

[puredanger]

I think I would much rather add support not by hacking it in like this, but by reworking test-runner to insert a test framework abstraction, and then implement that for both clojure.test and lazytest. For things like output reporting, that could then be pluggable as well, so as to open up support for clojure.test output reporters (see #5).

Should also consider whether there are other frameworks to include. I know test.check has some bridges into clojure.test but maybe that's a good example of something that could be better integrated and more configurable - surfacing rep counts for example.

This is probably some (but not a lot?) work to figure out, but I would much rather do that than to merge these changes in as is. We could collab on that if you want to, or you could just wait for me to define the abstraction and create the lazytest implementation, I don't know what your urgency or availability is in that regard.

[seancorfield]

Thanks. Yes, I figured the current approach would be considered too "hacky" -- I'm not entirely happy with it either, if only because you end up with multiple passes over the nses: one for each test framework, and there's the issue of mapping options from the CLI to each framework (clojure.test has no built-in filtering so every test runner writes their own; LazyTest has all the filtering built-in but it doesn't quite match everything that test-runner offers). And, of course, each framework has different options for controlling output/formatting -- which also plays into the multiple passes aspect.

Everything is a lot simpler when you're only dealing with one test framework.

Since I'm now effectively maintaining two test runners that both support clojure.test-compatible stuff -- along with filtering logic in each -- and LazyTest stuff, I'll think about how this could all be streamlined.

However, an "ideal" world would see clojure.test evolve to support filtering directly and a way to separate reporting from test running, so that machinery didn't need to be in every test runner, and that would make clojure.test more similar to what some other test frameworks provide and potentially make it easier for test runners to support multiple frameworks?

[puredanger]

Also open to working on clojure.test stuff, I know the recent ask at https://clojure.atlassian.net/browse/CLJ-2882 is probably in this vein.

[seancorfield]

Yeah, some of the problems listed/referenced in that ticket were motivations for moving forward with LazyTest, I suspect.

I would be interested in collaborating on clojure.test but I'd really like to see it split out as a Contrib library that Clojure 1.13 could pull in a version of, like is currently done with Spec stuff.

[seancorfield]

I've been giving this some thought, and the common structure is something along these lines:

• create / initialize the test library (provide options etc)
• find test namespaces -- test-runner does this, based on nses / patterns options
• load test namespaces -- test-runner does this
• enable filtering -- for clojure.test, this is the existing logic to swap metadata based on filtering options; for lazytest this is a no-op
• run tests -- test library specific, with the options passed in (clojure.test filtering has happened by this point; lazytest uses options to do filtering
• disable filtering -- for clojure.test, this is the existing logic to swap back metadata; for lazytest this is a no-op

A protocol can be provided for those last three steps, and a fully-qualified symbol (FQS) for the create / initialize function would need to be provided to test-runner.

To handle alternative output formats, for clojure.test, those could be wrapped in the "run tests" implementation, and either be based off additional options passed in or by using a different FQS for create / initialize. I'd lean toward options, since lazytest already has an output option (that accepts a symbol) which needs to be passed to "run tests".

The test-runner would remain responsible for finding / filtering / loading namespaces.

The default FQS for clojure.test should become part of the public API so that it's straightforward to compose with 3rd party test libraries -- a "pure" lazytest runner could stub the filtering fns and just run lazytest tests; a combined runner could delegate to the existing filtering fns (for clojure.test) and then run both clojure.test tests and lazytest tests.

Thoughts?

[puredanger]

I think this is a good breakdown, I would be interested in seeing a structure like that. Curious if a 3rd data point (spec.test?, expectations?) would add or change anything. FQS stuff makes sense.

You are thinking that multiple test runners run and choose their own subsets? Or that just one runner runs in an execution?

[seancorfield]

The test-runner would find, filter, and load nses matching the current CLI options.

The protocol implementation would then do additional filtering, running of the actual tests, and restoring the state if needed.

I would expect a LazyTest implementation of the protocol to run both clojure.test and LazyTest tests, by constructing two implementations: the default one for clojure.test and its own for LazyTest, and then running both implementations -- allowing for migration between test styles.

Expectations is purely clojure.test-compatible so test-runner already handles that (via its clojure.test machinery).

I haven't looked at spec.test yet but that already has clojure.test compatibility and can be run as-is via test-runner, so the question there is whether a "better" runner could be devised -- I don't know yet but my instinct says that this proposal wouldn't make that any harder down the road and would, at minimum, enable a community effort to build such an implementation.

[seancorfield]

PR #49 submitted for feedback.