Wrong subtests are shown in /import-maps/not-overridden/failed-resolution.html

[allstarschh]

For the WPT
https://github.com/web-platform-tests/wpt/blob/master/import-maps/not-overridden/failed-resolution.html

wpt.fyi shows only 1 subtest
https://wpt.fyi/results/import-maps/not-overridden/failed-resolution.html?label=experimental&label=master&aligned

According to the documentation
https://web-platform-tests.org/writing-tests/testharness-api.html#Test

 class Test(name)
    A single subtest. A Test is not constructed directly but via the test(), 

There should be 4 subtests in that file

1. Resolution after import map should not be redefined
2. Resolution after import map should not be redefined for bare prefixes or exact matches
3. Resolution after import map should be redefined for non-prefixes
4. Rules for failed resolution are not dropped

[gsnedders]

If we run a first test only after we're otherwise all done, then we terminate as soon as the first test completes:

wpt/resources/testharness.js

Lines 3875 to 3881 in 7e0fdfa

	Tests.prototype.all_done = function() {
	return (this.tests.length > 0 || this.remotes.length > 0) &&
	test_environment.all_loaded &&
	(this.num_pending === 0 || this.is_aborted) && !this.wait_for_finish &&
	!this.processing_callbacks &&
	!this.remotes.some(function(w) { return w.running; });
	};

We have no way of knowing that in the same async/deferred script execution more tests are going to be run — and deferring running test() would be a massive API breakage.

However, I wonder if we should simply disallow a first test to be created once we're already done, because that seems relatively likely to end up in this state?

We don't change the status of the harness when we see the further subtests, because we can't change the result once we're done:

wpt/resources/testharness.js

Lines 2759 to 2764 in 7e0fdfa

	// Tests declared following harness completion are likely an indication
	// of a programming error, but they cannot be reported
	// deterministically.
	if (tests.phase === tests.phases.COMPLETE) {
	return;
	}

Though we could potentially do better at surfacing it (via callbacks and via output in the default HTML results view), even if that is only informational.

On a test basis today, calling setup({explicit_done: true}) before the first test and done() at the end of the script would fix this.