Supporting tags when running `dart path/to/my_package/test/my_test.dart`

[dcharkes]

In https://github.com/dart-lang/native, we use package:test. But these tests also roll into the Dart SDK and run there with the Dart SDKs test.py.

We can only run a subset of tests on the Dart SDK CI, due to having no cross compilation tools available. We run the full set of tests on the GitHub CI. The compilers on the GitHub CI and Dart SDK CI are not the same, so we want to run in both places for coverage.

Now, we've been managing the set of tests that runs on the Dart CI via status files in the Dart CI, but this requires updating those files on rolls.

Instead, we'd like to use tags. So we'd tag all our cross compilation tests with cross-compilation for example.

However, since test.py is not package:tests bin/ runner, there's no way to pass --exclude-tags.

An idea on how to make this work:

• Put the exclude tags in environment variables. And one of the the following:
  • make test/group read those env variables if not run via package:test
  • enable setupAll to set tags, so that all the test and group calls after it in the file respect those tags instead
  • add a new configureTestRunner(...) method that can set all the variables that would be usually set via the package:test command-line arguments.

Doing something in this direction would also enable configuring what to run in a test file when running it from source in the debugger.

@natebosch @jakemac53 Any thoughts or ideas? Is there already a way to do this?

cc @goderbauer

[natebosch]

• Put the exclude tags in environment variables. And one of the the following:

Do you mean that we'd handle only the tags passed to test and group and not handle the @Tag metadata? Handling the annotations is infeasible.

Handling the tags passed at runtime maybe be feasible with the directRunTests API (requires a /src/ import currently - may see breaking changes, although it hasn't been touched in a while). #1332

As long as you don't need the test to be compatible with both the direct run and test-runner runtimes we can look at what it will take to add support for restricting by tag. If tag behavior relies on code in test or test_core we'll need to also check the feasibility for migrating it to test_api but I wouldn't expect that to be a problem here.

However, since test.py is not package:tests bin/ runner, there's no way to pass --exclude-tags.

Why can't test.py invoke the package:test runner?

[dcharkes]

Why can't test.py invoke the package:test runner?

That's not how test.py is set up. It scan itself for files ending in _test.dart and reports it to the Dart SDK test status database. I don't think test.py can infer whether a file ending _test.dart should be run with package:test or not. There are lots of tests in the Dart SDK that don't use package:test as a dependency. These tests just rely on the exit code being zero or not.

Do you mean that we'd handle only the tags passed to test and group and not handle the @Tag metadata? Handling the annotations is infeasible.

Yes, this is already the pattern we use for OSes. The @TestOn('mac-os') isn't interpreted by test.py, so we end up also having a Platform.isMacOS check: https://github.com/dart-lang/native/blob/9d68fa4a8b71fc5946be4d604a6b6c766d5d5dc4/pkgs/native_toolchain_c/test/clinker/objects_cross_ios_test.dart#L16-L20

Handling the tags passed at runtime maybe be feasible with the directRunTests API (requires a /src/ import currently - may see breaking changes, although it hasn't been touched in a while).

I was thinking the other way around. I don't want to be spawning the test runner, I want to run the test file as a standalone script. But be able add some code in there that would configure the test runner include/exclude tags:

void main() {
  final testContext = Declarer.current;
  if (!testContext.isTestRunner) {
    testContext.excludeTags = [
      if (Platform.environment['EXCLUDE_TAG_MAC_OS'] != null) 'mac-os',
    ];
  }

  test('test name', tags: ['mac-os'], () {
    // ...
  });
}

We can probably come up with a much better API to do so. This is just to illustrate the idea.

[jakemac53]

Are there packages which have a mixture of _test.dart files both using and not using package:test?

If not, I would just have some way for the test.py to support using the test runner for the package that do use package:test, and then not use it for the ones that are doing their own thing (like language tests).

I would be pretty surprised if packages are mixing and matching both things, and even if there are a few it could probably be cleaned up without too much work.

[dcharkes]

Are there packages which have a mixture of _test.dart files both using and not using package:test?

Probably not. I guess this can go on the list of our next engprod fixit then. 😅

I guess I was more thinking along the lines of #1310, I very often run test files in isolation just as a dart script. Especially when I have to debug a test. So having more features when test scripts are run by themselves seems like a good design goal to me.

Maybe we should close this as a duplicate issue of:

• Add an API to do some customization and then directly run tests in process #1310

[jakemac53]

Fwiw there really isn't much of a reason to run tests directly, we support it because people expect it, but you can always do dart test <script> instead of dart <script> or dart run <script> and get essentially the same behavior but with support for all features (and specifically annotations).

[dcharkes]

Fwiw there really isn't much of a reason to run tests directly, we support it because people expect it, but you can always do dart test <script> instead of dart <script> or dart run <script> and get essentially the same behavior but with support for all features (and specifically annotations).

How do you run the debugger on your test with dart test <script>? It's not the same at all? Or am I missing something?

[jakemac53]

I just use the debug configuration in my IDE, but the test runner does have a flag --pause-after-load and that should enable debugging

[dcharkes]

I just use the debug configuration in my IDE, but the test runner does have a flag --pause-after-load and that should enable debugging

Maybe I'm holding it wrong: I'm using a launch configuration for the the test script in vscode:

	"launch": {
		"version": "0.2.0",
		"configurations": [
			{
				"name": "Dart",
				"type": "dart",
				"request": "launch",
				"program": "path/to/my/cool_test.dart",
			}
		],
		"compounds": []
	}

There is a debug button above the test in the IDE as well, but that requires you to navigate to the file and scroll to the test to start it.

[natebosch]

I want to run the test file as a standalone script

That's what directRunTests is for.

[jakemac53]

There is a debug button above the test in the IDE as well, but that requires you to navigate to the file and scroll to the test to start it.

This is typically what I am doing for debugging because I am trying to debug a specific test anyways - I use the run button directly above the test I want to run.

I don't actually ever set up custom launch configurations really, but that looks like a configuration for just running a dart app - you might want to use a more generic configuration that just runs any command, so you can have it run dart test <path>. You might not even need to pass --pause-after-load in that case breakpoints set in your IDE would probably work.

[DanTup]

Maybe I'm holding it wrong: I'm using a launch configuration for the the test script in vscode:

Using this config in VS Code, the Dart-Code extension should decide whether to run the script using dart run or dart test. The logic for this is here:

https://github.com/Dart-Code/Dart-Code/blob/e7f00597ba36e060734d786cc71450bfe71735c1/src/extension/utils.ts#L98

By default, anything inside a test folder that ends with _test.dart is considered a test, and should run through dart test (and you can use toolArgs or args if you want to pass additional args). If your tests are outside of a test/integration_test folder, you'd need to set "dart.allowTestsOutsideTestFolder": true in the config for them to be treated as tests.

(Just pressing F5 inside a test file should do the same thing - you probably only need a launch config at all if you want to be able to start debugging the test file without it being the current active file)