Extend test case filtering

[tlemo]

This PR implements the suggestion described in ggml-org/ggml#1308 (originally created against the GGML repo as ggml-org/ggml#1309)

1. Allow passing multiple (comma-separated?) ops to test-backend-ops. This can be convenient when working on a set of ops, when you'd want to test them together (but without having to run every single op). For example:

test-backend-ops.exe test -o "ADD,RMS_NORM,ROPE,SILU,SOFT_MAX"

2. Support full test-case variation string in addition to basic op names. This would make it easy to select a single variation, either for testing or for benchmarking. It can be particularly useful for profiling a particular variation (ex. a CUDA kernel), for example:

test-backend-ops.exe perf -b CUDA0 -o "MUL_MAT(type_a=f16,type_b=f32,m=4096,n=512,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=2)"

These two can be combined. As the current -o, this change doesn't try to detect/report an error if an filter doesn't name existing ops (ex. misspelled)

[tlemo]

Ideally, this method could be made const. That would require making test_case::op_desc() and test_case::vars() const as well, which is a trivial, but noisy change.

If reviewers agree with that, I'm happy to make the additional change.

[CISC]

Probably not worth it.

[slaren]

It would be a welcome change in a separate PR.

[Copilot]

Pull Request Overview

This PR extends the test case filtering functionality in the backend operations testing framework to support comma-separated operation lists and full test case variation strings. The enhancement allows developers to test specific groups of operations or individual test variations without running the entire test suite.

• Added matches_filter method to parse comma-separated operation filters and full test case strings
• Updated all evaluation methods to use the new filtering mechanism instead of simple string comparison
• Modified command-line interface to accept multiple operations and full test case specifications

[CISC]

Can be a follow-up PR, but regex-matching would be nice.

[am17an]

-p already provides an option to filter ops by regex. The comma separated list of ops is the only thing missing at the moment, the exact param stuff can already be done via regex anyway (although you need to escape the special chars)

[CISC]

-p already provides an option to filter ops by regex. The comma separated list of ops is the only thing missing at the moment, the exact param stuff can already be done via regex anyway (although you need to escape the special chars)

Ah, yes, it was mainly the params I was thinking of, thanks for the reminder. :)

[CISC]

So, perhaps being able to specify params in the ops list is a bit superfluous.

[tlemo]

So, perhaps being able to specify params in the ops list is a bit superfluous.

Thanks @am17an , @CISC. I discovered -p too as I made this change, and I agree that it makes the passing of the full variation name to -o less important. The reason I kept it, is that 1. passing regex patterns on the command line can be a bit less friendly (requiring shell escapes) and 2. it's convenient to support multiple (comma-separated) variations, something which -p doesn't support. The first one (requiring careful escapes, is also the reason I decided against regex support in the new matches_filter().

Either way, I can remove the support for full variation name if that's what the consensus is, but it doesn't simplify the code much, and we'd lose the ability to simply copy & paste the variation name into -o.

[CISC]

Either way, I can remove the support for full variation name if that's what the consensus is, but it doesn't simplify the code much, and we'd lose the ability to simply copy & paste the variation name into -o.

As you say, it has its use and removing it doesn't do much, let's keep it.

[CISC]

passing regex patterns on the command line can be a bit less friendly (requiring shell escapes)

BTW, in case you didn't know, it's super helpful to use single tick quotation marks for f.ex. regex strings as it will then be interpreted literally, no escaping necessary.

[tlemo]

I see one CI failure, although it doesn't appear related to my changes:

The following tests FAILED:
	 28 - test-arg-parser (Subprocess aborted)              main
Error: Process completed with exit code 8.

[tlemo]

passing regex patterns on the command line can be a bit less friendly (requiring shell escapes)

BTW, in case you didn't know, it's super helpful to use single tick quotation marks for f.ex. regex strings as it will then be interpreted literally, no escaping necessary.

Thanks. Most, but not all, shells have some "raw" quotation indeed, but the escaping is required not just for the shell but for the regex syntax. For example, you won't be able to use use "type_a=f16,type_b=f32,m=4096,n=512,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0" as a regex since the square brackets have a special regex meaning.