Generate plugin spec

[bentsherman]

This PR adds a buildSpec task which generates a plugin spec using a custom entrypoint in the Nextflow runtime

It is a Java task that loads the Nextflow runtime (requires 25.08.0-edge or later) and the plugin that was just built, so that it can load the plugin extension points and generate the index file using the PluginSpecWriter class in the Nextflow runtime.

Depends on nextflow-io/nextflow#6361

[pditommaso]

@bentsherman Let's aim moving this to "Ready"

[bentsherman]

I am testing the entire flow with the registry locally. Before we merge this PR, we need to:

1. Finalize and merge https://github.com/seqeralabs/plugin-registry/issues/171
2. Finalize and merge Generate plugin spec nextflow#6361 (I need to review the schema structure)
3. Release Nextflow 25.08.0-edge

[pditommaso]

You should be able to test everything in the local dev. Check this for using unreleased plugin https://github.com/nextflow-io/nextflow/blob/a5c19b895a5d64ef4ae76907b5f38dc7074df22e/settings.gradle#L17-L19

[bentsherman]

@pditommaso this is ready to merge, tested with registry-dev and Nextflow 25.09.0-edge

[pditommaso]

I'm getting this using Gradle 9

startup failed:
/Users/pditommaso/Projects/nextflow-plugin-gradle/src/main/groovy/io/nextflow/gradle/BuildIndexTask.groovy: 12: Can't have an abstract method in a non-abstract class. The class 'io.nextflow.gradle.BuildIndexTask' must be declared abstract or the method 'org.gradle.api.model.ObjectFactory getObjectFactory()' must be implemented.
 @ line 12, column 1.
   class BuildIndexTask extends JavaExec {
   ^

/Users/pditommaso/Projects/nextflow-plugin-gradle/src/main/groovy/io/nextflow/gradle/BuildIndexTask.groovy: 12: Can't have an abstract method in a non-abstract class. The class 'io.nextflow.gradle.BuildIndexTask' must be declared abstract or the method 'org.gradle.api.internal.provider.PropertyFactory getPropertyFactory()' must be implemented.
 @ line 12, column 1.
   class BuildIndexTask extends JavaExec {
   ^

/Users/pditommaso/Projects/nextflow-plugin-gradle/src/main/groovy/io/nextflow/gradle/BuildIndexTask.groovy: 12: Can't have an abstract method in a non-abstract class. The class 'io.nextflow.gradle.BuildIndexTask' must be declared abstract or the method 'org.gradle.process.internal.ExecActionFactory getExecActionFactory()' must be implemented.
 @ line 12, column 1.
   class BuildIndexTask extends JavaExec {
   ^

/Users/pditommaso/Projects/nextflow-plugin-gradle/src/main/groovy/io/nextflow/gradle/BuildIndexTask.groovy: 12: Can't have an abstract method in a non-abstract class. The class 'io.nextflow.gradle.BuildIndexTask' must be declared abstract or the method 'org.gradle.jvm.toolchain.JavaToolchainService getJavaToolchainService()' must be implemented.
 @ line 12, column 1.
   class BuildIndexTask extends JavaExec {
   ^

/Users/pditommaso/Projects/nextflow-plugin-gradle/src/main/groovy/io/nextflow/gradle/BuildIndexTask.groovy: 12: Can't have an abstract method in a non-abstract class. The class 'io.nextflow.gradle.BuildIndexTask' must be declared abstract or the method 'org.gradle.api.provider.ProviderFactory getProviderFactory()' must be implemented.
 @ line 12, column 1.
   class BuildIndexTask extends JavaExec {
   ^

5 errors

I had the sam issue, for other tasks, the solution was to mark as abstract

7f78665

[bentsherman]

It looks like your local clone is out of date. BuildIndexTask doesn't exist anymore

[pditommaso]

Correct, but the problem is still the same

startup failed:
/Users/pditommaso/Projects/nextflow-plugin-gradle/src/main/groovy/io/nextflow/gradle/BuildSpecTask.groovy: 14: Can't have an abstract method in a non-abstract class. The class 'io.nextflow.gradle.BuildSpecTask' must be declared abstract or the method 'org.gradle.api.model.ObjectFactory getObjectFactory()' must be implemented.
 @ line 14, column 1.
   class BuildSpecTask extends JavaExec {
   ^

/Users/pditommaso/Projects/nextflow-plugin-gradle/src/main/groovy/io/nextflow/gradle/BuildSpecTask.groovy: 14: Can't have an abstract method in a non-abstract class. The class 'io.nextflow.gradle.BuildSpecTask' must be declared abstract or the method 'org.gradle.api.internal.provider.PropertyFactory getPropertyFactory()' must be implemented.
 @ line 14, column 1.
   class BuildSpecTask extends JavaExec {
   ^

/Users/pditommaso/Projects/nextflow-plugin-gradle/src/main/groovy/io/nextflow/gradle/BuildSpecTask.groovy: 14: Can't have an abstract method in a non-abstract class. The class 'io.nextflow.gradle.BuildSpecTask' must be declared abstract or the method 'org.gradle.process.internal.ExecActionFactory getExecActionFactory()' must be implemented.
 @ line 14, column 1.
   class BuildSpecTask extends JavaExec {
   ^

/Users/pditommaso/Projects/nextflow-plugin-gradle/src/main/groovy/io/nextflow/gradle/BuildSpecTask.groovy: 14: Can't have an abstract method in a non-abstract class. The class 'io.nextflow.gradle.BuildSpecTask' must be declared abstract or the method 'org.gradle.jvm.toolchain.JavaToolchainService getJavaToolchainService()' must be implemented.
 @ line 14, column 1.
   class BuildSpecTask extends JavaExec {
   ^

/Users/pditommaso/Projects/nextflow-plugin-gradle/src/main/groovy/io/nextflow/gradle/BuildSpecTask.groovy: 14: Can't have an abstract method in a non-abstract class. The class 'io.nextflow.gradle.BuildSpecTask' must be declared abstract or the method 'org.gradle.api.provider.ProviderFactory getProviderFactory()' must be implemented.
 @ line 14, column 1.
   class BuildSpecTask extends JavaExec {
   ^

5 errors

[pditommaso]

I've added the abstract but now I've this problem

» ./gradlew releasePluginToRegistryIfNotExists

[Incubating] Problems report is available at: file:///Users/pditommaso/Projects/nextflow/build/reports/problems/problems-report.html

FAILURE: Build failed with an exception.

* Where:
Build file '/Users/pditommaso/Projects/nextflow/plugins/nf-amazon/build.gradle' line: 17

* What went wrong:
An exception occurred applying plugin request [id: 'io.nextflow.nextflow-plugin', version: '1.0.0-beta.10']
> Failed to apply plugin 'io.nextflow.nextflow-plugin'.
   > Cannot add a configuration with name 'specFileImplementation' as a configuration with that name already exists.

What's more concerning this PR introduces a circular dependencies with nextflow main runtime makes difficult to this plugin with nextflow itself, because when releasing a new nextflow version, the plugin will try to pull a nextflow dependency that does not yet exist.

I'm starting to think the spec should not be generated by this plugin, but instead it should be defined by each plugin itself. The build process should only invoked a well defined plugin interface

[bentsherman]

I'm confused. This project doesn't even use Gradle 9:

nextflow-plugin-gradle/gradle/wrapper/gradle-wrapper.properties

Line 3 in 8b413b3

distributionUrl=https\://services.gradle.org/distributions/gradle-8.10.2-bin.zip

I think that's why I'm not getting a build error

[bentsherman]

As far as I can tell there is no circular dependency:

1. build the nextflow plugin
2. run the PluginSpecWriter from nextflow with the plugin JAR and user-defined extension points to build the spec
3. package and publish the plugin

In other words the plugin JAR is built before the spec file, and the same nextflow runtime specified by the plugin is used to build the spec. For this reason it only builds the spec if the nextflow version is >=25.09.0-edge

[pditommaso]

The problem arise when using from nextflow build

[bentsherman]

I see. Well, this is a good example where mavenLocal works better 😄

[pditommaso]

it remains a mess, for example when trying to make 25.10.0 and having a plugin pinning that version, the gradle plugin will fail to resolve it

[bentsherman]

You mean for core plugins? The core plugins don't need to generate plugin specs, for that case we could just disable it

[bentsherman]

@pditommaso I added a flag to not build the plugin spec, which should be used by the core plugins. I also fixed the issues with Gradle 9 with some help from Claude. Let me know if it works for you

[pditommaso]

IMO it would be preferable a solution that works the same both for "internal" and external plugins.

Exploring the annotation processor i've ended up with this solution. Have a look.

[bentsherman]

It looks like your solution is more or less the same. You still use a class loader with the Nextflow runtime and the plugin JAR, only now you are loading all of the annotation classes by name instead of just the plugin spec entrypoint (nextflow.plugin.spec.PluginSpecWriter)

The problem here is that now the code for generating the JSON metadata is duplicated in Nextflow and the Gradle plugin. You still need this code in the Nextflow runtime to generate this metadata for the language server

So if you want something that works for both core plugins and community plugins, the solution is what we merged in nextflow-io/nextflow#6361 in combination with this PR.

• The code for generating spec definitions is in the Nextflow runtime
• The language server uses it at build time to extract core definitions (core runtime + core plugins)
• The Gradle plugin uses it at plugin build time to extract a plugin spec
  • core plugins should skip this part because they don't need plugin specs

Let me know if you encounter any more issues using this PR

[pditommaso]

Another option could be moving this project into the nextflow repo. That would allow the plugin to stay in sync core codebase.

[bentsherman]

I'm not sure how that would work. The Gradle plugin is intended to be used with any version of Nextflow, but moving it into the Nextflow codebase would tie it to a specific Nextflow version

[pditommaso]

In the codebase it uses the "inline" version. Other plugins the canonical distribution

[bentsherman]

That's essentially what I have achieved in this PR -- the core plugins should just use useDefaultDependencies = false and buildSpec = false. We could collapse these into a single flag like corePlugin if you prefer.