Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 13 additions & 13 deletions docs/developer/config-scopes.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ This page provides guidance on defining configuration scopes in the Nextflow run

The Nextflow configuration is defined as a collection of *scope classes*. Each scope class defines the set of available options, including their name, type, and an optional description for a specific configuration scope.

Scope classes are used to generate a configuration schema, which is in turn used for several purposes:
Scope classes are used to generate a configuration spec, which is in turn used for several purposes:

- Validating config options at runtime (`nextflow run` and `nextflow config`)

Expand All @@ -29,15 +29,15 @@ For example:
```groovy
package nextflow.hello

import nextflow.config.schema.ConfigScope
import nextflow.config.schema.ScopeName
import nextflow.config.spec.ConfigScope
import nextflow.config.spec.ScopeName

@ScopeName('hello')
class HelloConfig implements ConfigScope {
}
```

A scope class must provide a default constructor, so that it can be instantiated as an extension point. If no such constructor is defined, the config scope will not be included in the schema. In the above example, this constructor is implicitly defined because no constructors were declared.
A scope class must provide a default constructor, so that it can be instantiated as an extension point. If no such constructor is defined, the config scope will not be detected by Nextflow. In the above example, this constructor is implicitly defined because no constructors were declared.

The fully-qualified class name (in this case, `nextflow.hello.HelloConfig`) must be included in the list of extension points.

Expand All @@ -59,7 +59,7 @@ The `@ConfigOption` annotation can specify an optional set of types that are val
String tags
```

The field type and any additional types are included in the schema, allowing them to be used for validation.
The field type and any additional types are included in the config spec, allowing them to be used for validation.

The field type can be any Java or Groovy class, but in practice it should be a class that can be constructed from primitive values (numbers, booleans, strings). For example, `Duration` and `MemoryUnit` are standard Nextflow types that can each be constructed from an integer or string.

Expand All @@ -83,7 +83,7 @@ See `AzBatchOpts` and `AzPoolOpts` for an example of how placeholder scopes are

### Descriptions

Top-level scope classes and config options should use the `@Description` annotation to provide a description of the scope or option. This description is included in the schema, which is in turn used by the language server to provide hover hints.
Top-level scope classes and config options should use the `@Description` annotation to provide a description of the scope or option. This description is included in the config spec, which is in turn used by the language server to provide hover hints.

For example:

Expand Down Expand Up @@ -121,9 +121,9 @@ The Nextflow runtime adheres the following best practices where appropriate:
For example:

```groovy
import nextflow.config.schema.ConfigOption
import nextflow.config.schema.ConfigScope
import nextflow.config.schema.ScopeName
import nextflow.config.spec.ConfigOption
import nextflow.config.spec.ConfigScope
import nextflow.config.spec.ScopeName

@ScopeName('hello')
class HelloConfig implements ConfigScope {
Expand All @@ -147,7 +147,7 @@ class HelloConfig implements ConfigScope {

### Runtime

Nextflow validates the config map after it is loaded. Top-level config scopes are loaded by the plugin system as extension points and converted into a schema, which is used to validate the config map.
Nextflow validates the config map after it is loaded. Top-level config scopes are loaded by the plugin system as extension points and converted into a config spec, which is used to validate the config map.

Plugins are loaded after the config is loaded and before it is validated, since plugins can also define config scopes. If a third-party plugin declares a config scope, it must be explicitly enabled in order to validate config options from the plugin. Otherwise, Nextflow will report these options as unrecognized.

Expand All @@ -165,12 +165,12 @@ new ExecutorConfig( Global.session.config.executor as Map ?: Collections.emptyMa
In practice, it is better to avoid the use of `Global` and provide an instance of `Session` to the client class instead.
:::

### JSON Schema
### Config spec

Config scope classes can be converted into a schema with the `SchemaNode` class, which uses reflection to extract metadata such as scope names, option names, types, and descriptions. This schema is rendered to JSON and used by the language server at build-time to provide code intelligence such as code completion and hover hints.
Config scope classes can be converted into a config spec with the `SpecNode` class, which uses reflection to extract metadata such as scope names, option names, types, and descriptions. This spec is rendered to JSON and used by the language server at build-time to provide code intelligence such as code completion and hover hints.

### Documentation

The schema described above can also be rendered to Markdown using the `MarkdownRenderer` class. It produces a Markdown document approximating the {ref}`config-options` page.
The config spec described above can be rendered to Markdown using the `MarkdownRenderer` class. It produces a Markdown document approximating the {ref}`config-options` page.

This approach to docs generation is not yet complete, and has not been incorporated into the build process yet. However, it can be used to check for discrepancies between the source code and docs when making changes. The documentation should match the `@Description` annotations as closely as possible, but may contain additional details such as version notes and extra paragraphs.
2 changes: 2 additions & 0 deletions docs/migrations/25-10.md
Original file line number Diff line number Diff line change
Expand Up @@ -156,6 +156,8 @@ This feature addresses previous inconsistencies in timestamp representations.

- The AWS Java SDK used by Nextflow was upgraded from v1 to v2, which introduced some breaking changes to the `aws.client` config options. See {ref}`the guide <aws-java-sdk-v2-page>` for details.

- The `nextflow.config.schema` package was renamed to `nextflow.config.spec`. Plugin developers that define custom {ref}`configuration scopes <dev-plugins-extension-points-config>` will need to update their imports accordingly.

## Deprecations

- The legacy type detection of CLI parameters is disabled when using the strict syntax (`NXF_SYNTAX_PARSER=v2`). {ref}`Legacy parameters <workflow-params-legacy>` in the strict syntax should not rely on legacy type detection. Alternatively, use the new `params` block to convert CLI parameters based on their type annotations. Legacy type detection can be disabled globally by setting the environment variable `NXF_DISABLE_PARAMS_TYPE_DETECTION=true`.
Expand Down
12 changes: 9 additions & 3 deletions docs/plugins/developing-plugins.md
Original file line number Diff line number Diff line change
Expand Up @@ -170,6 +170,8 @@ nextflow plugin my-plugin:hello --alpha --beta

See the {ref}`cli-plugin` for usage information.

(dev-plugins-extension-points-config)=

### Configuration

Plugins can access the resolved Nextflow configuration through the session object using `session.config.navigate()`. Several extension points provide the session object for this reason. This method allows you to query any configuration option safely. If the option isn’t defined, it will return null.
Expand Down Expand Up @@ -205,12 +207,16 @@ myplugin {
:::{versionadded} 25.04.0
:::

:::{versionchanged} 25.10.0
The `nextflow.config.schema` package was renamed to `nextflow.config.spec`.
:::

Plugins can declare their configuration options by implementing the `ConfigScope` interface and declaring each config option as a field with the `@ConfigOption` annotation. For example:

```groovy
import nextflow.config.schema.ConfigOption
import nextflow.config.schema.ConfigScope
import nextflow.config.schema.ScopeName
import nextflow.config.spec.ConfigOption
import nextflow.config.spec.ConfigScope
import nextflow.config.spec.ScopeName
import nextflow.script.dsl.Description

@ScopeName('myplugin')
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,9 @@ package nextflow.conda
import java.nio.file.Path

import groovy.transform.CompileStatic
import nextflow.config.schema.ConfigOption
import nextflow.config.schema.ConfigScope
import nextflow.config.schema.ScopeName
import nextflow.config.spec.ConfigOption
import nextflow.config.spec.ConfigScope
import nextflow.config.spec.ScopeName
import nextflow.script.dsl.Description
import nextflow.util.Duration

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,9 @@
package nextflow.config

import groovy.transform.CompileStatic
import nextflow.config.schema.ConfigOption
import nextflow.config.schema.ConfigScope
import nextflow.config.schema.ScopeName
import nextflow.config.spec.ConfigOption
import nextflow.config.spec.ConfigScope
import nextflow.config.spec.ScopeName
import nextflow.script.dsl.Description
/**
* Represent Nextflow config as Map
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,9 @@ package nextflow.config

import groovy.transform.CompileStatic
import groovy.util.logging.Slf4j
import nextflow.config.schema.ConfigScope
import nextflow.config.schema.SchemaNode
import nextflow.config.schema.ScopeName
import nextflow.config.spec.ConfigScope
import nextflow.config.spec.SpecNode
import nextflow.config.spec.ScopeName
import nextflow.plugin.Plugins
import nextflow.script.dsl.Description
/**
Expand Down Expand Up @@ -62,20 +62,20 @@ class ConfigValidator {
/**
* Additional config scopes added by third-party plugins
*/
private SchemaNode.Scope pluginScopes
private SpecNode.Scope pluginScopes

ConfigValidator() {
loadPluginScopes()
}

private void loadPluginScopes() {
final children = new HashMap<String, SchemaNode>()
final children = new HashMap<String, SpecNode>()
for( final scope : Plugins.getExtensions(ConfigScope) ) {
final clazz = scope.getClass()
final name = clazz.getAnnotation(ScopeName)?.value()
final description = clazz.getAnnotation(Description)?.value()
if( name == '' ) {
children.putAll(SchemaNode.Scope.of(clazz, '').children())
children.putAll(SpecNode.Scope.of(clazz, '').children())
continue
}
if( !name )
Expand All @@ -84,9 +84,9 @@ class ConfigValidator {
log.warn "Plugin config scope `${clazz.name}` conflicts with existing scope: `${name}`"
continue
}
children.put(name, SchemaNode.Scope.of(clazz, description))
children.put(name, SpecNode.Scope.of(clazz, description))
}
pluginScopes = new SchemaNode.Scope('', children)
pluginScopes = new SpecNode.Scope('', children)
}

void validate(ConfigMap config) {
Expand Down Expand Up @@ -131,15 +131,15 @@ class ConfigValidator {
}

/**
* Determine whether a config option is defined in the schema.
* Determine whether a config option is defined in the spec.
*
* @param names
*/
boolean isValid(List<String> names) {
if( names.size() == 1 && names.first() in HIDDEN_OPTIONS )
return true
final child = SchemaNode.ROOT.getChild(names)
if( child instanceof SchemaNode.Option || child instanceof SchemaNode.DslOption )
final child = SpecNode.ROOT.getChild(names)
if( child instanceof SpecNode.Option || child instanceof SpecNode.DslOption )
return true
if( pluginScopes.getOption(names) )
return true
Expand All @@ -164,18 +164,18 @@ class ConfigValidator {
* @param names Config option split into individual names, e.g. 'process.resourceLimits' -> [process, resourceLimits]
*/
private boolean isMapOption(List<String> names) {
return isMapOption0(SchemaNode.ROOT, names)
return isMapOption0(SpecNode.ROOT, names)
|| isMapOption0(pluginScopes, names)
}

private static boolean isMapOption0(SchemaNode.Scope scope, List<String> names) {
SchemaNode node = scope
private static boolean isMapOption0(SpecNode.Scope scope, List<String> names) {
SpecNode node = scope
for( final name : names ) {
if( node instanceof SchemaNode.Scope )
if( node instanceof SpecNode.Scope )
node = node.children().get(name)
else if( node instanceof SchemaNode.Placeholder )
else if( node instanceof SpecNode.Placeholder )
node = node.scope()
else if( node instanceof SchemaNode.Option )
else if( node instanceof SpecNode.Option )
return node.type() == Map.class
else
return false
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,9 @@ import java.util.stream.Collectors

import groovy.transform.CompileStatic
import groovy.transform.EqualsAndHashCode
import nextflow.config.schema.ConfigOption
import nextflow.config.schema.ConfigScope
import nextflow.config.schema.ScopeName
import nextflow.config.spec.ConfigOption
import nextflow.config.spec.ConfigScope
import nextflow.config.spec.ScopeName
import nextflow.exception.AbortOperationException
import nextflow.script.dsl.Description

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,9 @@
package nextflow.config

import groovy.transform.CompileStatic
import nextflow.config.schema.ConfigOption
import nextflow.config.schema.ConfigScope
import nextflow.config.schema.ScopeName
import nextflow.config.spec.ConfigOption
import nextflow.config.spec.ConfigScope
import nextflow.config.spec.ScopeName
import nextflow.script.dsl.Description

@ScopeName("workflow")
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package nextflow.config.schema
package nextflow.config.spec

import groovy.transform.TypeChecked
import nextflow.plugin.Plugins
Expand All @@ -23,8 +23,7 @@ import nextflow.script.dsl.Description
class MarkdownRenderer {

String render() {
final schema = getSchema()
final entries = schema.entrySet().sort { entry -> entry.key }
final entries = getSpec().entrySet().sort { entry -> entry.key }
final result = new StringBuilder()
entries.each { entry ->
final scopeName = entry.key
Expand All @@ -45,24 +44,24 @@ class MarkdownRenderer {
result.append("\n${fromDescription(description)}\n")
result.append("\nThe following settings are available:\n")

final options = scope.children().findAll { name, node -> node instanceof SchemaNode.Option }
final options = scope.children().findAll { name, node -> node instanceof SpecNode.Option }
renderOptions(options, scopeName, result)

final scopes = scope.children().findAll { name, node -> node instanceof SchemaNode.Scope }
final scopes = scope.children().findAll { name, node -> node instanceof SpecNode.Scope }
renderOptions(scopes, scopeName, result)
}
return result.toString()
}

private static Map<String,SchemaNode.Scope> getSchema() {
final result = new HashMap<String,SchemaNode.Scope>()
private static Map<String,SpecNode.Scope> getSpec() {
final result = new HashMap<String,SpecNode.Scope>()
for( final scope : Plugins.getExtensions(ConfigScope) ) {
final clazz = scope.getClass()
final scopeName = clazz.getAnnotation(ScopeName)?.value()
final description = clazz.getAnnotation(Description)?.value()
if( scopeName == null )
continue
final node = SchemaNode.Scope.of(clazz, description)
final node = SpecNode.Scope.of(clazz, description)
result.put(scopeName, node)
}
return result
Expand All @@ -72,24 +71,24 @@ class MarkdownRenderer {
return description.stripIndent(true).trim()
}

private static void renderOptions(Map<String,SchemaNode> nodes, String scopeName, StringBuilder result) {
private static void renderOptions(Map<String,SpecNode> nodes, String scopeName, StringBuilder result) {
final prefix = scopeName ? scopeName + '.' : ''
final entries = nodes.entrySet().sort { entry -> entry.key }
entries.each { entry ->
final name = entry.key
final node = entry.value
if( node instanceof SchemaNode.Option )
if( node instanceof SpecNode.Option )
renderOption("${prefix}${name}", node, result)
else if( node instanceof SchemaNode.Placeholder )
else if( node instanceof SpecNode.Placeholder )
renderOptions(node.scope().children(), "${prefix}${name}.${node.placeholderName()}", result)
else if( node instanceof SchemaNode.Scope )
else if( node instanceof SpecNode.Scope )
renderOptions(node.children(), "${prefix}${name}", result)
else
throw new IllegalStateException()
}
}

private static void renderOption(String name, SchemaNode.Option node, StringBuilder result) {
private static void renderOption(String name, SpecNode.Option node, StringBuilder result) {
final description = fromDescription(node.description())
if( !description )
return
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,9 @@
package nextflow.container

import groovy.transform.CompileStatic
import nextflow.config.schema.ConfigOption
import nextflow.config.schema.ConfigScope
import nextflow.config.schema.ScopeName
import nextflow.config.spec.ConfigOption
import nextflow.config.spec.ConfigScope
import nextflow.config.spec.ScopeName
import nextflow.script.dsl.Description
import nextflow.util.Duration

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,9 @@
package nextflow.container

import groovy.transform.CompileStatic
import nextflow.config.schema.ConfigOption
import nextflow.config.schema.ConfigScope
import nextflow.config.schema.ScopeName
import nextflow.config.spec.ConfigOption
import nextflow.config.spec.ConfigScope
import nextflow.config.spec.ScopeName
import nextflow.script.dsl.Description
import nextflow.util.Duration

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,9 @@ package nextflow.container
import groovy.transform.CompileStatic
import groovy.transform.EqualsAndHashCode
import groovy.util.logging.Slf4j
import nextflow.config.schema.ConfigOption
import nextflow.config.schema.ConfigScope
import nextflow.config.schema.ScopeName
import nextflow.config.spec.ConfigOption
import nextflow.config.spec.ConfigScope
import nextflow.config.spec.ScopeName
import nextflow.script.dsl.Description

@ScopeName("docker")
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,9 @@ package nextflow.container

import groovy.transform.CompileStatic
import groovy.transform.EqualsAndHashCode
import nextflow.config.schema.ConfigOption
import nextflow.config.schema.ConfigScope
import nextflow.config.schema.ScopeName
import nextflow.config.spec.ConfigOption
import nextflow.config.spec.ConfigScope
import nextflow.config.spec.ScopeName
import nextflow.script.dsl.Description

@ScopeName("podman")
Expand Down
Loading