Implement ffq

[drpatelh]

First pass implementation to use ffq to fetch metadata from public databases.

[github-actions]

nf-core lint overall result: Passed ✅ ⚠️

Posted for pipeline commit 4506619

+| ✅ 143 tests passed       |+
#| ❔   2 tests were ignored |#
!| ❗   4 tests had warnings |!

❗ Test warnings:

• files_exist - File not found: conf/igenomes.config
• files_exist - File not found: lib/WorkflowFetchngs.groovy
• nextflow_config - Config manifest.version should end in dev: '1.7'
• readme - README did not have a Nextflow minimum version badge.

❔ Tests ignored:

• files_unchanged - File ignored due to lint config: assets/sendmail_template.txt
• files_unchanged - File ignored due to lint config: lib/NfcoreTemplate.groovy

✅ Tests passed:

• files_exist - File found: .gitattributes
• files_exist - File found: .gitignore
• files_exist - File found: .nf-core.yml
• files_exist - File found: .editorconfig
• files_exist - File found: .prettierrc.yml
• files_exist - File found: CHANGELOG.md
• files_exist - File found: CITATIONS.md
• files_exist - File found: CODE_OF_CONDUCT.md
• files_exist - File found: CODE_OF_CONDUCT.md
• files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
• files_exist - File found: nextflow_schema.json
• files_exist - File found: nextflow.config
• files_exist - File found: README.md
• files_exist - File found: .github/.dockstore.yml
• files_exist - File found: .github/CONTRIBUTING.md
• files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
• files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
• files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
• files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
• files_exist - File found: .github/workflows/branch.yml
• files_exist - File found: .github/workflows/ci.yml
• files_exist - File found: .github/workflows/linting_comment.yml
• files_exist - File found: .github/workflows/linting.yml
• files_exist - File found: assets/email_template.html
• files_exist - File found: assets/email_template.txt
• files_exist - File found: assets/sendmail_template.txt
• files_exist - File found: assets/nf-core-fetchngs_logo_light.png
• files_exist - File found: conf/modules.config
• files_exist - File found: conf/test.config
• files_exist - File found: conf/test_full.config
• files_exist - File found: docs/images/nf-core-fetchngs_logo_light.png
• files_exist - File found: docs/images/nf-core-fetchngs_logo_dark.png
• files_exist - File found: docs/output.md
• files_exist - File found: docs/README.md
• files_exist - File found: docs/README.md
• files_exist - File found: docs/usage.md
• files_exist - File found: lib/nfcore_external_java_deps.jar
• files_exist - File found: lib/NfcoreSchema.groovy
• files_exist - File found: lib/NfcoreTemplate.groovy
• files_exist - File found: lib/Utils.groovy
• files_exist - File found: lib/WorkflowMain.groovy
• files_exist - File found: main.nf
• files_exist - File found: assets/multiqc_config.yml
• files_exist - File found: conf/base.config
• files_exist - File found: .github/workflows/awstest.yml
• files_exist - File found: .github/workflows/awsfulltest.yml
• files_exist - File found: modules.json
• files_exist - File not found check: Singularity
• files_exist - File not found check: parameters.settings.json
• files_exist - File not found check: .nf-core.yaml
• files_exist - File not found check: bin/markdown_to_html.r
• files_exist - File not found check: conf/aws.config
• files_exist - File not found check: .github/workflows/push_dockerhub.yml
• files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
• files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
• files_exist - File not found check: docs/images/nf-core-fetchngs_logo.png
• files_exist - File not found check: .markdownlint.yml
• files_exist - File not found check: .yamllint.yml
• files_exist - File not found check: .travis.yml
• nextflow_config - Config variable found: manifest.name
• nextflow_config - Config variable found: manifest.nextflowVersion
• nextflow_config - Config variable found: manifest.description
• nextflow_config - Config variable found: manifest.version
• nextflow_config - Config variable found: manifest.homePage
• nextflow_config - Config variable found: timeline.enabled
• nextflow_config - Config variable found: trace.enabled
• nextflow_config - Config variable found: report.enabled
• nextflow_config - Config variable found: dag.enabled
• nextflow_config - Config variable found: process.cpus
• nextflow_config - Config variable found: process.memory
• nextflow_config - Config variable found: process.time
• nextflow_config - Config variable found: params.outdir
• nextflow_config - Config variable found: params.input
• nextflow_config - Config variable found: params.show_hidden_params
• nextflow_config - Config variable found: params.schema_ignore_params
• nextflow_config - Config variable found: manifest.mainScript
• nextflow_config - Config variable found: timeline.file
• nextflow_config - Config variable found: trace.file
• nextflow_config - Config variable found: report.file
• nextflow_config - Config variable found: dag.file
• nextflow_config - Config variable (correctly) not found: params.version
• nextflow_config - Config variable (correctly) not found: params.nf_required_version
• nextflow_config - Config variable (correctly) not found: params.container
• nextflow_config - Config variable (correctly) not found: params.singleEnd
• nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
• nextflow_config - Config variable (correctly) not found: params.name
• nextflow_config - Config timeline.enabled had correct value: true
• nextflow_config - Config report.enabled had correct value: true
• nextflow_config - Config trace.enabled had correct value: true
• nextflow_config - Config dag.enabled had correct value: true
• nextflow_config - Config manifest.name began with nf-core/
• nextflow_config - Config variable manifest.homePage began with https://github.com/nf-core/
• nextflow_config - Config dag.file ended with .html
• nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
• nextflow_config - Config params.custom_config_version is set to master
• nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
• nextflow_config - Lines for loading custom profiles found
• files_unchanged - .gitattributes matches the template
• files_unchanged - .prettierrc.yml matches the template
• files_unchanged - CODE_OF_CONDUCT.md matches the template
• files_unchanged - LICENSE matches the template
• files_unchanged - .github/.dockstore.yml matches the template
• files_unchanged - .github/CONTRIBUTING.md matches the template
• files_unchanged - .github/ISSUE_TEMPLATE/bug_report.yml matches the template
• files_unchanged - .github/ISSUE_TEMPLATE/config.yml matches the template
• files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
• files_unchanged - .github/PULL_REQUEST_TEMPLATE.md matches the template
• files_unchanged - .github/workflows/branch.yml matches the template
• files_unchanged - .github/workflows/linting_comment.yml matches the template
• files_unchanged - .github/workflows/linting.yml matches the template
• files_unchanged - assets/email_template.html matches the template
• files_unchanged - assets/email_template.txt matches the template
• files_unchanged - assets/nf-core-fetchngs_logo_light.png matches the template
• files_unchanged - docs/images/nf-core-fetchngs_logo_light.png matches the template
• files_unchanged - docs/images/nf-core-fetchngs_logo_dark.png matches the template
• files_unchanged - docs/README.md matches the template
• files_unchanged - lib/nfcore_external_java_deps.jar matches the template
• files_unchanged - lib/NfcoreSchema.groovy matches the template
• files_unchanged - .gitignore matches the template
• actions_ci - '.github/workflows/ci.yml' is triggered on expected events
• actions_ci - '.github/workflows/ci.yml' checks minimum NF version
• actions_awstest - '.github/workflows/awstest.yml' is triggered correctly
• actions_awsfulltest - .github/workflows/awsfulltest.yml is triggered correctly
• actions_awsfulltest - .github/workflows/awsfulltest.yml does not use -profile test
• readme - README Nextflow minimum version in Quick Start section matched config. README: 21.10.3, Config: 21.10.3
• pipeline_todos - No TODO strings found
• pipeline_name_conventions - Name adheres to nf-core convention
• template_strings - Did not find any Jinja template strings (83 files)
• schema_lint - Schema lint passed
• schema_lint - Schema title + description lint passed
• schema_params - Schema matched params returned from nextflow config
• actions_schema_validation - Workflow validation passed: linting_comment.yml
• actions_schema_validation - Workflow validation passed: branch.yml
• actions_schema_validation - Workflow validation passed: linting.yml
• actions_schema_validation - Workflow validation passed: awstest.yml
• actions_schema_validation - Workflow validation passed: fix-linting.yml
• actions_schema_validation - Workflow validation passed: awsfulltest.yml
• actions_schema_validation - Workflow validation passed: ci.yml
• merge_markers - No merge markers found in pipeline files
• modules_json - Only installed modules found in modules.json
• multiqc_config - 'assets/multiqc_config.yml' follows the ordering scheme of the minimally required plugins.
• multiqc_config - 'assets/multiqc_config.yml' contains a matching 'report_comment'.
• multiqc_config - 'assets/multiqc_config.yml' contains 'export_plots: true'.

Run details

• nf-core/tools version 2.4.1
• Run at 2022-07-01 16:22:50

[drpatelh]

Tests are expected to fail here due to recent breaking changes introduces in the NCBI API.

The plan is to try and get the tests passing again using ffq instead and to try and maintain feature parity with the existing release in terms of the samplesheet output and id compatibility.