Add `fastq-dl` as download method

[camlloyd]

Description of feature

Hey folks, I found this tool: https://github.com/rpetit3/fastq-dl and would like to try to add it to fetchngs since I encountered a couple of problems with sra-tools and thought this might be a nice alternative?

From the # fetchngs Slack channel: https://nfcore.slack.com/archives/C024915L4KB/p1741267281724959

[tanaes]

Thanks for working on this!

I've given your branch a shot on some files I'm downloading. Currently both FTP and Aspera downloads have been virtually unusable for me, rendering fetchngs essentially useless.

Unfortunately, i'm encountering a problem where download failures from wget are not throwing errors in the job itself. For example, this error log:

2025-08-14 05:08:05 INFO     2025-08-14 05:08:05:root:INFO -     download.py:189
                             Query: SRR29301909                                 
                    INFO     2025-08-14 05:08:05:root:INFO -     download.py:190
                             Archive: ena                                       
                    INFO     2025-08-14 05:08:05:root:INFO -     download.py:195
                             Total Runs To Download: 1                          
                    INFO     2025-08-14 05:08:05:root:INFO -     download.py:214
                             Working on run SRR29301909...                      
                    INFO     2025-08-14 05:08:05:root:INFO -          ena.py:167
                             SRR29301909_1.fastq.gz FTP download                
                             attempt 1                                          
2025-08-14 05:08:45 INFO     2025-08-14 05:08:45:root:INFO -          ena.py:195
                             Successfully downloaded                            
                             SRR29301909_1.fastq.gz                             
                    INFO     2025-08-14 05:08:45:root:INFO -          ena.py:167
                             SRR29301909_2.fastq.gz FTP download                
                             attempt 1                                          
2025-08-14 05:08:49 ERROR    2025-08-14 05:08:49:root:ERROR - "wget  utils.py:67
                             --quiet -O SRR29301909_2.fastq.gz                  
                             ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR2            
                             93/009/SRR29301909/SRR29301909_2.fastq.            
                             gz" return exit code 4                             
                    ERROR    2025-08-14 05:08:49:root:ERROR - Retry  utils.py:76
                             execution (1 of 10)                                
2025-08-14 05:10:31 ERROR    2025-08-14 05:10:31:root:ERROR - "wget  utils.py:67
                             --quiet -O SRR29301909_2.fastq.gz                  
                             ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR2            
                             93/009/SRR29301909/SRR29301909_2.fastq.            
                             gz" return exit code 4                             
                    ERROR    2025-08-14 05:10:31:root:ERROR - Retry  utils.py:76
                             execution (2 of 10)  
....
                    ERROR    2025-08-14 05:15:21:root:ERROR - Retry  utils.py:76
                             execution (9 of 10)                                
2025-08-14 05:16:09 ERROR    2025-08-14 05:16:09:root:ERROR - "wget  utils.py:67
                             --quiet -O SRR29301909_2.fastq.gz                  
                             ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR2            
                             93/009/SRR29301909/SRR29301909_2.fastq.            
                             gz" return exit code 4                             
                    INFO     2025-08-14 05:16:09:root:INFO -     download.py:232
                             SRR29301909 not found on ENA,                      
                             retrying from SRA                                  
                    INFO     2025-08-14 05:16:09:root:INFO -     download.py:311
                             Writing metadata to                                
                             ./SRX24819028_SRR29301909-run-info.                
                             tsv                                                
ls: *-run-mergers.tsv: No such file or directory
                              

resulted in an empty fastq.gz file for the _2 read, but the job still exited with code 0 (good), so the pipeline didn't see any problems.

[camlloyd]

Hi @tanaes, thank you for testing this! Please note this branch is still a work-in-progress.

Could you please share:

1. The command you used to run the pipeline
2. The samplesheet you used

[camlloyd]

Currently both FTP and Aspera downloads have been virtually unusable for me, rendering fetchngs essentially useless.

@tanaes Have you ruled out a firewall issue? Have you tried running this on a different network?

[tanaes]

Sure thing! Here's the command:

nextflow run camlloyd/fetchngs \
-r fastqdl \
-profile docker \
-w "az://{azure-path}/work/Smith_2022/" \
--input "./ids.csv" \
--outdir "az://{azure-path}/Smith_2022/RNA-seq/" \
--nf_core_pipeline rnaseq \
--nf_core_rnaseq_strandedness auto \
--download_method aspera \
-resume \
-c "./azurebatch.config" \
-c "./nextflow.config" \
--download_method fastq-dl \
--az_location $AZURE_BATCH_LOCATION         \
--batch_name $AZURE_BATCH_ACCOUNT_NAME         \
--batch_key $AZURE_BATCH_ACCOUNT_KEY         \
--storage_name $AZURE_STORAGE_ACCOUNT_NAME         \
--storage_key $AZURE_STORAGE_ACCOUNT_KEY         \
--vm_type 'Standard_E4ads_v5_spot'         \
-with-report

and the ids:

SRR29301894
SRR29301895
SRR29301900
SRR29301902
SRR29301903
SRR29301904
SRR29301919
SRR29301920
SRR29301888
SRR29301891
SRR29301898
SRR29301899
SRR29301905
SRR29301906
SRR29301907
SRR29301908
SRR29301909
SRR29301910
SRR29301911
SRR29301913
SRR29301917
SRR29301921
SRR29301887
SRR29301889
SRR29301890
SRR29301892
SRR29301893
SRR29301896
SRR29301897
SRR29301901
SRR29301912
SRR29301914
SRR29301915
SRR29301916
SRR29301918
SRR29301922

I'm sure there are some network issues at the heart of my problems -- these are being run on Azure VMs. I haven't been able to hunt down the fundamental cause yet unfortunately. :(

[camlloyd]

--download_method aspera \

Have you tried using --download_method fastq-dl here?

I haven't had the chance to update the documentation on my branch yet, sorry about that!

EDIT: I see you've passed --download_method twice.

[tanaes]

haha i had just noticed that too when you posted this! Totally my mistake. But since it called fastq-dl later that's what it used in that run.