[SOLVED] The nextflow script didn't run exactly what I command

Issue

My main idea is to download 2 files from the internet by their download URLs. My script is as follows.

#!/usr/bin/env nextflow
params.link_download = Channel.of(
    "https://data.qiime2.org/2023.2/tutorials/moving-pictures/sample_metadata.tsv",
    "https://data.qiime2.org/2023.2/tutorials/moving-pictures/emp-single-end-sequences/barcodes.fastq.gz"
)

process DOWNDATA{

    publishDir "$projectDir", mode: 'copy'
    
    input:
    tuple val(link)
    
    output:
    path "download"
    
    """
    mkdir "download"
    name_file=\$(echo "$link" | grep --perl-regexp '[^\/]+\$' --only-matching)
    echo "\$name_file"
    wget \
      -O "download/\$name_file" \
      "$link"
    """
}


workflow{
    DOWNDATA(params.link_download)
}

But when I run my code, it appears error.

N E X T F L O W  ~  version 22.10.6
Launching `tur1.nf` [adoring_heyrovsky] DSL2 - revision: 5d71abc518
Error executing process > 'DOWNDATA (1)'
...

Caused by:
  Process `DOWNDATA (1)` terminated with an error exit status (2)

Command executed:

  mkdir "download"
  name_file=$(echo "https://data.qiime2.org/2023.2/tutorials/moving-pictures/sample_metadata.tsv/]+$' --only-matching)
  echo "$name_file"
  wget    -O "download/$name_file"    "https://data.qiime2.org/2023.2/tutorials/moving-pictures/sample_metadata.tsv"

Command exit status:
  2

Command output:
  (empty)

Command error:
  .command.sh: line 5: unexpected EOF while looking for matching `"'

I guess the problem here is in the line name_file=\$(echo "$link" | grep --perl-regexp '[^\/]+\$' --only-matching).

Why does nextflow run that line as name_file=$(echo "https://data.qiime2.org/2023.2/tutorials/moving-pictures/sample_metadata.tsv/]+$' --only-matching)? It strips the part | grep --perl-regexp '[^\.

Could someone explain to me Why this error occur? and How to fix it?. Thank you.

Solution

You get that error because you've escaped the slash. I think what you need is just: grep -oP '[^/]+\$'. Note that you could also just use basename for this, for example: name_file="\$(basename "${link}")".

Note that Nextflow can already stage the remote files for you. You might prefer the following for example:

params.sample_metadata_tsv = "https://data.qiime2.org/2023.2/tutorials/moving-pictures/sample_metadata.tsv"
params.barcodes_fastq = "https://data.qiime2.org/2023.2/tutorials/moving-pictures/emp-single-end-sequences/barcodes.fastq.gz"

params.outdir = './results'

process do_something {

    publishDir "${params.outdir}/something", mode: 'copy'

    input:
    path metadata_tsv
    path barcodes

    output:
    path 'afile.txt'

    """
    # do something with the files
    ls -1 "${metadata_tsv}"
    ls -1 "${barcodes}"

    touch 'afile.txt'
    """
}


workflow{
    
    sample_metadata_tsv = file( params.sample_metadata_tsv )
    barcodes_fastq = file( params.barcodes_fastq )

    do_something( sample_metadata_tsv, barcodes_fastq )
}

Results:

$ nextflow run main.nf 
N E X T F L O W  ~  version 23.04.1
Launching `main.nf` [serene_bassi] DSL2 - revision: 6eab1d3ed0
executor >  local (1)
[9b/dcda4a] process > do_something [100%] 1 of 1 ✔
Staging foreign file: https://data.qiime2.org/2023.2/tutorials/moving-pictures/sample_metadata.tsv
Staging foreign file: https://data.qiime2.org/2023.2/tutorials/moving-pictures/emp-single-end-sequences/barcodes.fastq.gz

$ find results/ -type f
results/something/afile.txt

Answered By - Steve

Answer Checked By - Mary Flores (WPSolving Volunteer)