[SOLVED] using GNU parallel with multiple commands

Issue

I am using GNU parallel to download a batch of file then I would like to compress them at the same time with gzip.

I have a list of https address of the file (http_list)

cat htpp_list | parallel -j 72 wget

it will download files with extension *.jpeg

then I do this:

ls -al *.jpeg | awk '{print $NF}' > file_list
cat file_list | parallel -j 72 gzip

Is there a way to write only in one line of code to download and compress at same time?

Solution

Do not parse ls. Use glob or find.

Do not abuse cats. Use <file instead of cat file | command.

Check your scripts with shellcheck.

I would like to compress them at the same time with gzip.

I see no reason to store them uncompressed then.

work() {
   name=$(basename "$1")
   curl "$1" | gzip > "$name".jpeg.gz
}
export -f work
parallel -j 72 work <htpp_list

Answered By - KamilCuk

Answer Checked By - Pedro (WPSolving Volunteer)