Issue

I am trying to use scp to copy multiple files from a remote server.

I can successfully do it for a single argument as script file_number

Where script is:

#!/bin/bash
echo Insert your username:
read user
scp $user@server:/path/run$1*.lz4 ./
lz4 -mdv --rm run$1*.lz4

All my files have the format: runXXXXsubY.lz4

This way I can copy all the files that have XXXX==file_number, but I want to pass multiple arguments, and copy all those files e.g. script 1 2 to copy all the files with xxxx==1 and XXXX==2.

I tried with:

scp $user@server:/path/run"$@"*.lz4 ./

But $@ gets expanded without appending the *.lz4

I appreciate any help in the right direction.

Thanks

Solution

The basic problem is that when the shell expands $user@server:/path/run"$@"*.lz4, it doesn't copy the $user:... and *.lz4 parts for each argument, it just kind of blindly adds the list of arguments -- including word breaks between arguments -- into the middle. So if the args are 1 and 2, it essentially expands to:

scp $user@server:/path/run"1" "2"*.lz4 ./

...so $user@server:/path/run"1" and "2"*.lz4 are separate arguments to scp, which isn't useful. What you can do is create an array based on the arguments, and then use that as the source list for scp. Something like this:

sources=()
for runNum in "$@"; do
    sources+=("$user@server:/path/run${runNum}*.lz4")
done
scp "${sources[@]}" ./

And then use a separate loop for the lz4 command:

for runNum in "$@"; do
    lz4 -mdv --rm run${runNum}*.lz4
done

EDIT: to avoid having to authenticate multiple times, you can open a master SSH connection and let all the scp transfers piggyback on that. Here's an example, based heavily on Felix Rabe's answer here:

# Create the array of source file patterns to fetch
sources=()
for runNum in "$@"; do
    sources+=("$user@server:/path/run${runNum}*.lz4")
done

# Open master SSH connection:
# (This is the only time you have to enter the password)
sshSocket=~/"$user@server"
ssh -M -f -N -S "$sshSocket" "$user@server"

# Actually copy the files:
scp -o ControlPath="$sshSocket" "${sources[@]}" ./

# Close master connection:
ssh -S "$sshSocket" -O exit "$user@server"

Answered By - Gordon Davisson

Tuesday, November 2, 2021

[SOLVED] scp multiple arguments of bash script

Issue

Solution