Tuesday, January 30, 2024

[SOLVED] how to eliminate trailing newlines and store the result in variables

Issue

As the title says, I want to eliminate trailing newlines and store the result in variables (Bash).

The following does not work.

[xxx@ ~]$ var=`echo 'abc\ndef\n' | sed 's/\\n$//g'`
[xxx@ ~]$ echo $var
abc\ndef\n

The following works!

[xxx@ ~]$ var=`echo 'abc\ndef\n' | sed 's/\\\\n$//g'`
[xxx@ ~]$ echo $var
abc\ndef

I thought that only one backslash escape was needed for sed since it is enclosed in single quotes, but I guess my understanding is wrong.

Please tell me why the second command works correctly?


Solution

Regarding:

Please tell me why the second command works correctly?

This command produces the shown output:

$ echo 'abc\ndef\n' | sed 's/\\\\n$//g'
abc\ndef\n

As for why you get this unexpected output with old, deprecated backticks:

$ var=`echo 'abc\ndef\n' | sed 's/\\\\n$//g'`
$ echo "$var"
abc\ndef

which is different from this expected output with modern command substitution ($(...)):

$ var=$(echo 'abc\ndef\n' | sed 's/\\\\n$//g')
$ echo "$var"
abc\ndef\n

it's because backslashes (\) inside backticks are handled in a non-obvious manner as stated and demonstrated at the top of the related FAQ, https://mywiki.wooledge.org/BashFAQ/082, and that is one of the reasons to use $(...) instead of backticks to execute commands.

So, if you want to remove a trailing \n string using a pipe to sed then do this:

$ var=$(echo 'abc\ndef\n' | sed 's/\\n$//g')
$ echo "$var"
abc\ndef

otherwise, if you really want to remove newlines as your Subject line says then read on:

echo 'abc\ndef\n' produces this output:

$ echo 'abc\ndef\n'
abc\ndef\n

The only newline present in that output is the one at the end of the line that makes that a valid POSIX text file. The \ns are literally the 2 characters \ and \n.

You should use printf instead of echo for portability and consistency. In this case it would convert \n strings to newlines:

$ printf 'abc\ndef\n'
abc
def

Now, to get RID of the newlines - sed by default reads input 1 line at a time so you can't remove a newline from a string that sed sees as there cannot be a newline within 1 line that's part of newline-separated input. You can do things with sed to make it read multiple lines but it's more robust and portable to use awk instead, e.g.:

to remove just the final newline:

$ printf 'abc\ndef\n' | awk '{printf "%s%s", sep, $0; sep=ORS}'
abc
def             <--- no newline at the end

or to remove all newlines:

$ printf 'abc\ndef\n' | awk -v ORS= '1'
abcdef          <--- no newline at the end

or to remove all except the final newline:

$ printf 'abc\ndef\n' | awk -v ORS= '1; END{print RS}'
abcdef          <--- newline at the end

or similar depending on whether you want the string to end in a newline (and so be a valid POSIX text line/file) or not.

To execute a command, store it's result in a variable, and then print the contents of that variable, one way using command substitution would be:

$ var=$(printf 'abc\ndef\n' | awk -v ORS= '1; END{print RS}')
$ printf '%s\n' "$var"
abcdef

Note the use of $(...) instead of the long deprecated backticks (https://mywiki.wooledge.org/BashFAQ/082), and the necessary quotes "..." around the shell variable (see https://mywiki.wooledge.org/Quotes). Get into the habit of copy/pasting your code into http://shellcheck.net and it'll help you find and fix errors.



Answered By - Ed Morton
Answer Checked By - David Goodson (WPSolving Volunteer)
at