[SOLVED] Airflow bash operator: Problem using sed in hdfs

Issue

I have a airflow task that I am trying to use sed command for replacing LF with CRLF:

hdfs dfs -cat /test/file.txt | sed 's/$/\r/g' | hdfs dfs -put -f - /test/file.txt

I get following error:

error: sed: -e expression #1, char 4: unterminated `s' command 

I think it is due to \r which it is conflicting with. How do I solve this problem?

Solution

I found the reason, the \ is a special character in Python. To solved it I just added an extra \ is it becomes sed 's/$/\\r/g' , another option is to use prefixing.

Answered By - Alex K

Answer Checked By - Timothy Miller (WPSolving Admin)