Saturday, March 12, 2022

[SOLVED] Jenkinsfile and BASH shell code - Why some comments lines are honored and some are ignored and giving errors

Issue

Jenkins version: 2.164.3

I have my Jenkinsfile and in one of the stage, I'm calling the following code but it gives me an error. I don't see why it's giving me an error when syntax wise, it looks ok.

Jenkinsfile code snapshot:

 stages {
     stage("Checkout Dependent Code") {
         steps {
             sh  '''
                     set +x
                     echo -e "\n-- Checking out dependent code:"

                     echo -e "\n-- Cloning Common Code.\n"
                     git clone -b ${COMMON_REPO_BRANCH} ${SSH_GIT_URL}/project/common_utilities.git

                     ## Comment 1. Old repo
                     echo -e "\n-- Cloning Exporter Tool\n"
                     git clone -b ${TOOL_REPO_BRANCH} ${SSH_GIT_URL}/project/jira-exporter.git

                     ## Comment 2. New - 3 new repos. Comment the code for now.
                     #echo -e "\n-- Cloning Some Exporter Tool Repos\n"
                     #for r in core client extractor;
                     #do
                     #    echo -e "\n   -- Cloning: ${r}"
                     #    git clone -b ${TOOL_REPO_BRANCH} ${SSH_GIT_URL}/project/${r}.git
                     #    echo
                     #done

                     echo -e "\n\n`echo -en "\n-- Current Directory: "; pwd; echo; ls -l`\n\n"
                 '''
         }
     }
 }

The error message that I'm getting is:

10:38:38 -- Checking out dependent code:
10:38:38 
10:38:38 -- Cloning Common Code.
10:38:38 
10:38:38 Cloning into 'common_utilities'...
10:38:39 
10:38:39 -- Cloning Exporter Tool
10:38:39 
10:38:39 Cloning into 'jira-exporter'...
10:38:39 /jenkins_workspaces/workspace/development/project/my_jenkins_job/251@tmp/durable-f88d4c2a/script.sh: line 21: --: command not found

Question:

The first comment (Comment 1 line) in the shell script logic, is honored by Jenkinsfile and no syntax issue there and we can see the output the following echo and git command works (I can see in my workspace, git repository has been cloned successfully).

Starting 2nd comment Comment 2. line onwards, the next few lines are all commented out in Shell logic but script fails for a line (which is commented out) somewhere before the last echo line where I'm printing Current Directory: .. line and this last echo line is not printed at all (as error happened before reaching this last echo line. If all lines before this last echo line were commented, then why did I get an error. Running the shell code (from a file) works fine on the machine.


Solution

So after some digging, I found this:

The problem is \n and how it's treated in Groovy and Shell code.

In Jenkinsfile (Groovy) code, when it reads the code for the above mentioned stage (in the post), it says, oh I got this SHELL sh ''' .... ''' code and I'll go parse this in my own fancy Groovy world and create a dynamic temporary shell script (aka ..@tmp/durable-f88d4c2a/script.sh).

First Groovy comes in (due to Jenkinsfile written in Groovy) and breaks all statements wrapped in sh ''' ... ''' code into a new line and then, Jenkins creates a dynamic .sh Shell script (to do the work).

This works if a command which was using \n was NOT commented out, but doesn't work if the command using \n was ACUTUALLY COMMENTED out.

To investigate the issue, I added the following line inside sh ''' ... ''' section after set +x line.

I added cat -n $0 (just after set +x line, to print the whole shell dynamic temp .sh script) and to see HOW Jenkinsfile Groovy logic actually parsed this visual looking sh ''' ... ''' code in my Jenkinsfile ---> into a true Shell .sh script before executing it.

I found that, it treated/broke/parsed sh ''' ... ''' section; for example, a line like:

echo -e "\n-- Cloning Exporter Tool\n" into the following code lines (in the dynamically created .sh script, created by Jenkins):

echo -e " on a line in that dynamic temprarory .sh script then

-- Cloning Exporter Tool in another line, then

" in another line.

Now, the above code works fine as " double quotes are maintained and -- actually has a valid echo command which uses this -- characters to print on console output.



When the line was:

#echo -e "\n-- Cloning Exporter Tool\n" into the following code:

Groovy parsed it into a dynamic temp .sh file as:

#echo -e " on a line in that dynamic temprarory .sh script then

-- Cloning Exporter Tool in another line, then

" in another line.

and here it barfs as expected (as per shell logic) and there's no COMMAND (wrapper like echo as it's commented out) for -- ... to work. Thus, we got -- command not found

10:38:38     19                         ## Comment 2. New - 3 new repos. Comment the code for now.
10:38:38     20                         #echo -e "
10:38:38     21 -- Cloning Some Exporter Tool Repos
10:38:38     22 "

Zeeesus!

Conclusion: You must use \n carefully in sh ''' ... ''' section of Jenkinsfile, either for comments or real code lines otherwise, you'll see <some character/word> command not found error. Or, better call the above code lines via a script available in some version control tool (GIT) (rather than putting all those code lines in Jenkinsfile itself).



Answered By - AKS
Answer Checked By - Terry (WPSolving Volunteer)
at