[SOLVED] Make Azure devops pipeline not to read the output of remotly running script

Issue

I have created some script that's checking if there is a newer version of the image inside AWS ECR, and if it does pull it, and remove the previous.

I have created an azure-pipeline.yml script that's running after push a new version to my repo.

On the last step of it, I am executing an SSH command to another machine, and executing the script I have mentioned above.

## Docker
# Build a Docker image
# https://docs.microsoft.com/azure/devops/pipelines/languages/docker

trigger:
  - master

resources:
  - repo: self

variables:
  - group: AWS_ECR

stages:
  - stage: Build
    displayName: Build image
    jobs:
      - job: Build
        displayName: Build
        pool:
          vmImage: ubuntu-latest
        steps:
          - checkout: self
            fetchDepth: 0
          - task: gitversion/setup@0
            displayName: Install GitVersion
            inputs:
              versionSpec: "5.10.x"

          - task: gitversion/execute@0
            displayName: Determine Version
            inputs:
              useConfigFile: true
              configFilePath: ./GitVersion.yml
          - script: |
              aws ecr get-login-password --region eu-central-1 | docker login --username AWS --password-stdin $(AWS_ACCOUNT_ID).dkr.ecr.$(AWS_REGION).amazonaws.com
            displayName: 'Login to AWS'
            env:
              AWS_ACCESS_KEY_ID: $(AWS_ACCESS_KEY_ID)
              AWS_SECRET_ACCESS_KEY: $(AWS_SECRET_ACCESS_KEY)
          - task: Docker@2
            displayName: Build an image
            inputs:
              repository: $(BACKEND_REPOSITORY_URI)
              command: buildAndPush
              dockerfile: '$(Build.SourcesDirectory)/Dockerfile'
              tags: |
                $(GitVersion.FullSemVer)
                latest
          - task: SSH@0
            displayName: 'Run updateImage script'
            inputs:
              sshEndpoint: 'ssh'
              runOptions: inline
              inline: |
                cd /home/ubuntu/composer
                nohup ./updateImage.sh $(AWS_ACCOUNT_ID) $(AWS_REGION) $(BACKEND_REPOSITORY_NAME) &
            continueOnError: true

I have tried to run it with nohup & to get it on the background, but for some reason the pipeline is getting the output from the script.

updateImage.sh:

#!/bin/bash

# Check if required arguments are provided
if [ "$#" -lt 3 ]; then
  echo "Usage: $0 <AWS_ACCOUNT_ID> <AWS_REGION> <REPOSITORY_NAME>"
  exit 1
fi

# Assign script arguments to variables
AWS_ACCOUNT_ID=$1
AWS_REGION=$2
REPOSITORY_NAME=$3

# Navigate to the directory
cd /home/ubuntu/HighScaleComposer

# Pull the newest image
docker pull "${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/${REPOSITORY_NAME}:latest"

# Run Docker Compose with the newest image
docker compose -f docker-compose.yaml up -d && sleep 5

# Remove other images (excluding latest)
docker images --format "{{.ID}} {{.Repository}}:{{.Tag}}" | grep "${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/${REPOSITORY_NAME}" | grep -v latest | awk '{print $1}' | xargs -I {} docker rmi {}

This is the logs from the pipeline thats failed:

542bd72cf3c7:  Verifying Checksum

542bd72cf3c7:  Download complete

542bd72cf3c7: Pull complete

b57d17fdbf95: Pull complete

Digest: sha256:57abc895af88a97389baf56321ff17c9ef7b55e605aceed002e2c3be87ad019e

Status: Downloaded newer image for
***.dkr.ecr.eu-central-1.amazonaws.com/highscale_backend:latest

***.dkr.ecr.eu-central-1.amazonaws.com/highscale_backend:latest

##[error] Container frontend  Running

##[error] Container mysql_container  Running

##[error] Container backend  Recreate

##[error] Container backend  Recreated

##[error] Container backend  Starting

##[error] Container backend  Started

Untagged:
***.dkr.ecr.eu-central-1.amazonaws.com/highscale_backend@sha256:cd50c51220508921088e2871c76b64464ef4a607c7457026439486e20ff255d1

Deleted: sha256:39b2f42e37ac24ecb06a637b346a6cb6dd3ef8eacb12dd2731fe58476c2573c1

Deleted: sha256:ffc8e4f5ddfb398588b2a792ce93853d15b877a588ae179e1d5c44c9906b49f3

Deleted: sha256:a6318b95525b5d1ed1ec6a61ab9ec1b48d236d7ef3e26a2bf5542e4160c01f25

Finishing: Run updateImage script

This is the output for executing the script from the machine:

ubuntu@ip-172-31-9-59:~/composer$ ./updateImage.sh 274129698771 eu-central-1 backend
latest: Pulling from backend
96526aa774ef: Already exists
872bc298c5d2: Already exists
2a645fa80061: Pull complete
6dd5299324c9: Pull complete
Digest: sha256:cc5331ad95b28bbcb72de4e0519c24b2e5537d174a48f7304c60e355c8ca74e4
Status: Downloaded newer image for *.dkr.ecr.eu-central-1.amazonaws.com/backend:latest
*.dkr.ecr.eu-central-1.amazonaws.com/_backend:latest
[+] Running 3/3
 ✔ Container frontend         Running                                                                                                                                                                                                                   0.0s
 ✔ Container mysql_container  Running                                                                                                                                                                                                                   0.0s
 ✔ Container backend          Started                                                                                                                                                                                                                  10.6s
Untagged: *.dkr.ecr.eu-central-1.amazonaws.com/backend@sha256:e12d269915ec79709e219c4bc9d3c83e64cccf00bce9833124f607a7915ff80a
Deleted: sha256:c09bedd861bcd75329a6dc42b13e39ff0966aeadac1c95647e9f40af99bc708d
Deleted: sha256:a7e3e4a081b86994a08d544dc404dc66e6f1abcf6882b36d92c4e32f71c01ad4
Deleted: sha256:32815dad1d9b147e30fe198808ebcc546b380737d02aaf7693cb532a681f42e5

My guess it's finding that error word and per this it's failing my pipeline, how can I handle this?

Solution

You can try to use the gnome-terminal command to open a new terminal session in Bash to run the script.

For example:

gnome-terminal -x bash -c "./updateImage.sh $(AWS_ACCOUNT_ID) $(AWS_REGION) $(BACKEND_REPOSITORY_NAME); exec bash"

The 'exec bash' at the end of this command will keep the terminal session open after completing the commands. If you don't need this, you can remove it from the command.

Answered By - Bright Ran-MSFT

Answer Checked By - David Marino (WPSolving Volunteer)