[SOLVED] Parse curl response in bash without jq

Issue

I'm trying to parse an array, child of array, in json response returned from a curl request in bash. Unfortunately i can't use jq because it can't be installed on production servers. I've try using awk or sed in curl request but result is empty. So i'm using grep with regex but i can't iterate properly.

#!/bin/bash

PARAM=$1

# Run environment variables
export $(grep -v '^#' ${PWD}/.env | xargs)

# Step 1 : Get JWT Token
TOKEN=$(curl \
  --silent \
  --request POST "${URL_API}"/login \
  --header 'Content-Type: application/json' \
  --header 'Accept: application/json' \
  -d '{"username": "'${API_USERNAME}'", "password": "'${API_PWD}'" }' \
  | sed -n 's|.*"token":"\([^"]*\)".*|\1|p'
)

# Step 2 curl GET request and put it in json file
JSON_RESULT=$(curl \
  --silent \
  --output test.json \
  --header 'Content-Type: application/json' \
  --header 'Accept: application/json' \
  --header 'Authorization: Bearer '${TOKEN}'' \
  --request GET "${URL_API}"/sites/list?value="${PARAM}" \
)

Json response from API :

{
  "data": [
    {
      "id": 212,
      "list": [
        "value1",
        "value2",
        "value3",
        "value4"
      ],

      "createdAt": "2023-02-09T15:47:38+01:00",
      "foo": "bar",
      "foo2": "ba2r",
    }
  ],
  "metas": {
    "code": 200,
    "count": 1,
    "total": 5
  }
}

I want as array type the result of "list" key. I'm trying like this :

# Step 3 retrieve data in "list" sub-key in json
TEST=$(grep -zoP '"list":\s*\K[^\s]*(?=\s*,)' test.json)
echo ${TEST}

Display :

> scripts/test.sh: line 35: warning: command substitution: ignored null byte in input
["value1","value2","value3","value4"]

How can parse this as array for doing something like this :

# step 4 iterate
IFS=',' read -ra ADDR <<< "$TEST"
for i in "${ADDR[@]}"; do
  echo $i
done

?

Solution

With sed, using a file as input -

$: sed -nE 'H; ${x; s/[ \n]+//g; s/^.*"list":\[/\[/;s/\].*/]/;p;}' file
["value1","value2","value3","value4"]

This works even if the file has no embedded spaces or newlines.

Breakout:

-nE just stacks -n and -E.
-n says no output unless explicitly requested. -E ues Extended pattern matching, so you don't have to backslash-quote your square brackets.

In the actual command list,
H; stacks each line of input onto the Hold buffer.
No other commands are executed on any line but the last.

${...} means "on the last line, do these commands..."

x exchanges the input buffer and the Hold buffer, putting all those lines you saved into the workspace.

s/[ \n]+//g will substitute spaces and newlines with nothing globally across the buffer, stacking it all into one long line with any spaces removed.

s/^.*"list":\[/\[/ says replace everything from the start(^) through the literal list:[ with just [.

s/\].*/]/ says replace everything from the remaining first ] on with just ].

p will print the workspace buffer, which at this point should just be the data you asked for.

Answered By - Paul Hodges

Answer Checked By - Marilyn (WPSolving Volunteer)