[SOLVED] Pass keys from array from bash into jq -- output all nulls

Issue

I'm trying to do something akin to this:

jq -r  '. | ."Time Series (Daily)"."2020-12-02" | ."1. open"' newdata.json

...but with the key coming from a variable, as in:

jq -r --arg key "$key" '. | ."Time Series (Daily)"."[$key]" | ."1. open"' newdata.json

The first one works just fine, but when I assign the date to a variable called key and then try to get the data, it fails.

I tried This answer and This answer. But did not work for me.

{
    "Meta Data": {
        "1. Information": "Daily Prices (open, high, low, close) and Volumes",
        "2. Symbol": "AB",
        "3. Last Refreshed": "2020-12-02",
        "4. Output Size": "Compact",
        "5. Time Zone": "US/Eastern"
    },
    "Time Series (Daily)": {
        "2020-12-02": {
            "1. open": "32.6700",
            "2. high": "33.3300",
            "3. low": "32.5000",
            "4. close": "33.1200",
            "5. volume": "273799"
        },
        "2020-12-01": {
            "1. open": "32.1500",
            "2. high": "32.8000",
            "3. low": "32.0000",
            "4. close": "32.6000",
            "5. volume": "265086"
        },
        "2020-11-30": {
            "1. open": "32.3800",
            "2. high": "32.4900",
            "3. low": "31.7500",
            "4. close": "31.8700",
            "5. volume": "251970"
        }
    }
}

The above is the newdata.json file. What I want to get is the "1. open" value. I am using a for loop to iterate over all the keys of "Time Series (Daily)" and the keys are generated correctly. There is no issue with that. I then want to use the $key variable in each iteration to get the data I need.

readarray keys <<< "$(jq  '."Time Series (Daily)" |  keys[]' newdata.json)"
for key in "${keys[@]}"; do
  jq -r --arg key "$key"  '. | ."Time Series (Daily)" | .[$key] | ."1. open"' newdata.json
done

Solution

Focusing On The Immediate Issue

The problem isn't how you're passing key to jq; the problem is how you're populating the key variable in the first place.

Change:

readarray keys <<< "$(jq  '."Time Series (Daily)" |  keys[]' newdata.json)"

...to:

readarray -t keys <<< "$(jq -r '."Time Series (Daily)" |  keys[]' newdata.json)"

There are two changes here:

1. We added the -t argument to readarray, so it no longer includes the newline ending each line in the variable itself.
2. We added the -r argument to jq, so it no longer adds literal quotes around the strings.

---

Sidebar: Retrieving both keys and values at the same time

There's no reason to do one pass to retrieve keys and another to retrieve values -- better to just get them all at once:

dates=( )
opening_prices=( )
while IFS=$'\t' read -r date opening_price; do
  dates+=( "$date" )
  opening_prices+=( "$opening_price" )
done < <(
  jq -r '
    ."Time Series (Daily)" | to_entries[] | [.key, .value."1. open"] | @tsv
  ' <newdata.json
)

...after which, declare -p dates opening_prices emits:

declare -a dates=([0]="2020-12-02" [1]="2020-12-01" [2]="2020-11-30")
declare -a opening_prices=([0]="32.6700" [1]="32.1500" [2]="32.3800")

---

Original response (before population of keys was shown)

Here's a different approach that only calls jq once, instead of once per item, while still getting your keys from an array. It does this by using -R to read raw strings as input; . is then used to address those inputs (which we rename to $key to make it clear how this lines up with the old code).

keys=("2020-12-02" "2020-12-01" "2020-11-30")
readarray -t openingPrices < <(
  jq -Rr --slurpfile indatas newdata.json '
    $indatas[0] as $indata | . as $key |
    $indata."Time Series (Daily)"[$key]["1. open"]
  ' < <(printf '%s\n' "${keys[@]}")
)

After running that, declare -p keys openingPrices (to show how both arrays are defined) emits:

declare -a keys=([0]="2020-12-02" [1]="2020-12-01" [2]="2020-11-30")
declare -a openingPrices=([0]="32.6700" [1]="32.1500" [2]="32.3800")

...so you have an output array that lines up with your input array (so long as the latter isn't sparse).

Answered By - Charles Duffy