[SOLVED] Prettify json in a streaming log

Issue

I am trying to prettify aws logs stream

log format:

2022-09-07T17:14:35.987000+00:00 2022/09/07/[$LATEST]58f517bs4ebcsae033sd953bs734224 {"cold_start":true,"function_memory_size":512,"function_request_id":"a2cfdd5d-4b0b-4cfa-97c2-4af9ba6a1056","level":"ERROR","message":"Unexpected","sampling_rate":1,"timestamp":"2022-09-07T17:14:35.987Z","xray_trace_id":"1-6318d178-0e96c13a1399a74b4f353edd","extra":"{\"error\":\"AccessDeniedException: User: arn:aws:sts::\",\"stack\":\"AccessDeniedException: User: arn:aws:sts::\"}"}

I want to change this to

2022-09-07T17:14:35.987000+00:00 2022/09/07/[$LATEST]58f51004277b4ebcae033d953b734224 {
  "cold_start": true,
  "function_memory_size": 512,
  "function_request_id": "a2cfdd5d-4b0b-4cfa-97c2-4af9ba6a1056",
  "level": "ERROR",
  "message": "Unexpected",
  "sampling_rate": 1,
  "timestamp": "2022-09-07T17:14:35.987Z",
  "xray_trace_id": "1-6318d178-0e96c13a1399a74b4f353edd",
  "extra": '{"error":"AccessDeniedException: User: arn:aws:sts::","stack":"AccessDeniedException: User: arn:aws:sts::"}',
}

I tried this

aws logs tail /aws/lambda/lambda-name | sed -u -e "s/\({.*}\)/$(echo \'\\1\' | jq)/"

but I am getting this error

parse error: Invalid numeric literal at line 2, column 0

How do I do this?

Solution

If you feed your log sample to:

jq -rR 'index("{") as $ix | .[0:$ix], ( .[$ix:]|fromjson)'

you will get the log stamp followed by a copy of the corresponding valid JSON.

Your subject line indicates you want to "prettify" the JSON rather than mangle it, so I'll assume you didn't intend the latter.

If you really want the initial '{' to appear with the log stamp, then assuming a bash or bash-like shell, you could pipe the above into:

    sed $'/^[0-9][0-9][0-9][0-9]-/ {N; s/\\n{/{/; }'

Answered By - peak

Answer Checked By - Marilyn (WPSolving Volunteer)