Parse JSON Responses Using Bash Scripts | NOC.org

Why Parse JSON in Bash?

Modern APIs return data in JSON format. Whether you are querying a NOC.org API, pulling data from cloud provider endpoints, or processing webhook payloads, you often need to extract specific fields from JSON responses within bash scripts. While bash itself has no built-in JSON parser, several tools make this task straightforward.

Installing jq

jq is the de facto standard for JSON processing on the command line. It is lightweight, fast, and available in every major package manager:

# Ubuntu / Debian
sudo apt install jq

# CentOS / RHEL / Fedora
sudo yum install jq
# or
sudo dnf install jq

# macOS
brew install jq

# Verify installation
jq --version

Basic jq Syntax

jq reads JSON from standard input and applies a filter expression to produce output. The simplest filter is . which outputs the entire JSON document, pretty-printed:

echo '{"name":"NOC.org","type":"infrastructure"}' | jq '.'

{
  "name": "NOC.org",
  "type": "infrastructure"
}

Extracting a Single Field

echo '{"name":"NOC.org","type":"infrastructure"}' | jq '.name'

"NOC.org"

To get the raw string without quotes, use the -r (raw output) flag:

echo '{"name":"NOC.org","type":"infrastructure"}' | jq -r '.name'

NOC.org

Nested Objects

echo '{"server":{"hostname":"web01","ip":"10.0.0.1","status":"active"}}' | jq '.server.ip'

"10.0.0.1"

Multiple Fields

echo '{"name":"web01","ip":"10.0.0.1","status":"active"}' | jq '{name: .name, status: .status}'

{
  "name": "web01",
  "status": "active"
}

Working with Arrays

JSON arrays are common in API responses that return lists of items:

DATA='[{"name":"web01","status":"active"},{"name":"web02","status":"inactive"},{"name":"web03","status":"active"}]'

# Get all items
echo "$DATA" | jq '.[]'

# Get the first item
echo "$DATA" | jq '.[0]'

# Get a specific field from all items
echo "$DATA" | jq '.[].name'

# Get the count of items
echo "$DATA" | jq 'length'

Filtering Arrays

Use select() to filter array elements based on conditions:

# Get only active servers
echo "$DATA" | jq '.[] | select(.status == "active")'

# Get names of active servers only
echo "$DATA" | jq -r '.[] | select(.status == "active") | .name'

web01
web03

Mapping and Transforming

# Create a new array with transformed data
echo "$DATA" | jq '[.[] | {server: .name, running: (.status == "active")}]'

[
  {"server": "web01", "running": true},
  {"server": "web02", "running": false},
  {"server": "web03", "running": true}
]

Parsing curl API Responses

The most common use case is parsing JSON returned by API calls made with curl:

# Fetch and parse in one pipeline
curl -s https://api.example.com/servers | jq '.data[].hostname'

# Store response and parse multiple fields
RESPONSE=$(curl -s https://api.example.com/servers)
echo "$RESPONSE" | jq -r '.data[] | "\(.hostname) \(.ip) \(.status)"'

Handling API Pagination

#!/bin/bash
# Fetch all pages from a paginated API

PAGE=1
TOTAL_PAGES=1

while [ $PAGE -le $TOTAL_PAGES ]; do
    RESPONSE=$(curl -s "https://api.example.com/servers?page=$PAGE")
    TOTAL_PAGES=$(echo "$RESPONSE" | jq '.pagination.total_pages')

    # Process items on this page
    echo "$RESPONSE" | jq -r '.data[] | "\(.hostname): \(.status)"'

    PAGE=$((PAGE + 1))
done

Practical API Example: CDN/WAF API

Here is an example of parsing responses from a CDN/WAF API:

#!/bin/bash
# List all domains and their current status

API_KEY="your-api-key"
API_URL="https://api.noc.org/v1"

# Fetch domain list
DOMAINS=$(curl -s -H "Authorization: Bearer $API_KEY" "$API_URL/domains")

# Check if the request was successful
if echo "$DOMAINS" | jq -e '.error' > /dev/null 2>&1; then
    echo "API Error: $(echo "$DOMAINS" | jq -r '.error.message')"
    exit 1
fi

# Parse and display domain information
echo "Domain Status Report"
echo "===================="
echo "$DOMAINS" | jq -r '.data[] | "  \(.domain) - \(.status) (SSL: \(.ssl_status))"'

# Count active vs inactive
ACTIVE=$(echo "$DOMAINS" | jq '[.data[] | select(.status == "active")] | length')
TOTAL=$(echo "$DOMAINS" | jq '.data | length')
echo ""
echo "Active: $ACTIVE / $TOTAL"

Using python -m json.tool

If jq is not available and you cannot install it, Python (which is pre-installed on most Linux systems) provides a basic JSON formatter:

# Pretty-print JSON
echo '{"name":"NOC.org","type":"infrastructure"}' | python3 -m json.tool

{
    "name": "NOC.org",
    "type": "infrastructure"
}

For extracting specific fields without jq, use Python inline:

# Extract a field using Python
echo '{"name":"NOC.org","type":"infrastructure"}' | python3 -c "
import sys, json
data = json.load(sys.stdin)
print(data['name'])
"

NOC.org

While functional, this approach is more verbose than jq and is best reserved for environments where installing additional packages is not an option.

Looping Through JSON Arrays in Bash

A common pattern is iterating over JSON array elements in a bash loop:

#!/bin/bash
# Process each server from an API response

RESPONSE='[{"name":"web01","ip":"10.0.0.1"},{"name":"web02","ip":"10.0.0.2"},{"name":"web03","ip":"10.0.0.3"}]'

# Method 1: Loop using jq with -c (compact output, one object per line)
echo "$RESPONSE" | jq -c '.[]' | while read -r SERVER; do
    NAME=$(echo "$SERVER" | jq -r '.name')
    IP=$(echo "$SERVER" | jq -r '.ip')
    echo "Checking $NAME at $IP..."
    ping -c 1 -W 2 "$IP" > /dev/null 2>&1 && echo "  UP" || echo "  DOWN"
done

# Method 2: Extract values directly for simple iterations
for NAME in $(echo "$RESPONSE" | jq -r '.[].name'); do
    echo "Processing server: $NAME"
done

Error Handling

Robust scripts must handle cases where the JSON is invalid, the expected field does not exist, or the API returns an error:

#!/bin/bash
# Safe JSON parsing with error handling

RESPONSE=$(curl -s https://api.example.com/status)

# Check if curl succeeded
if [ $? -ne 0 ]; then
    echo "ERROR: curl request failed"
    exit 1
fi

# Check if response is valid JSON
if ! echo "$RESPONSE" | jq empty 2>/dev/null; then
    echo "ERROR: Invalid JSON response"
    echo "Response was: $RESPONSE"
    exit 1
fi

# Check for null or missing fields
STATUS=$(echo "$RESPONSE" | jq -r '.status // "unknown"')
if [ "$STATUS" = "null" ] || [ "$STATUS" = "unknown" ]; then
    echo "WARNING: Status field is missing or null"
fi

echo "Service status: $STATUS"

The // operator in jq provides a default value when the field is null or missing. This is the alternative operator, and it is essential for defensive scripting.

Advanced jq Techniques

Conditional Logic

# If-then-else in jq
echo '{"status":"critical","cpu":95}' | jq 'if .cpu > 90 then "ALERT: High CPU" else "OK" end'

String Interpolation

echo '{"name":"web01","ip":"10.0.0.1"}' | jq -r '"Server \(.name) has IP \(.ip)"'

Server web01 has IP 10.0.0.1

Sorting and Grouping

# Sort by a field
echo "$DATA" | jq 'sort_by(.name)'

# Group by status
echo "$DATA" | jq 'group_by(.status) | map({status: .[0].status, count: length})'

Combining Multiple JSON Inputs

# Merge two JSON objects
jq -s '.[0] * .[1]' file1.json file2.json

Summary

Parsing JSON in bash scripts is a routine task for anyone working with APIs and modern infrastructure. jq is the tool of choice — it is fast, expressive, and handles everything from simple field extraction to complex array transformations and conditional logic. For environments where jq is unavailable, Python's json.tool module provides a basic alternative. Always include error handling in production scripts to deal with invalid JSON, missing fields, and API failures gracefully. For more on working with NOC.org's APIs, see the CDN/WAF API documentation and our getting started guide.