Jq

jq & Data Processing

Overview

jq is "sed for JSON" — a lightweight command-line processor for JSON data. Combined with curl, it's the standard way to interact with REST APIs from the terminal. yq extends the same idea to YAML.

Installation

jq Basics

Identity (pretty-print)

echo '{"name":"Alice","age":30}' | jq '.'

Field access

echo '{"name":"Alice","address":{"city":"NYC"}}' | jq '.name'
# "Alice"

echo '{"name":"Alice","address":{"city":"NYC"}}' | jq '.address.city'
# "NYC"

Array indexing

echo '[10,20,30,40,50]' | jq '.[0]'     # 10
echo '[10,20,30,40,50]' | jq '.[-1]'    # 50
echo '[10,20,30,40,50]' | jq '.[2:5]'   # [30,40,50]

Array iteration

echo '[{"name":"Alice"},{"name":"Bob"}]' | jq '.[]'

Pipe

curl -s https://api.example.com/data | jq '.users[] | .name'

Select / filter

echo '[{"name":"Alice","age":35},{"name":"Bob","age":25}]' | jq '.[] | select(.age > 30)'

Map

echo '[{"name":"Alice"},{"name":"Bob"}]' | jq '[.[] | .name]'
# ["Alice", "Bob"]

Keys, length, type

echo '{"a":1,"b":2}' | jq 'keys'       # ["a","b"]
echo '[1,2,3]' | jq 'length'            # 3
echo '"hello"' | jq 'type'              # "string"

Construct objects

echo '{"first":"Alice","contact":{"email":"a@b.com"}}' | jq '{name: .first, email: .contact.email}'

String interpolation

echo '{"name":"Alice","email":"a@b.com"}' | jq '"\(.name) - \(.email)"'
# "Alice - a@b.com"

Common jq Recipes

Extract specific fields

jq '.[] | {name, email}' users.json

Filter by value

jq '[.[] | select(.status == "active")]' users.json

Count items

jq '.users | length' data.json

Sort by field

jq '.users | sort_by(.age)' data.json

Group by field

jq '.users | group_by(.department)' data.json

Flatten arrays

jq '[.[][]]' nested.json

JSON to CSV

jq -r '.[] | [.name, .email, .age] | @csv' users.json

Unique values

jq '[.[].category] | unique' items.json

Merge objects

jq -s '.[0] * .[1]' defaults.json overrides.json

Conditional

echo '{"age":21}' | jq 'if .age > 18 then "adult" else "minor" end'

yq Basics

Read a field

yq '.metadata.name' deployment.yaml

Update a field

yq '.spec.replicas = 3' -i deployment.yaml

Convert YAML to JSON

yq -o json deployment.yaml

Convert JSON to YAML

yq -P data.json

Merge YAML files

yq eval-all '. as $item ireduce({}; . * $item)' base.yaml override.yaml

Combining with curl

curl -s https://api.github.com/repos/torvalds/linux | jq '{name: .name, stars: .stargazers_count, language: .language}'

Output:

{
  "name": "linux",
  "stars": 178000,
  "language": "C"
}

Best Practices

• Use jq for any JSON manipulation in shell scripts — it is purpose-built and reliable
• Pipe curl -s to jq for API work — this is the standard pattern for CLI-based API interaction
• Use yq for editing Kubernetes manifests, Helm values, and CI config files in place
• Prefer jq over grep/awk for JSON — regex on JSON is fragile and error-prone
• Use -r for raw string output in scripts to avoid quoted strings in downstream processing
• Learn select() and map() — they handle the vast majority of filtering and transformation tasks