jq-json-processing

# jq JSON Processing Expert knowledge for processing, querying, and transforming JSON data using jq, the lightweight and flexible command-line JSON processor. ## Core Expertise **JSON Operations** - Query and filter JSON with path expressions - Transform JSON structure and shape - Combine, merge, and split JSON data - Validate JSON syntax and structure **Data Extraction** - Extract specific fields from JSON objects - Filter arrays based on conditions - Navigate nested JSON structures - Handle null values and missing keys ## Essential Commands ### Basic Querying ```bash # Pretty-print JSON jq '.' file.json # Extract specific field jq '.fieldName' file.json # Extract nested field jq '.user.email' file.json # Extract array element jq '.[0]' file.json jq '.items[2]' file.json ``` ### Array Operations ```bash # Get array length jq '.items | length' file.json # Map over array jq '.items | map(.name)' file.json # Filter array jq '.items[] | select(.active == true)' file.json jq '.users[] | select(.age > 18)' file.json # Sort array jq '.items | sort_by(.name)' file.json jq '.items | sort_by(.date) | reverse' file.json # Get first/last elements jq '.items | first' file.json jq '.items | last' file.json # Unique values jq '.tags | unique' file.json ``` ### Object Operations ```bash # Get all keys jq 'keys' file.json jq '.config | keys' file.json # Get all values jq '.[] | values' file.json # Select specific fields jq '{name, email}' file.json jq '{name: .fullName, id: .userId}' file.json # Add field jq '. + {newField: "value"}' file.json # Delete field jq 'del(.password)' file.json # Merge objects jq '. * {updated: true}' file.json ``` ### Filtering and Conditions ```bash # Select with conditions jq 'select(.status == "active")' file.json jq '.[] | select(.price < 100)' file.json # Multiple conditions (AND) jq '.[] | select(.active and .verified)' file.json jq '.[] | select(.age > 18 and .country == "US")' file.json # Multiple conditions (OR) jq '.[] | select(.type == "admin" or .type == "moderator")' file.json # Exists / has field jq '.[] | select(has("email"))' file.json jq 'select(.optional != null)' file.json # Not condition jq '.[] | select(.status != "deleted")' file.json ``` ### String Operations ```bash # String interpolation jq '"Hello, \(.name)"' file.json # Convert to string jq '.id | tostring' file.json # String contains jq '.[] | select(.email | contains("@gmail.com"))' file.json # String starts/ends with jq '.[] | select(.name | startswith("A"))' file.json jq '.[] | select(.file | endswith(".json"))' file.json # Split string jq '.path | split("/")' file.json # Join array to string jq '.tags | join(", ")' file.json # Lowercase/uppercase jq '.name | ascii_downcase' file.json jq '.name | ascii_upcase' file.json ``` ### Pipes and Composition ```bash # Chain operations jq '.items | map(.name) | sort | unique' file.json # Multiple filters jq '.users[] | select(.active) | select(.age > 18) | .email' file.json # Group by jq 'group_by(.category)' file.json jq 'group_by(.status) | map({status: .[0].status, count: length})' file.json ``` ### Output Formatting ```bash # Compact output (no pretty-print) jq -c '.' file.json # Raw output (no quotes for strings) jq -r '.message' file.json jq -r '.items[] | .name' file.json # Output as tab-separated values jq -r '.[] | [.name, .age, .email] | @tsv' file.json # Output as CSV jq -r '.[] | [.name, .age, .email] | @csv' file.json # Output as JSON array on one line jq -c '[.items[]]' file.json ``` ### Input/Output Options ```bash # Read from stdin cat file.json | jq '.items' curl -s https://api.example.com/data | jq '.results' # Multiple input files jq -s '.' file1.json file2.json # Slurp into array # Write to file jq '.filtered' input.json > output.json # In-place edit (use sponge from moreutils) jq '.updated = true' file.json | sponge file.json ``` ### Advanced Patterns ```bash # Recursive descent jq '.. | .email? // empty' file.json # Reduce (sum, accumulate) jq '[.items[].price] | add' file.json jq 'reduce .items[] as $item (0; . + $item.price)' file.json # Variable assignment jq '.items[] | . as $item | $item.name + " - " + ($item.price | tostring)' file.json # Conditional (if-then-else) jq '.items[] | if .price > 100 then "expensive" else "affordable" end' file.json jq 'if .error then .error else .data end' file.json # Try-catch for error handling jq '.items[] | try .field catch "not found"' file.json # Flatten nested arrays jq '.items | flatten' file.json jq '.items | flatten(1)' file.json # Flatten one level ``` ### Real-World Examples ```bash # Extract all emails from nested structure jq '.. | .email? // empty' users.json # Get unique list of all tags across items jq '[.items[].tags[]] | unique' data.json # Count items by status jq 'group_by(.status) | map({status: .[0].status, count: length})' items.json # Transform API response to simple list jq '.results[] | {id, name: .full_name, active: .is_active}' response.json # Filter GitHub workflow runs (recent failures) gh run list --json status,conclusion,name,createdAt | \ jq '.[] | select(.conclusion == "failure") | {name, createdAt}' # Extract package.json dependencies with versions jq '.dependencies | to_entries | map("\(.key)@\(.value)")' package.json # Merge two JSON files jq -s '.[0] * .[1]' base.json override.json # Create summary from log data jq 'group_by(.level) | map({level: .[0].level, count: length, samples: [.[].message][:3]})' logs.json ``` ## Best Practices **Query Construction** - Start simple, build complexity incrementally - Test filters on small datasets first - Use `-c` flag for compact output in scripts - Use `-r` flag for raw strings (no quotes) **Performance** - Use `select()` early in pipeline to reduce data - Use targeted queries for large files (instead of `..` recursive descent) - Stream large JSON with `--stream` flag - Consider `jq -c` for faster processing **Error Handling** - Use `?` operator for optional access: `.field?` - Use `// empty` to filter out nulls/errors - Use `try-catch` for graceful error handling - Check for field existence with `has("field")` **Readability** - Break complex queries into multiple steps - Use variables with `as $var` for clarity - Add comments in shell scripts - Format multi-line jq programs for readability ## Common Patterns ### API Response Processing ```bash # GitHub API: Get PR titles and authors gh pr list --json title,author,number | \ jq -r '.[] | "#\(.number) - \(.title) by @\(.author.login)"' # REST API: Extract and flatten pagination curl -s "https://api.example.com/items" | \ jq '.data.items[] | {id, name, status}' ``` ### Configuration Files ```bash # Extract environment-specific config jq '.environments.production' config.json # Update configuration value jq '.settings.timeout = 30' config.json > config.updated.json # Merge base config with environment overrides jq -s '.[0] * .[1]' base-config.json prod-config.json ``` ### Log Analysis ```bash # Count errors by type jq 'select(.level == "error") | .type' logs.json | sort | uniq -c # Extract error messages with timestamps jq -r 'select(.level == "error") | "\(.timestamp) - \(.message)"' logs.json # Group by hour and count jq -r '.timestamp | split("T")[1] | split(":")[0]' logs.json | sort | uniq -c ``` ### Data Transformation ```bash # CSV to JSON (with headers) jq -R -s 'split("\n") | .[1:] | map(split(",")) | map({name: .[0], age: .[1], email: .[2]})' data.csv # JSON to CSV jq -r '.[] | [.name, .age, .email] | @csv' data.json # Flatten nested structure jq '[.items[] | {id, name, category: .meta.category}]' nested.json ``` ## Troubleshooting ### Invalid JSON ```bash # Validate JSON syntax jq empty file.json # Returns exit code 0 if valid # Find syntax errors jq '.' file.json 2>&1 | grep "parse error" ``` ### Empty Results ```bash # Debug: Print entire structure jq '.' file.json # Debug: Check field existence jq 'keys' file.json jq 'type' file.json # Check if array, object, etc. # Debug: Show all values jq '.. | scalars' file.json ``` ### Type Errors ```bash # Check field types jq '.field | type' file.json # Convert types safely jq '.id | tonumber' file.json jq '.count | tostring' file.json # Handle mixed types jq '.items[] | if type == "array" then .[] else . end' file.json ``` ### Performance Issues ```bash # Stream large files jq --stream '.' large-file.json # Process line by line cat large.json | jq -c '.[]' | while read -r line; do echo "$line" | jq '.field' done ``` ## Integration with Other Tools ```bash # With curl (API calls) curl -s "https://api.github.com/users/octocat" | jq '.name, .bio' # With gh CLI (GitHub operations) gh api repos/owner/repo/issues | jq '.[] | {number, title, state}' # With find (batch processing) find . -name "package.json" -exec jq '.version' {} \; # With xargs (parallel processing) cat urls.txt | xargs -I {} curl -s {} | jq '.data' # With yq (YAML to JSON conversion) yq eval -o=json file.yaml | jq '.specific.field' ``` ## Quick Reference ### Operators - `.field` - Access field - `.[]` - Iterate array/object - `|` - Pipe (chain operations) - `,` - Multiple outputs - `?` - Optional (suppress errors) - `//` - Alternative operator (default value) ### Functions - `keys`, `values` - Object keys/values - `length` - Array/object/string length - `map()`, `select()` - Array operations