Three jq Tips You May Need to Know

jq is famously known as a powerful command-line JSON parser. Besides performing as a parser, jq has many other handy usages as well. Here, I'd like to share 3 jq tips you may need to know.

JSON Prettier

When we curl API to fetch JSON response with raw data. It certainly looks messy:

$ curl https://api.spacexdata.com/v4/launches/latest
{"fairings":{"reused":false,"recovery_attempt":true,"recovered":null,"ships":["5ea6ed2e080df4000697c907","5ea6ed2e080df4000697c908"]},..."id":"5ef6a2e70059c33cee4a8293"}

Now let's toss raw data into jq and make it look prettier:

$ curl https://api.spacexdata.com/v4/launches/latest | jq
{
  "fairings": {
    "reused": false,
    "recovery_attempt": true,
    "recovered": null,
    "ships": [
      "5ea6ed2e080df4000697c907",
      "5ea6ed2e080df4000697c908"
    ]
  },
  ...
  "id": "5ef6a2e70059c33cee4a8293"
}

In the opposite way, when we got some pretty-printed JSON data and want to concatenate them to a one-line string, using jq -c:

$ curl https://api.spacexdata.com/v4/launches/latest | jq > example.json

$ cat example.json
{
  "fairings": {
    "reused": false,
    "recovery_attempt": true,
    "recovered": null,
    "ships": [
      "5ea6ed2e080df4000697c907",
      "5ea6ed2e080df4000697c908"
    ]
  },
  ...
  "id": "5ef6a2e70059c33cee4a8293"
}

$ cat example.json | jq -c
{"fairings":{"reused":false,"recovery_attempt":true,"recovered":null,"ships":["5ea6ed2e080df4000697c907","5ea6ed2e080df4000697c908"]},..."id":"5ef6a2e70059c33cee4a8293"}

JSON Validator

jq can be served as a simple JSON validator, for instance:

$ echo '{"fairings":{"reused":false,, "recovery_attempt":true}}' | jq
parse error: Expected value before ',' at line 1, column 29

Because its output doesn't give much information to pinpoint directly the error in an obvious way, it's better to use a proper validator for some complex cases. But for some quick checks to validate JSON data, jq can be considered as a suitable choice.

JSON Pathfinder

Analyzing JSON files and finding the correct path syntax to a certain value sometimes can be troublesome. Many times, I was punched hard by the frustration of trial and error. Until one day, this magic jq command line brings me the light: jq -r 'paths(scalars) as $p | "." + ([([$p[] | tostring] | join(".")), (getpath($p) | tojson)] | join(": "))'.

Let's use the same curl data from the previous example, in this case, I want to get the path syntax of ships ids:

$ curl https://api.spacexdata.com/v4/launches/latest \
| jq -r 'paths(scalars) as $p | "." + ([([$p[] | tostring] | join(".")), (getpath($p) | tojson)] | join(": "))' \
| grep '.ships'
.fairings.ships.0: "5ea6ed2e080df4000697c907"
.fairings.ships.1: "5ea6ed2e080df4000697c908"
.ships.0: "5ea6ed2f080df4000697c910"
.ships.1: "5ea6ed2f080df4000697c90b"
.ships.2: "5ea6ed2e080df4000697c907"
.ships.3: "5ea6ed2e080df4000697c908"

From the output, now I know the right path syntax is .ships[].

As you can see from this example, this magic jq command line will print each path and value pair from the input JSON. Combing with grep command, it's sufficient to be used as a handy JSON pathfinder.

If you do know other hot jq tips, do not hesitate to share them with me.