DEV Community

ente
ente

Posted on • Originally published at ente.io

An introduction to the magic of jq

#json #beginners #tutorial #cli

jq is a magical tool for dealing with JSON on the CLI. Like its famous predecessors, sed and awk, it is unfriendly to beginners (or as the neckbeards of old would say, it is just choosy in who it considers as friends).

The unfriendliness comes in large part because of a historical tendency in the man pages of these tools to not provide examples. The man pages of all these are extremely well written, and a pleasure to read (yes, I'm that sort of a person who likes spending Sundays reading man pages). But because of a lack of examples, they are opaque to beginners.

Enough ranting, let's show some of the magic of jq.

jq examples

The task I wanted to do was, well, compare two lists. First a bit of background: we have a security group on Scaleway that restricts inbound connections to our API server instances. The only entity we want to be able to open connections to these is Cloudflare, which provides the load balancer that api.ente.io resolves to, and then reverse proxies incoming connections to our API server instances. So the IP ranges from which Cloudflare can connect to our instances should be whitelisted in our Scaleway security group.

So my goal is to ensure that the two lists -- the allowed IP ranges in the security group, and Cloudflare's IP ranges -- match.

This is perfect for demonstrating the power of jq. We'll see filtering, joining, sorting, and more. The hope is that with a real example (this is something I actually needed to do), the usefulness of jq is even more apparent.

First, let's start off by getting the ID of our security group. We'll use the Scaleway CLI tool (called scw).

$ scw instance security-group list name=scg-prod
ID                                    NAME      DESCRIPTION
xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx  scg-prod  Security group - ...
Enter fullscreen mode Exit fullscreen mode

We want the output in JSON format, for which the scw command provides the handy --output json switch.

$ scw instance security-group list --output json name=scg-prod
[{"id":"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx","name":"scg-prod",...}]
Enter fullscreen mode Exit fullscreen mode

We get big blob of JSON, machine readable, but unreadable to us. Let's prettify it by piping it into jq.

$ scw instance security-group list --output json name=scg-prod | jq
[
  {
    "id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
    "name": "scg-prod",
    ...
   }
]
Enter fullscreen mode Exit fullscreen mode

If you take nothing else away from this post, just remember that | jq is a quick way to prettify JSON.

Moving on though. We only want the "id" of the security group (we need it for the next command that gets the security group details). For doing this, we can use .id to get jq to only emit the value of the "id" field in the JSON object we give it.

The JSON we see above is an array, but we only care about the first element, so we can use first to get at that. Combining the first (to get the first element), and .id (to get the value of "id" from within that element), we get

$ scw instance security-group list --output json name=scg-prod | jq 'first.id'
"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
Enter fullscreen mode Exit fullscreen mode

We just want the value without the quotes, so let us ask jq to give us raw output by specifying the "--raw-output" flag (or its shorthand, "-r").

$ scw instance security-group list --output json name=scg-prod | jq -r 'first.id'
xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Enter fullscreen mode Exit fullscreen mode

Perfect.

Now let's get the details of this security group. We could copy paste the output above, but we're cooler than that. So we'll use the $( ) syntax to run a
subcommand whose result is then used as the argument to our top level command.

$ scw instance security-group get "$(scw instance security-group list --output json name=scg-prod | jq -r 'first.id')"
Security Group:
ID                     xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
...
Enter fullscreen mode Exit fullscreen mode

Again we want JSON output, and to make it easier to inspect we want to prettify it by passing it to jq

$ scw instance security-group get --output json "$(scw instance security-group list --output json name=scg-prod | jq -r 'first.id')" | jq
{
  "id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
  "name": "scg-prod",
  ...
  "Rules": [
    {
      "id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
      "protocol": "TCP",
      "direction": "outbound",
      "action": "drop",
      "ip_range": "0.0.0.0/0",
      "dest_port_from": 465,
      "dest_port_to": null,
      "position": 2,
      "editable": false,
      "zone": "fr-par-1"
    },
    ...
Enter fullscreen mode Exit fullscreen mode

So what we have is an JSON object representing the security group. It has an array called "Rules", each of which is a security group rule to apply.

We're interested only in the rules, so we can ask jq to only give us the Rules array by using

jq '.Rules'
Enter fullscreen mode Exit fullscreen mode

Now, of all the rules we're only interested in the "inbound" rules (not the "outbound" ones). So we ask jq to select only those rules which have "direction" equal to "inbound" by saying select(.direction == "inbound").

What select does is only allow only those objects that satisfy the given criteria to pass, the rest are discarded. However, select works on objects; Rules is an array. So we need a way to apply select to each object in the Rules
array. Ta-da! map!

jq '.Rules | map(select(.direction == "inbound"))'
Enter fullscreen mode Exit fullscreen mode

We also want to filter only those rules that are for port 443 (the HTTPS port where our API instances are listening on).

jq '.Rules | map(select(.direction == "inbound" and .dest_port_from == 443))'
Enter fullscreen mode Exit fullscreen mode

Finally, this will give us all the Rules objects that satisfy the condition, but we only want the IP ranges from each rule. So let's add another step in the pipeline to extract only the "ip_range" from each of the filtered rules.

jq '.Rules | map(select(.direction == "inbound" and .dest_port_from == 443)) | map(.ip_range)'
Enter fullscreen mode Exit fullscreen mode

The last two steps are both maps, and they can be combined to still get the same effect.

jq '.Rules | map(select(.direction == "inbound" and .dest_port_from == 443) | .ip_range)'
Enter fullscreen mode Exit fullscreen mode

Let us also sort the ranges (this will simplify the comparison later).

jq '.Rules | map(select(.direction == "inbound" and .dest_port_from == 443) | .ip_range) | sort'
Enter fullscreen mode Exit fullscreen mode

In full context,

$ scw instance security-group get --output json "$(scw instance security-group list --output json name=scg-prod | jq -r 'first.id')" | jq '.Rules | map(select(.direction == "inbound" and .dest_port_from == 443) | .ip_range) | sort'
[
  "103.21.244.0/22",
  "103.22.200.0/22",
  "103.31.4.0/22",
  "104.16.0.0/13",
  "104.24.0.0/14",
  "108.162.192.0/18",
  "131.0.72.0/22",
  "141.101.64.0/18",
  "162.158.0.0/15",
  "172.64.0.0/13",
  "173.245.48.0/20",
  "188.114.96.0/20",
  "190.93.240.0/20",
  "197.234.240.0/22",
  "198.41.128.0/17",
  "2400:cb00::/32",
  "2405:8100::/32",
  "2405:b500::/32",
  "2606:4700::/32",
  "2803:f800::/32",
  "2a06:98c0::/29",
  "2c0f:f248::/32"
]
Enter fullscreen mode Exit fullscreen mode

Brilliant!

So now let us get the other half of the data -- the list of Cloudflare IP addresses. These can be obtained from the Cloudflare API.

$ curl -s 'https://api.cloudflare.com/client/v4/ips'
{"result":{"ipv4_cidrs":...
Enter fullscreen mode Exit fullscreen mode

Again, let's pipe it to jq to get some idea what the JSON structure is

$ curl -s 'https://api.cloudflare.com/client/v4/ips' | jq
{
  "result": {
    "ipv4_cidrs": [
      "173.245.48.0/20",
      ...
    ],
    "ipv6_cidrs": [
      "2400:cb00::/32",
      ...
    ],
    ...
  },
  "success": true,
  "errors": [],
  "messages": []
}
Enter fullscreen mode Exit fullscreen mode

So we're interested in the "ipv4_cidrs" and "ipv6_cidrs" arrays inside the "result" object of the response JSON.

jq has an interesting comma operator that duplicates th incoming stream and sends it to both operations of a comma. So we can in effect get both "ipv4_cidrs" and "ipv6_cidrs" from the same result object.

jq '.result | .ipv4_cidrs,.ipv6_cidrs'
Enter fullscreen mode Exit fullscreen mode

This will, however, give us two arrays -- the "ipv4_cidrs" array, and the "ipv6_cidrs" array. What we want is a single array. We can flatten the two arrays into a single one using the "flatten" operator. However, before flattening we'll need to put both the arrays into a single container array. This
is easy - just wrapping them in [].

jq '.result | [.ipv4_cidrs,.ipv6_cidrs] | flatten'
Enter fullscreen mode Exit fullscreen mode

Finally, let's also sort them

jq '.result | [.ipv4_cidrs,.ipv6_cidrs] | flatten | sort'
Enter fullscreen mode Exit fullscreen mode

In full context,

$ curl -s 'https://api.cloudflare.com/client/v4/ips' | jq '.result | [.ipv4_cidrs,.ipv6_cidrs] | flatten | sort'
[
  "103.21.244.0/22",
  "103.22.200.0/22",
  "103.31.4.0/22",
  "104.16.0.0/13",
  "104.24.0.0/14",
  "108.162.192.0/18",
  "131.0.72.0/22",
  "141.101.64.0/18",
  "162.158.0.0/15",
  "172.64.0.0/13",
  "173.245.48.0/20",
  "188.114.96.0/20",
  "190.93.240.0/20",
  "197.234.240.0/22",
  "198.41.128.0/17",
  "2400:cb00::/32",
  "2405:8100::/32",
  "2405:b500::/32",
  "2606:4700::/32",
  "2803:f800::/32",
  "2a06:98c0::/29",
  "2c0f:f248::/32"
]
Enter fullscreen mode Exit fullscreen mode

Yay! So all that's left now is to compare the two lists, which we can do using the diff command.

If the full command that'll come later is too confusing, you can also do it by saving each list to a temporary file, and then comparing the temporary files:

diff /tmp/output-of-scw-instances /tmp/output-of-cf-api
Enter fullscreen mode Exit fullscreen mode

But being the rockstars we are, we'll do everything in a single command by using the "<()" process substition syntax that is supported by many shells. The shell will take the output of whatever is inside "<()", put that in a temporary file, and pass that file to the top level command. Which is sort of what we would've done by hand.

Let's try running the monster, in her full glory.

$ diff <( scw instance security-group get --output json "$(scw instance
security-group list --output json name=scg-prod | jq -r 'first.id')" | jq
'.Rules | map(select(.direction == "inbound" and .dest_port_from == 443) |
.ip_range) | sort' ) <( curl -s 'https://api.cloudflare.com/client/v4/ips' | jq
'.result | [.ipv4_cidrs,.ipv6_cidrs] | flatten | sort' )`
$
Enter fullscreen mode Exit fullscreen mode

There was no output, which is good! It means there was no difference between the two lists.

So our work here is done, let us rest now, for a day another comes.

If you like what you saw here, say hi on Twitter or come hang out with us on Discord. Till then, stay cool and jq 😎

Top comments (2)

Collapse
 
cerchie profile image
Lucia Cerchie •

This was a fabulous article! It's important to me to understand developer tooling by seeing the broad concepts instantiated in different examples, and I feel like this really got me stated on jq. Thanks!

Collapse
 
enteio profile image
ente •

Thank you for the kind words 😊 and so glad that you found it useful!

Read next

mikeyoung44 profile image

Extracting Sensitive Data via Remote Timing Attacks on Efficient Language Models

Mike Young -

mikeyoung44 profile image

Robust Interpretable Reasoning via Neurosymbolic Program Synthesis

Mike Young -

mywillandprobate profile image

Navigating inheritance disputes: Why you need estate planning solicitors in London

My Will And Probate -

srbhr profile image

7 AI Open Source Libraries To Build RAG, Agents & AI Search

Saurabh Rai -