Tag Archives: json

Command-line JSON Formatting with jq

Posted on by

About 2 or 3 months ago, when testing a deployment of a microservice at work
with Eric, our head Ops admin, we were looking at the JSON output of one of
the REST endpoints. Rather than looking at the raw output from curl, I
piped the output through JSON tool in the Python standard library:

$ curl -X GET -s http://rookeries.org/status | python -m json.tool
{
    "app": "rookeries",
    "version": "0.4.9"
}

(I will give you a couple of examples based on calls to Rookeries, rather than
the actual service call, since that API isn’t available publicly yet.)

Eric seeing that suggested I try out a utility that he uses: jq which is
great for formatting and querying JSON.

Installing jq

jq is a utility in C, and fortunately there are binary packages available for
it in the latest Ubuntu LTS (14.10). Installing it via aptitude (or apt-get):

$ sudo aptitude install jq

Pretty Printing JSON

The simplest use case for jq is to simply pretty print JSON output. This is
done by piping the result of a CURL command to jq with the parameter ‘.’:

$ curl -X GET -s http://status.bitbucket.org/api/v2/status.json | jq .
{
  "status": {
    "description": "All Systems Operational",
    "indicator": "none"
  },
  "page": {
    "updated_at": "2015-09-03T08:03:55.275Z",
    "url": "http://status.bitbucket.org",
    "name": "Atlassian Bitbucket",
    "id": "bqlf8qjztdtr"
  }
}

Formatting JSON in jq

It is also possible to format the output of the JSON to display only relevant
information. For instance if I want to find out the status of the components
that make up Bitbucket I can do the following:

$ curl -X GET -s http://status.bitbucket.org/api/v2/components.json | jq .

That however will give me a whole lot of extra data that I might not want. So
instead I might to narrow down and re-format the JSON data to something more
manageable with some jq magic:

$ curl -X GET -s http://status.bitbucket.org/api/v2/components.json | \ 
jq '{src: .page, components: [.components[] | \
    {id: .id, name: .name, status: .status}]}'

{
  "components": [
    {
      "status": "operational",
      "name": "Website",
      "id": "g0lfj4sv2fhf"
    },
    {
      "status": "operational",
      "name": "API",
      "id": "k0x2yw1435v7"
    },
    {
      "status": "operational",
      "name": "SSH",
      "id": "qmh4tj8h5kbn"
    },
    {
      "status": "operational",
      "name": "Git via HTTPS",
      "id": "c1qmcrcbc5zy"
    },
    {
      "status": "operational",
      "name": "Mercurial via HTTPS",
      "id": "vmbzxbbjz05j"
    },
    {
      "status": "operational",
      "name": "Webhooks",
      "id": "rfzky0v13fbp"
    },
    {
      "status": "operational",
      "name": "Source downloads",
      "id": "28h8dvv2qfzw"
    }
  ],
  "src": {
    "updated_at": "2015-09-03T08:03:55.275Z",
    "url": "http://status.bitbucket.org",
    "name": "Atlassian Bitbucket",
    "id": "bqlf8qjztdtr"
  }
}

I won’t explain that particular string in detail. But it will basically
craft a new JSON object, and generate new filtered objects when iterating
over the old array. Overall jq is pretty neat and is extremely fast to work
with.

There is also a Python bindings library for jq. I am considering using it to
help with mapping JSON into Python objects. However I have not played around
with it long enough to know if the extra dependencies are worthwhile or
whether or not it will bring a lot of benefits to Rookeries.

Reference Links