Why this post?
If you spend a lot of your time-consuming APIs and having to build data pipelines, you will enjoy this post (if you don't know fx yet).
According to several sources, data scientists spend between 70-80% of their time normalizing data before they begin to play with it. That's a lot of time, so it's a good time investment to have powerful tools at your disposal without a steep learning curve.
When we talk about data processing, one of my favorite anecdote is the following one: Command-line Tools can be 235x Faster than your Hadoop Cluster. Sometimes people spend thousands of dollars on new software without realizing they can do the same task faster and cheaper with some alternatives. You've got me, I'm an old-school command-line guy ;-)
The point I'm trying to make here is that there is a ton of tools out there, but my advice to you is: "be conservative with your toolset". Spend more time with the tools you use daily. If you detect a recurrent issue in several scenarios, then it's time to look for alternatives.
I used jq for a couple of years but every time I struggled with some data transformation, the pain came from the same place: learning the concrete syntax for that tool. In that sense, fx freed me from that inconvenience.
Don't get me wrong. jq it's a very useful tool, but it takes time to control it. In my case, I don't want to spend more time learning a concrete syntax that is only useful with jq.
In this post, I'm not going to cover how to use jq, but if you're interested in it, here are some useful references:
fx
How to install
Prerequisite: You have to install nodejs on your computer.
npm i -g fx
fx can do a lot of things (see What can I do with fx below), but let me first explain the two modes on which fx operates:
fx modes
- Interactive: When you are not familiar with the data (JSON) you're playing with, this mode is quite useful because it lets you explore the data structure, find values, filter them, apply some transformation... Think interactive mode as a playground. Here you can see a sneak peak of fx in action:
As you can see, it's pretty intuitive! You can explore any JSON data in the same way you do when you are accessing an object in Javascript. Bonus point: it supports auto-completion.
- Commandline (CLI): Once you know the data, it's time to apply some transformations. This mode can be used in scripts, it's pipe-friendly, so you can concatenate several fx commands in a one-liner. Think cli mode as a grep,sed or awk command, but fx reads JSONs instead of lines. Let's see another visual example:
TIP: When you want to select text in interactive mode you need to press Alt / Fn key depending on your terminal
Once we saw the available modes of fx, let's practice with some examples.
What can I do with fx
If you want to follow the examples, just type in your terminal:
curl -sS "https://jsonplaceholder.typicode.com/users" -o users.json
Quick JSON exploration
When you are in interactive mode you can search for strings or use regular expressions. If you are familiar with vim editor you'll feel at home. If you're not familiar with regular expressions, you can start just typing "/" followed by the string you are looking for.
By default, the search is case-insensitive, so you don't have to worry about that.
To navigate across search results, just press Enter to go to the next match.
If you don't feel comfortable with regular expressions but you want to practice, I recommend you to take a look at RegEx101. It's a playground where you can start to master regular expressions.
Transform
From each user, I want to keep only the website and the "geo" keys.
Easy peasy! Because in fx we can use plain Javascript, let's translate this scenario to a Javascript code, and later apply this directly in fx:
For each user (object) we can make use of destructuring technique, to get the keys we want (website,geo) from the object and discard the rest:
var user = {
"id": 1,
"name": "Leanne Graham",
"username": "Bret",
"email": "Sincere@april.biz",
"address": {
"street": "Kulas Light",
"suite": "Apt. 556",
"city": "Gwenborough",
"zipcode": "92998-3874",
"geo": {
"lat": "-37.3159",
"lng": "81.1496"
}
},
"phone": "1-770-736-8031 x56442",
"website": "hildegard.org",
"company": {
"name": "Romaguera-Crona",
"catchPhrase": "Multi-layered client-server neural-net",
"bs": "harness real-time e-markets"
};
var {website,geo,...rest} = user; // user is our object
// website = "hildegard.org"
// geo = {"lat": "-37.3159", "lng": "81.1496"}
Now we want to apply this operation on each user, so let's do it with .map
users.map(({website,geo,...rest}) => ({website,geo}))
In fx, we will do it like this:
cat users.json | fx '.map(({website,geo,...rest}) => ({website,geo}))'
Isn't it beautiful?
You may think: "this is very basic stuff Jorge." Yep, indeed. But put yourself in the shoes of someone who has to do lots of different data transformations every day, or someone who is just getting insights from different data sources each time... Do you think that person is going to write a script every time?
That's the beauty of fx for me, it lets you do things very quickly without the need to learn anything more!
Filter
From our JSON file, I want to filter the company names of those which email has the domain .biz
cat users.json | fx '.filter(({email,...rest}) => /\.biz$/.test(email))' '.map(user => user.company.name)'
Got it!
Extra ball: Could I convert the results above in a CSV format (company and mail by line)?
cat users.json | fx '.filter(({email,...rest}) => /\.biz$/.test(email))' '.map(user => `${user.company.name};${user.email}`)' '.join("\n")'
Use your favorite npm module along fx
fx offers a way to include npm modules in the execution context.
When you are dealing with data structures in Javascript, lodash is a very handy option. Also, dayjs let us play with dates and time data easily.
Let's see how to use it along with fx:
Create .fxrc file in
$HOMEdirectory, and require any packages or define global functions.Install lodash and dayjs globally in your computer:
npm i -g lodash dayjs
- Set
NODE_PATHenv variable. This step is IMPORTANT to allow fx to make use of globally installed packages.
export NODE_PATH=`npm root -g`
- Put in your
.fxrcfile:
Object.assign(global, require('lodash/fp'))
global.dayjs = require("dayjs");
Now, let's play this scenario: I want to have a list of the 5 most recent github repositories I have that includes the day of the week I created each of them.
curl -sS "https://api.github.com/users/ntkog/repos" | \
fx '.map(({name,created_at,clone_url,...rest}) => ({name,created_at,clone_url}))' \
'sortBy("created_at")' \
'reverse' \
'take(5)' \
'map(repo => ({...repo, weekDay : dayjs(repo.created_at).format("dddd")}))'
Let's look at each step (we can chain several transformations in fx as you can see)
"\" at the end of the line it's just for separating one command into several lines
- Get all info of my github repos and pipe it to fx
curl -sS "https://api.github.com/users/ntkog/repos" |
- Keep only name,created_at,clone_url from each object
fx '.map(({name,created_at,clone_url,...rest}) => ({name,created_at,clone_url}))' \
- Sort array by created_at key
'sortBy("created_at")' \
- Invert the order of the results
'reverse' \
- Take 5 objects
'take(5)' \
- Add weekDay key to each object
'map(repo => ({...repo, weekDay : dayjs(repo.created_at).format("dddd")}))'
It's a very expressive way to transform the data step by step, don't you think?
Explore more possibilities
I didn't find a lot of articles talking about fx, but this talk from Антон Медведев gave me a lot of ideas on how to get the most out of fx.
Give fx a try, you won't regret it! Let me know if you find other tricks!
Happy hacking :-)
Photo by Samuel Sianipar on Unsplash
Top comments (0)