Data and System Visualization Tools That Will Boost Your Productivity

As files, datasets and configurations grow, it gets increasingly difficult to navigate them. There are however many tools out there, that can help you to be more productive when dealing with large JSON and YAML files, complicated regular expressions, confusing SQL database relationships, complex development environments and many others.

JSON

JSON is one of those formats which is nice for computers, but not for humans. Even relatively small JSON object can be pretty difficult to read and traverse, but there's a tool that can help!

JSON Visio is a tool that generates graph diagrams from JSON objects. These diagrams are much easier to navigate than the textual format and to make it even more convenient, the tool also allows you to search the nodes. Additionally, the generated diagrams can also be downloaded as image.

You can use the web version at https://jsonvisio.com/editor or also run it locally as Docker container.

Regular Expressions

Regular expressions (RegEx) are notorious for being extremely unreadable. There are 2 tools I recommend to help make sense of complex RegExes - first one them being https://regex101.com/:

Which can help you build and test RegExes, as well as break them down and identify its individual parts.

The second one is https://regex-vis.com which generates a graph from a RegEx which is very helpful for understanding what the expression actually does:

YAML

YAML, the language that was meant to be readable, expect it oftentimes isn't. AS we all know, long YAML documents with many levels of indentations can be very hard to navigate and troubleshoot.

To avoid spending unreasonable amount of time trying to find that one wrong indent, I recommend you use schema validation and let your IDE do all the work. You can use validation schemas from https://schemastore.org/json or custom schemas such as these for Kubernetes to validate your files. These will work both with JetBrains products (e.g. Pycharm, IntelliJ) as well as VSCode (see this guide)

If you prefer to use vim as your editor, I recommend using custom formatting that can help you spot mistakes. My preferred configuration looks like so:

# Add this line to "~/.vimrc"
autocmd FileType yaml setlocal ai et cuc sw=2 ts=2

And the resulting formatting will look like:

On top of the above-mentioned tools, it's also a good idea to use YAML linter such this one or its CLI equivalent, which will validate and cleanup your documents.

Finally, if you're working with OpenAPI/Swagger YAML specs, then you can use https://editor.swagger.io/ to view/validate/edit you schemas.

SQL

There's a lot of software for working with relational databases, most of them however focus on connecting to database instances and running SQL queries. These capabilities are very handy, but don't help with the fact that navigating database with possibly hundreds of tables can be very difficult. One tool that can solve that is Jailer:

Jailer is a tool which can - among other things - navigate through the database by following foreign keys. Couple videos showcasing all the features of Jailer can be found here.

Git

Git - a software with absolutely terrible user experience which we all use every day - could also use some tooling for navigating history (log), staging/committing files, viewing diffs or for example rebasing branches.

My tool of choice for all the above is IntelliJ/PyCharm git integration - in my opinion there's really no better tool for dealing with git-related stuff.

If you're not an IntelliJ/PyCharm user or if you prefer to stay in terminal, then following commands can make your life a little bit easier:

git log --graph --abbrev-commit --decorate --all \
    --format=format:"%C(bold blue)%h%C(reset) - %C(bold cyan)%aD%C(dim white) \
    - %an%C(reset) %C(bold green)(%ar)%C(reset)%C(bold yellow)%d%C(reset)%n %C(white)%s%C(reset)"

The above command will print a somewhat readable and pretty graph of commits history, which would look something like:

If you also desire a bit improved diffing functionality, then you can use git diff ... --word-diff --color-words:

As you can see from above 2 examples, with enough effort you might be able to make commandline git experience somewhat bearable, for more tips like these see my previous article at https://martinheinz.dev/blog/43.

If you want to avoid bare-bones git CLI altogether, you can try using forgit - a TUI for using git interactively:

The above screenshot shows interactive and pretty git log. The tool supports pretty much all of the git subcommands, most importantly rebase, cherry-pick, diff and commit. For full list of features and more screenshots see project's repository.

Beyond using git or your IDEs features you can also grab other tools that can help with complex diffing of files. One very powerful tool for this is Difftastic. Difftastic language-aware diffing tools which has support for wide range of languages. For demo of the tool see this video.

If you want a tool specifically for diffing structured languages like JSON, XML, HTML, YAML, then Graphtage is a great choice. Graphtage compares documents semantically and will find differences correctly even when you change ordering of elements.

Finally, if you're more into visual tools, then you might find Meld useful as it provides similar experience to JetBrains' products.

Docker

On the DevOps side of things, when working with Docker, it's not uncommon to spin up bunch of containers with docker-compose and end-up with a mess that's very hard to troubleshoot.

Lazydocker is a great tool for dealing with multiple Docker containers at the same time. If you don't believe me, then check out the "elevator pitch" in the project's repository.

If you're more into browser-based tools you might want to try out Portainer which provides dashboard for navigating/inspecting Docker containers, volumes, images, etc.

Kubernetes

A lot of things to cover when it comes to Kubernetes, considering that each API resource could use a visualization tool. There are however, a couple notable areas (pain points really), that oftentimes require visual tool to setup/manage effectively.

First one being NetworkPolicy which can be visualized and also configured using https://editor.cilium.io/. Even if you prefer to craft your policies by hand, I would still recommend visually verifying them with this tool.

Another similar tool is Network Policy Viewer, which focuses just on the visualization of policies and has simpler and more readable diagrams.

I'd recommend using this collection of network policy recipes to test out these 2 tools and see how they can be helpful to your workflow.

Another pain point in configuring Kubernetes is RBAC, more specifically Roles, RoleBindings and their Cluster-scoped alternatives. There are a couple of tools that can help with those:

Krane is a tool that can generate graph showing relationships between all roles and subjects. Krane also has many more features, including RBAC risk assessment, reporting and alerting, as well as querying/interrogating RBAC rules with CypherQL.

A simpler alternative to Krane is rbac-tool, which can be installed as kubectl plugin. It can also analyze, audit, interrogate RBAC rules, but most importantly, it can visualize them:

Finally, if you're more concerned with easily configuring RBAC rather than viewing pretty diagrams, then Permission manager is a tool for you. See GIFs in GitHub repository for demonstration of what the tool can do.

Instead of specialized tools for things like network policies or RBAC you can also make use of general purpose dashboards, such as:

• Lens - the Kubernetes IDE which brings some sanity into managing clusters, especially when paired with Lens Resource Map which displays Kubernetes resources and their relations as a real-time force-directed graph.

• As usual, there's also CLI based tool which provides more capabilities than the bare kubectl. It's called k9s and definitely does make it easier to navigate, observe and manage your deployed applications in Kubernetes:

Closing Thoughts

This article focused on developer/DevOps tooling, but there are 2 more things - honorable mentions - that deserve a shout-out. First being Mermaid.js for creating beautiful diagrams (as a code), which is now integrated with GitHub markdown. The other one - MathJax - for visualizing mathematical expressions, as of recent also supported by GitHub markdown.

These are the tools I like or the tools I made a use of in the past, so it's by no means an exhaustive list of things out there. If you have found different (better?) tools, please do share, so others can make use of them too.