Introducing rack-dev_insight: A New Development Tool for Rack (Rails) Application

Hi! I'm excited to introduce a tool I've been developing over the past few months: rack-dev_insight.
This tool, comprising a gem and a chrome extension, is designed to provide insights into your Rack applications.
While it's still in its early stages, it's already valuable to be added to your development toolbox.

Table of Contents

• What is rack-dev_insight?
• Overview video of rack-dev_insight
• Unique features of rack-dev_insight
  • 1. Grouping SQL queries by CRUD operation
  • 2. Grouping SQL queries by Normalized SQL
  • 3. Detailed HTTP request / response data
  • 4. Functional devtools panel
• What is next?

What is rack-dev_insight?

rack-dev_insight is a simple, yet unique monitoring tool for Rack applications.
The rack-dev_insight gem serves as a middleware for recording and analyzing SQL queries and HTTP request / response data.
To visualize this data, Chrome Devtools panel is offered, making it a developer-friendly tool.
Currently, this tool is intended for use in development environments only.

Overview video of rack-dev_insight

Here is overview video of rack-dev_insight. Detailed explanations of its key features are described in next section.

Unique features of rack-dev_insight

Inspired by tools like rails_panel and rack-mini-profiler, rack-dev_insight stands out with its unique features:

1. Grouping SQL queries by CRUD operation

This feature basically creates a CRUD matrix.

What is CRUD matrix?

A CRUD matrix is a tabular representation showing how different entities (tables) are involved in CRUD (Create, Read, Update, Delete) operations in a business process. It's helpful for visualizing the interactions between various business processes and the tables involved.

For example, let's take a business process where a user updates a comment on a post. This action typically requires reading data from the users and posts tables and updating the comments table. The CRUD matrix would look like this:

Visualization in Devtools panel

In devtools panel this scenario is displayed as follows.

Usefulness

This feature is particularly useful in complex scenarios where dozens of tables are involved or when working with unfamiliar APIs, helping you recognize exhaustive table lists involved, and understand data dependencies more clearly.

Details

When extracting CRUD operations for the matrix, the following subtle cases are considered:

Conversion of Table Alias Names to Original Table Names

In scenarios where table aliases are used, such as in the following SQL query,

SELECT a FROM t1 AS t1_alias

it is counted as READ on the original table t1, instead of the alias t1_alias.

Recognition of Read Operations in CREATE, UPDATE, DELETE Queries

Not just the primary operation but also implicit operations within a query are considered.
For instance, the following SQL query

INSERT INTO t1 (a) SELECT a FROM t2 INNER JOIN t3 USING (id) WHERE t3.b = 1

is counted as INSERT operation for t1, and READ operation for t2 and t3.

2. Grouping SQL queries by Normalized SQL

This feature groups SQL queries by Normalized SQL, which is a uniform format of SQL statement.

How Normalization works

Normalization works similarly to pt-fingerprint.

For instance, consider the following set of SQL queries:

SELECT * FROM users WHERE id = 1
SELECT * FROM users WHERE id = 2
SELECT * FROM users WHERE id = 3

These are normalized into a uniform format:

SELECT * FROM users WHERE id = ?

Visualization in Devtools panel

This is displayed in the Devtools panel as follows.

Usefulness

This feature is helpful to detect so-called N+1 queries, but not limited to SELECT statements.

Details

Other DML queries are normalized in the same way like the following:

INSERT INTO users (name) VALUES (?)
UPDATE users SET name = ? WHERE id = ?
DELETE FROM users WHERE id = ?

3. Detailed HTTP request / response data

rack-dev_insight captures detailed HTTP request and response data, displayed in a Devtools panel similar to Chrome's network panel.
It currently supports the net-http gem, with future plans to include more HTTP clients, GraphQL, and gRPC.

4. Functional devtools panel

The Devtools panel has the following features:

Editor integration

• Open files directly in your editor by clicking their paths in the backtrace.
• Set up path mapping between local and remote file systems for seamless navigation.

Recording control

• Start or stop data recording, and flush recorded data, similar to the functionality of Chrome's network panel.

Sorting by columns

• Easily sort data by clicking on column headers for quick organization and analysis.

What is next?

If you are interested in this tool, please try it out! I would love to hear your feedback.
The following enhancements are currently under consideration:

• Additional options to normalization format
• Recording only the specified block
• Support for HTTP clients other than net-http
• Support for GraphQL and gRPC
• Support for other SQL dialects such as Microsoft SQL Server
• Alerting and reporting
• Background job (remote machines) support

Stay tuned for more updates. Your suggestions and ideas are always welcome.

Thank you for your time and interest!