Overview
Data management can be complex, especially when dealing with high-volume data pipelines. Whether you're moving data between databases, from file systems to databases, or vice versa, the Extract & Load (EL) approach can streamline these processes. Enter Slingdata, a powerful, free CLI tool built with Go that offers a straightforward solution for creating and maintaining robust data pipelines.
What is Slingdata?
Slingdata is a passion project turned into a practical tool that simplifies data movement across various systems. Its key focus is on the Extract & Load (EL) process, enabling efficient data transfer between:
- Database to Database
- File System to Database
- Database to File System
Key Features of Slingdata
- Single Binary deployment (built with Go). See the installation page.
- Use Custom SQL as a stream.
- Manage / View / Test / Discover your connections with the
sling conns
sub-command. - Use Environment Variables for connections.
- Provide YAML or JSON configurations (perfect for git version control).
- Powerful Replication logic, to replicate many tables with a wildcard (my_schema.*).
- Reads your existing DBT connections.
- Use your environment variable in your YAML / JSON config (
SELECT * from my_table where date = '{date}'
). - Convenient Transformations, such as the flatten option, which auto-creates columns from your nested fields.
- Run Pre & Post SQL commands.
- And many more!
Sample EL Process: ClickHouse to PostgreSQL
To showcase Slingdata’s capabilities, let's walk through a sample Extract & Load process moving data from ClickHouse to PostgreSQL.
Prerequisites
Ensure you have:
- ClickHouse installed and running.
- PostgreSQL installed and running.
- Slingdata installed.
Step 1: Configure the Source Database
Let's assume our source database is ClickHouse. We can export a new environment variable or add the connection to the env.yaml
of Slingdata files:
export MARKETING_URL="postgresql://postgres:postgres@localhost:5432/postgres?sslmode=disable"
To test this connection:
sling conns test MARKETING_URL
You should see the result:
INF success!
We will do the same for ClickHouse:
export DATAWAREHOUSE_URL="http://sling_user:sling_pass@clickhouse:8123/marketing"
To test this connection:
sling conns test DATAWAREHOUSE_URL
You should see the result:
INF success!
Now, let's create a database and a table in ClickHouse:
create database marketing;
CREATE TABLE `marketing`.users (
id UInt64,
name String,
family String,
age UInt8
) ENGINE = MergeTree()
ORDER BY id;
And in PostgreSQL, create a table with the same schema:
create table users(
id serial not null primary key,
name text not null,
family text,
age text not null
);
Insert data into ClickHouse:
insert into marketing.users values(1,'Shahab', 'Ranjbary', 28);
insert into marketing.users values(2,'Sepher', 'Ranjbary', 18);
insert into marketing.users values(3,'Mostafa', 'Fekri', 38);
Now, to move data to PostgreSQL, you can either create a replication file or run the following command:
sling run --src-conn DATAWAREHOUSE_URL --src-stream marketing.users --tgt-conn MARKETING_URL --tgt-object public.users
Conclusion
Slingdata offers a powerful and flexible way to manage data pipelines with ease. Its extensive features, combined with simple configurations and robust performance, make it an excellent tool for anyone dealing with high-volume data transfers.
Top comments (0)