DEV Community

Cover image for Creating a database from scratch with Node.js - Day 3
Luis Felipe Ciochetta
Luis Felipe Ciochetta

Posted on

Creating a database from scratch with Node.js - Day 3

Hey folks!

Hope everyone had a nice Christmas.

I am back writing about my database project!

So, I've been studying a lot about databases and database engineering to understand what are the expected features for a database.

I've learned about indexing, partitioning, and sharding (mostly from this channel), and I am pretty sure I will add indexing and partitioning in this project, even though I think it will take a good time to code the partitioning logic.


New Features

Parser improvement

Now I can send objects to the database commands, this won't be useful now, but I intend to have a nice integration with Node.js in general, and reading objects will really help to build those integrations (I believe so).

The parser can also read arrays, so I can pass every parameter of a given command inside the same array if needed, this should be very useful, as it allows me to not write any logic to start and end of the parameter.

Example:

Alt Text

Here I've created a new table with the columns "firstName", "lastName", "login", and "password"

Query improvement

Now I don't have to return every table from a query, the user can inform the keys he wants in the row structure.

This will not be very useful now, but it should really help when I start working with indexing, as a user could declare an index with two columns that he uses frequently, and then instead of searching inside the document, I could just return the result he wants right from the index if his query matches the fields from the index (I will try to explain what I mean better when I start implementing it).

Examples:

Alt Text

I've inserted some users and then wrote a query that retrieves their login

Alt Text

Here I am just showing that it can be done with multiple fields

Other changes

I've added unit testing to the project, finally!

the parser function is getting very complex (and ugly, tbh), so I've decided to add some tests to guarantee that it keeps working as expected as I keep adding things to it.

BTW: if anyone reading this knows how to write a good parser or have any material that could help me, I would really appreciate it, the code for my parser is getting uglier and uglier by the day.

Also, I've changed the code structure again, as the parser was getting way too big, I've separated it from the evaluator, which has it's own file now.


That's all, folks, if anyone wants to check out the project code or try to use it, this is the repository for it:

GitHub logo ciochetta / learndb

Database project I've created for learning purposes

LuisDB

My study on how to create a database

Hello, this is the repository for my database project, I am trying to learn how to create a database from scratch using Node.js

My objective is to understand a little better how databases work internally

Installation

You need to have Node.js and NPM to install and use this project

npm i learndb
Enter fullscreen mode Exit fullscreen mode

Usage

you can use this in two ways, either as an REPL or as a driver for the database

REPL

If you want to use this project as a REPL I would adivise to clone it instead of installing it, if you have it cloned, to access the REPL, all you need to do is type this on the terminal:

node index.js repl
Enter fullscreen mode Exit fullscreen mode

REPL Commands

using [database name]

Either loads a database from the directory you are currently in or creates a database with the specified name if none…

Discussion (2)

Collapse
oguimbal profile image
Olivier Guimbal • Edited on

I've done more or less the same thing as you're doing when writing pg-mem, which is an in-memory postgres database emulation.

So to answer your question: I've published a separate npm package for its parser pgsql-ast-parser, which is using nearley, one of the many solutions that you can use to write a parser with nodejs (I've also played with pegjs when contributing to node-sql-parser, but I felt that Nearley is more powerful)

Collapse
ciochetta profile image
Luis Felipe Ciochetta Author

Hey Oliver

Thanks for the tips on the parser

I've done a quick reading on the nearly and will try to implement it tomorrow in this project