DEV Community

Cover image for File Database in Node Js from scratch part 2: Select function & more
sk
sk

Posted on • Updated on

File Database in Node Js from scratch part 2: Select function & more

introduction

Welcome to part 2 of trying to build from scratch and becoming a better programmer, if you just stumbled upon this post and have no idea what's going you can find part 1 here, else welcome back and thank you for your time again.

Part 1 was just a setup, nothing interesting happened really and since then I had some time to think about stuff, hence some refactor and lot of code in this part.

Database.js

Soon to be previous db function:



function db(options) {

    this.meta = {
           length: 0,
           types: {},   
           options 


    }
    this.store = {} 


}


Enter fullscreen mode Exit fullscreen mode

The first problem I noticed here is this.store is referenced a lot in operations.js by different functions, initially it may not seem like a big deal, but if you think for a minute as object are values by reference, meaning allowing access to a single object by multiple functions can cause a huge problem, like receiving an outdated object, trying to access a deleted value etc,

The functions(select, insert, delete_, update) themselves need to do heavy lifting making sure they are receiving the correct state, checking for correct values and so on, this leads to code duplication and spaghetti code.

I came up with a solution inspired by state managers, having a single store which exposes it's own API, and no function outside can access it without the API.

The API is responsible for updating the state, returning the state and maintaining the state, any function outside can request the store to do something and wait, code speaks louder than words, here is a refactored db function


import Store from "./Store.js"



function db(options) {
    this.store = new Store("Test db", options) // single endpoint to the database 


}





Enter fullscreen mode Exit fullscreen mode

I guess the lesson here is once everything starts getting out of hand and spiraling, going back to the vortex abstraction and creating a single endpoint to consolidate everything can be a solution. This will be apparent once we work on the select function.

one last thing we need is to remove select from operators to it's own file, select has a lot of code

updated Database.js



import {insert, update, delete_} from './operators.js' // remove select
import Store from "./Store.js"
import select from "./select.js" // new select

function db(options) {
    // minor problem: store can be accessed from db object in index.js
    // not a problem thou cause #data is private
    this.store = new Store("Test db", options)


}



db.prototype.insert = insert
db.prototype.update = update
db.prototype.select = select
db.prototype.delete_ = delete_


export default db 



Enter fullscreen mode Exit fullscreen mode

Store.js (new file)

I chose to use a class for store, you can definitely use a function, my reason for a class is, it is intuitive and visually simple for me to traverse, and easy to declare private variables

If you are unfamiliar with OOJS(Object Oriented JS) I do have two short articles here, and for this article you need to be familiar with the this keyword




export default class Store{
  // private variables start with a "#"
        #data = {}
        #meta = {
           length: 0,
          }

      // runs immediatley on class Instantiation
      constructor(name, options){

             this.#meta.name = name;
             this.#meta.options = options


      }


    // API 

    // getters and setters(covered in OOJS)
      //simply returns data  
     get getData(){

          return this.#data
      }

    // sets data 
    // data is type Object
    set setData(data){

        data._id = this.#meta.length

    if(this.#meta.options && this.#meta.options.timeStamp &&   this.#meta.options.timeStamp){
           data.timeStamp = Date.now()

        }
        this.#data[this.#meta.length] = data
        this.#meta.length++

    }

}


Enter fullscreen mode Exit fullscreen mode

Explaining setData

data._id = this.#meta.length  // _id is reserved so the documents(rows) can actually know their id's

Enter fullscreen mode Exit fullscreen mode

adding timeStamp


    if(this.#meta.options &&        this.#meta.options.timeStamp &&  this.#meta.options.timeStamp){
           data.timeStamp = Date.now()

        }

// this lines reads 
// if meta has the options object 
// and the options object has timeStamp
// and timeStamp(which is a boolean) is true 
  // add datetime to the data before commiting to the db 
// this check is necessary to avoid cannot convert null Object thing error


Enter fullscreen mode Exit fullscreen mode
// putting the document or row
this.#data[this.#meta.length] = data
// incrementing the id(pointer) for the next row
this.#meta.length++


Enter fullscreen mode Exit fullscreen mode

Now we can safely say we have a single endpoint to the database(#data) outside access should consult the API, and not worry about how it gets or sets the data

However using setData and getData sounds weird, we can wrap these in familiar functions and not access them directly

classes also have a proto object covered here


Store.prototype.insert = function(data){
     // invoking the setter 
    // this keyword points to the class(instantiated object)
    this.setData  = data
}



Enter fullscreen mode Exit fullscreen mode

now we can update operators.js insert

operators.js

// updating insert(letting the store handle everything)
export function insert(row){
            this.store.insert(row)
}

Enter fullscreen mode Exit fullscreen mode

Select.js

I had many ideas for select, mostly inspired by other db's, but I settled on a simple and I believe powerful enough API, for now I want select to just do two things select by ID and query the db based on certain filters.

let's start with select by id as it is simple



export default function select(option = "*"){
        // checking if option is a number
        if(Number(option) !== NaN){

             // return prevents select from running code below this if statement()
            // think of it as an early break
             return this.store.getByid(+option)
            // the +option converts option to a number just to make sure it is
        }

     // query mode code will be here

}


Enter fullscreen mode Exit fullscreen mode

based on the value of option we select to do one of two select by ID or enter what i call a query mode, to select by id all we need to check is if option is a number, if not we enter query mode

Store.js

we need to add the select by id function to the store


... 


Store.prototype.getByid = function(id){
    const data = this.getData  // get the pointer the data(cause it's private we cannot access it directly) 
    //object(remember the value by reference concept)

    if(data[id]){  // checking if id exists

          return data[id]  // returning the document
        }else{

          return "noDoc" // for now a str will do
            // but an error object is more appropriate(future worry)
        }

}



Enter fullscreen mode Exit fullscreen mode

Simple and now we can get a row by id, query mode is a little bit involved, more code and some helpers

Select.js query mode

The core idea is simple really, I thought of the db as a huge hub, a central node of sort, and a query is a small node/channel connected to the center, such that each query node is self contained, meaning it contains it's own state until it is closed.

example



let a = store.select() // returns a query chanel/node
let b = store.select() 

// a is not aware of b, vice versa, 
//whatever happens in each node the other is not concerned



Enter fullscreen mode Exit fullscreen mode

for this to work we need to track open channels and their state as the querying continues, an object is a simple way to do just that.


const tracker = {
     id: 0, // needed to ID each channel and retrieve or update it's state
 }


function functionalObj(store){
    this.id = NaN  // to give to tracker.id(self identity)

}


export default function select(option = "*"){
...

     // query mode code will be here

   // functionalObj will return the node/channel
    return  new functionalObj(this.store)
}




Enter fullscreen mode Exit fullscreen mode

functionalObj will have four functions:

beginQuery - will perform the necessary setup to open an independent channel/node to the db

Where - will take a string(boolean operators) to query the db e.g Where('age > 23') return all docs where the age is bigger than 23

endQuery - returns the queried data

close - destroys the channel completely with all it's data

beginQuery


...


function functionalObj(store){
    ...
    // channelName will help with Identifying and dubugging for the developer using our db
    this.beginQuery = (channelName = "") => {
          // safeguard not to open the same query/channel twice 
          if(tracker[this.id] && tracker[this.id].beganQ){ // checking if the channel already exists(when this.id !== NaN)
                  console.warn('please close the previous query');
                    return 
          }



         // opening a node/channel
             this.id = tracker.id
             tracker[this.id] = {
              filtered: [], // holds filtered data
              beganQ: false,  // initial status of the channel(began Query)
              cName : channelName === "" ? this.id : channelName  
             }

            tracker.id++  // for new channels


             // officially opening the channel to be queried

                                     // we will define the getAll func later
                                     // it basically does what it's says
               tracker[this.id].filtered = Object.values(store.getAll()) // to be filtered data
               tracker[this.id].beganQ = true  // opening the channel
               console.log('opening channel: ', tracker[this.id].cName) // for debugging 

    }
    // end of begin query function



}




Enter fullscreen mode Exit fullscreen mode

update Store.js and put this getAll func


Store.prototype.getAll = function(){

  return this.getData

}

Enter fullscreen mode Exit fullscreen mode

Where, endQuery, close



function functionalObj(store){
    this.beginQuery = (channelName = "") => {
    ... 
    }
    // end of beginQuery 

    this.Where = (str) => {
         // do not allow a query of the channel/node if not opened
         if(!tracker[this.id] || tracker[this.id] && !tracker[this.id].beganQ){
            console.log('begin query to filter')
            return
         }

         let f = search(str, tracker[this.id].filtered) // we will define search later(will return filtered data and can handle query strings)

            // update filtered data for the correct channel
            if(f.length > 0){

                tracker[this.id].filtered = f
             }


    }
    // end of where

    this.endQuery = () => {

                if(!tracker[this.id] || tracker[this.id] && !tracker[this.id].beganQ){
                            console.warn('no query to close')
                           return
                    }


            // returns data                
         return {data:tracker[this.id].filtered, channel: tracker[this.id].cName}     
            };

        // end of endQuery 

       this.close = ()=> {
                 // if a node/channel exist destroy it
               if(tracker[this.id] && !tracker[this.id].closed){
                          Reflect.deleteProperty(tracker, this.id) // delete 
                          console.log('cleaned up', tracker) 

                    }


        }
}







Enter fullscreen mode Exit fullscreen mode

Search


// comm - stands for commnads e.g "age > 23"
const search = function(comm, data){
     let split = comm.split(" ") // ['age', '>', 23]  
     // split[0] property to query 
     // split[1] operator 
     // compare against
     let filtered = []  

      // detecting the operator
      if(split[1] === "===" || split[1] === "=="){

            data.map((obj, i)=> {
                 // mapSearch maps every operator to a function that can handle it
                // and evalute it 
                // mapSearch returns a boolean saying whether the object fits the query if true we add the object to the filtered
                 if(mapSearch('eq' , obj[split[0]], split[2])){
                      // e.g here mapSearch will map each object with a function
                     // that checks for equality(eq)
                       filtered.push(obj)
                 }

            }) 

    }else if(split[1] === "<"){
             data.map((obj, i)=> {
                  // less than search
                     if(mapSearch('ls' , obj[split[0]], split[2])){

                           filtered.push(obj)
                     }

            })     

    }else if(split[1] === ">"){


                data.map((obj, i)=> {
                    // greater than search
                     if(mapSearch('gt' , obj[split[0]], split[2])){

                           filtered.push(obj)
                     }

             }) 
    }


       return filtered // assigned to f in Where function
}

function functionalObj(store){
... 
}



Enter fullscreen mode Exit fullscreen mode

mapSearch


// direct can be eq, gt, ls which directs the comparison 
// a is the property  --- age 
// b to compare against --- 23
const mapSearch = function(direct, a, b){


         if(direct === "eq"){
               // comparers defined func below
           return comparers['eq'](a, b) // compare for equality
     }else if(direct === "gt"){

              return comparers['gt'](a, b) // is a > b

        }else if(direct === "ls"){
            return comparers['ls'](a, b) // is a < b


            }else{

                console.log('Not handled')
            }


}





const search = function(comm, data){
... 
}
...

Enter fullscreen mode Exit fullscreen mode

Comparers

actually does the comparison and returns appropriate booleans to filter the data

  // return a boolean (true || false)
const comparers = {
  "eq": (a, b) => a === b,
   "gt": (a, b) => a > b,
    "ls": (a, b) => a < b


}

Enter fullscreen mode Exit fullscreen mode

Select should work now, we can query for data through dedicated channels

test.js

testing everything


import db from './index.js'



let store = new db({timeStamp: true})


store.insert({name: "sk", surname: "mhlungu", age: 23})
store.insert({name: "np", surname: "mhlungu", age: 19})
store.insert({name: "jane", surname: "doe", age: 0})


const c = store.select() // return a new node/channel to be opened

c.beginQuery("THIS IS CHANNEL C")  // opening the channel and naming it

c.Where('age < 23') // return all documents where age is smaller than 23


const d = store.select() // return a new node/channel 

d.beginQuery("THIS IS CHANNEL D") // open the channel
d.Where('age > 10') // all documents where age > 10

console.log('===============================================')
console.log(d.endQuery(), 'D RESULT age > 10')  // return d's data 
console.log('===============================================')
console.log(c.endQuery(), "C RESULT age <  23") // return c's data
console.log('===============================================')
c.close() // destroy c 
d.close()  // destroy d


Enter fullscreen mode Exit fullscreen mode

node test.js

Enter fullscreen mode Exit fullscreen mode

you can actually chain multiple where's on each node, where for now takes a single command

example

const c = store.select() 

c.beginQuery("THIS IS CHANNEL C")

c.Where("age > 23")
c.Where("surname === doe") // will further filter the above returned documents

Enter fullscreen mode Exit fullscreen mode

Problems

the equality sign does not work as expected when comparing numbers, caused by the number being a string



// "age === 23"

comm.split(" ") // ['age', '===', '23'] // 23 becomes a string 

23 === '23' // returns false

// while 'name === sk' will work

comm.split(" ") // ['name', '===', 'sk']

'sk'  === 'sk'


Enter fullscreen mode Exit fullscreen mode

a simple solution will be to check if each command is comparing strings or numbers, which in my opinion is very hideous and not fun to code really, so a solution I came up with is to introduce types for the db, meaning our db will be type safe, and we can infer from those types the type of operation/comparisons

for example a new db will be created like this:



let store = new db({
                 timeStamp: true,
                 types: [db.String, db.String, db.Number]  // repres columns

                   })



// if you try to put a number on column 1 an error occurs, because insert expect a string
Enter fullscreen mode Exit fullscreen mode

the next tut will focus on just that.

conclusion

If you want a programming buddy I will be happy to connect on twitter , or you or you know someone who is hiring for a front-end(react or ionic) developer or just a JS developer(modules, scripting etc) I am looking for a job or gig please contact me: mhlungusk@gmail.com, twitter will also do

Thank you for your time, enjoy your day or night. until next time

Discussion (0)