DEV Community

Cover image for Just in! A New Persistent NoSQL Database (18 KiB only!)
Trinmar Boado
Trinmar Boado

Posted on

Just in! A New Persistent NoSQL Database (18 KiB only!)

Welcome to Trin.DB!

A fast RESTful persistent or in memory NoSQL database (18 KiB only!)

Github Repo: https://github.com/trinly01/TrinDB

Installation

npm install trin.db

or

yarn add trin.db

Usage

const express = require('express')
const app = express()
const port = process.env.PORT || 3000
const trinDB = require('trin.db')

app.use(express.json())               // required for RESTful APIs

app.listen(port, async () => {
  app.trinDB = {
    todos: await trinDB({             // Todos Service
      filename: 'trinDb/todos.db',    // get records from a file
      inMemoryOnly: false,            // Optional
      restful                         // Optional
    })
  }
})

// Other Options

const restful = {                     // Optional
  app,                                // express app
  url: '/todos',                      // API end-point
  hooks                               // Optional
}

const hooks = ({app, service}) => ({  // Hooks Example
  before: {
    all: [
      (req, res, next) => {
        console.log('before all hook')
        next()
      }
    ],
    get: [],
    find: [],
    create: [],
    patch: [],
    remove: []
  }
  after: {
    all: [
      (result) => {
        console.log(result)
        return result
      }
    ],
  }
})

await trinDB(<object>)

Returns a trinDB Service

Object Prop Type Description Default Mode
filename <string> Path to file. Required if in persistent mode n/a persistent
inMemoryOnly <boolean> ( Optional ) If true, database will be in non-persistent mode false in-memory
restful <object> ( Optional ) { app, url, hooks } n/a persistent

service.create(<object>)

Returns the created <object>

/* example existing records (service.data)
  {
    asd: { _id: 'asd', text: 'Hello', read: true, nested: { prop: 'xander' } },
    zxc: { _id: 'zxc', text: 'World', read: false, nested: { prop: 'ford' } }
  }
*/

const result = service.create({
  text: 'Trinmar Pogi'
})

console.log(result)
// { _id: 'qwe', text: 'Trinmar Pogi' }

console.log(service.data)
/* service.data with the newly created object
  {
    asd: { _id: 'asd', text: 'Hello', read: true, nested: { prop: 'xander' } },
    zxc: { _id: 'zxc', text: 'World', read: false, nested: { prop: 'ford' } },
    qwe: { _id: 'qwe', text: 'Trinmar Pogi' }
  }
*/

RESTful API

curl --location --request POST 'http://localhost:3000/todos' \
--header 'Content-Type: application/json' \
--data-raw '{
  "text": "Trinmar Pogi"
}'

service.find(<object>)

Returns found data <object>

/* example existing records (service.data)
  {
    asd: { _id: 'asd', firstName: 'Trinmar', lastName: 'Pogi', age: 20 },
    zxc: { _id: 'zxc', firstName: 'Trinly Zion', lastName: 'Boado', age: 1 },
    qwe: { _id: 'qwe', firstName: 'Lovely', lastName: 'Boado', age: 18 }
  }
*/

// Equality
result = service.find({
  query: {
    lastName: 'Pogi' // equality
  },
  limit: 10, // default 10
  skip: 0 // default 0
})
console.log(result)
/*
  {
    total: 1,
    limit: 10,
    skip: 0,
    data: {
      asd: { _id: 'asd', firstName: 'Trinmar', lastName: 'Pogi', age: 20 }
    }
  }
*/

RESTful API

curl --location --request GET 'http://localhost:3000/todos?lastName=Pogi&$limit=10&$skip=0'

Complex Query (conditional >, >==, <, <==, &&, || )

// Map data or select specific props
result = service.find({
  query (obj) {
    return ob.age < 20
  },
  map (obj) {
    return {
      fullName: obj.firstName + ' '+  obj.lastName
    }
  }
})
console.log(result)
/*
  {
    total: 2,
    limit: 10,
    skip: 0,
    data: {
      zxc: { _id: 'zxc', firstName: 'Trinly Zion Boado' },
      qwe: { _id: 'qwe', firstName: 'Lovely Boado' }
    }
  }
*/

service.search(keywords)

fuzzy search finds data based on the keywords (<String>) and returns it sorted by _score

/* example existing records (service.data)
  {
    asd: { _id: 'asd', firstName: 'Trinmar', lastName: 'Boado' },
    zxc: { _id: 'zxc', firstName: 'Trinly Zion', lastName: 'Boado' },
    qwe: { _id: 'qwe', firstName: 'Lovely', lastName: 'Boado' }
  }
*/

result = service.search('ly oad')

console.log(result)
/*
  {
    total: 3,
    data: {
      qwe: { _score: 2, _id: 'qwe', firstName: 'Lovely', lastName: 'Boado', age: 18 },
      zxc: { _score: 2, _id: 'zxc', firstName: 'Trinly Zion', lastName: 'Boado', age: 1 },
      asd: { _score: 1, _id: 'asd', firstName: 'Trinmar', lastName: 'Pogi', age: 20 },
    }
  }
*/

RESTful API

curl --location --request GET 'http://localhost:3000/todos?$search=ly%20oad'

service.patch(_id, <object>)

Returns the created <object>

// { _id: 'q12m3k', firstName: 'Trinmar', lastName: 'Boado' nested: { counter: 123 } }

const result = service.patch('q12m3k', {
  lastName: 'Pogi',
  children: ['Trinly Zion'],
  'nested.counter': 456
})

console.log(result)
// { _id: 'q12m3k', lastName: 'Pogi' children: ['Trinly Zion'], 'nested.counter': 456 }

console.log(service.data['q12m3k'])
// { _id: 'q12m3k', firstName: 'Trinmar', lastName: 'Pogi', nested: { prop: 456 }, children: ['Trinly Zion'] }

RESTful API

curl --location --request PATCH 'http://localhost:3000/todos/:_id' \
--header 'Content-Type: application/json' \
--data-raw '{
    "lastName": "Pogi",
    "children": ["Trinly Zion"],
    "nested.counter": 456
}'

service.remove(_id)

Returns the removed <object>

service.remove('q12m3k')

console.log(service.data['q12m3k'])
// undefined

RESTful API

curl --location --request DELETE 'http://localhost:3000/todos/:_id'

service.removeProps(_id, <object>)

Returns the removed <object> props

// { _id: 'q12m3k', firstName: 'Trinmar', lastName: 'Pogi', nested: { prop: 456 }, children: ['Trinly Zion'] }

service.removeProps('q12m3k', {
  lastName: true,
  'nested.prop': true
  firstName: false
})

console.log(service.data['q12m3k'])
// { _id: 'q12m3k', firstName: 'Trinmar', children: ['Trinly Zion'] }

RESTful API

curl --location --request PATCH 'http://localhost:3000/todos/:_id' \
--header 'Content-Type: application/json' \
--data-raw '{
    "$action": "removeProps"
    "lastName": true,
    "nested.prop": true,
    "firstName": false
}'

service.inc(_id, <object>)

Increments specific props and returns the <object>

// { _id: 'q12m3k', firstName: 'Trinmar', lastName: 'Pogi', nested: { prop: 456 }, children: ['Trinly Zion'] }

service.inc('q12m3k', {
  'nested.prop': 5
})

console.log(service.data['q12m3k'])
// { _id: 'q12m3k', firstName: 'Trinmar', lastName: 'Pogi', nested: { prop: 461 }, children: ['Trinly Zion'] }

RESTful API

curl --location --request PATCH 'http://localhost:3000/todos/:_id' \
--header 'Content-Type: application/json' \
--data-raw '{
    "$action": "inc"
    "nested.prop": 5
}'

service.splice(_id, <object>)

removes element by index and returns the <object>

// { _id: 'q12m3k', children: ['Trinly Zion', 'Trinmar Boado'] }

service.splice('q12m3k', {
  'children': 1
})

console.log(service.data['q12m3k'])
// { _id: 'q12m3k', children: ['Trinly Zion'] }

RESTful API

curl --location --request PATCH 'http://localhost:3000/todos/:_id' \
--header 'Content-Type: application/json' \
--data-raw '{
    "$action": "splice"
    "children": 1
}'

service.push(_id, <object>)

adds one or more elements to the end of an array and returns the <object>

// { _id: 'q12m3k', children: ['Trinly Zion', 'Trinmar Boado'] }

service.push('q12m3k', {
  'children': 'Lovely Boado'
})

console.log(service.data['q12m3k'])
// { _id: 'q12m3k', children: ['Trinly Zion', 'Trinmar Boado', 'Lovely Boado'] }

RESTful API

curl --location --request PATCH 'http://localhost:3000/todos/:_id' \
--header 'Content-Type: application/json' \
--data-raw '{
    "$action": "push"
    "children": "Lovely Boado'"
}'

service.unshift(_id, <object>)

adds one or more elements to the beginning of an array and returns the <object>

// { _id: 'q12m3k', children: ['Trinly Zion', 'Trinmar Boado'] }

service.unshift('q12m3k', {
  'children': 'Lovely Boado'
})

console.log(service.data['q12m3k'])
// { _id: 'q12m3k', children: ['Lovely Boado', 'Trinly Zion', 'Trinmar Boado'] }

RESTful API

curl --location --request PATCH 'http://localhost:3000/todos/:_id' \
--header 'Content-Type: application/json' \
--data-raw '{
    "$action": "unshift"
    "children": "Lovely Boado'"
}'

service.sort(data,<object>)

Sorts the data based on the <object> and returns the sorted data

/* example existing records (service.data)
  {
    asd: { _id: 'asd', firstName: 'Trinmar', lastName: 'Pogi', age: 20 },
    zxc: { _id: 'zxc', firstName: 'Trinly Zion', lastName: 'Boado', age: 1 },
    qwe: { _id: 'qwe', firstName: 'Lovely', lastName: 'Boado', age: 18 }
  }
*/

// Descending (-1)
result = service.sort({
  data: service.data, // (Optional) if not defined, service.data will be used
  params: {
    age: -1
  }
})

console.log(result)
/*
  {
    asd: { _id: 'asd', firstName: 'Trinmar', lastName: 'Pogi', age: 20 },
    qwe: { _id: 'qwe', firstName: 'Lovely', lastName: 'Boado', age: 18 },
    zxc: { _id: 'zxc', firstName: 'Trinly Zion', lastName: 'Boado', age: 1 }
  }
*/

service.copmact(filename, <object>)

writes the compact data to a file
| param | Type | Description | Default |
|--|--|--|--|
| filename | <string> | (Optional) Path to file | current |
| | <object> | ( Optional ) a TrinDB object | service.data |

service.copmact('test.db', service.data)

Github Repo: https://github.com/trinly01/TrinDB

Join and support our Community
Web and Mobile Developers PH
[ Facebook Page | Group ]

Discussion (16)

Collapse
joeyhub profile image
Joey Hernández • Edited

This isn't very good:

module.exports = (path, data, fs) => {
  fs.writeFileSync(path, '')
  for (const key in data) {
    const line = JSON.stringify({
      action: 'create',
      data: data[key]
    })
    fs.appendFileSync(path, `${ line }\r\n`)
  }
}
  1. Open the file properly, consider if locks are needed.
  2. Make it non-blocking.
  3. Why break it into lines?
  4. Use binary serialization such as msgpack.
  5. Consider compression.
Collapse
trinly01 profile image
Trinmar Boado Author

npmjs.com/package/msgpack

node-msgpack is currently slower than the built-in JSON.stringify() and JSON.parse() methods. In recent versions of node.js, the JSON functions have been heavily optimized. node-msgpack is still more compact, and we are currently working performance improvements. Testing shows that, over 500k iterations, msgpack.pack() is about 5x slower than JSON.stringify(), and msgpack.unpack() is about 3.5x slower than JSON.parse().

Collapse
joeyhub profile image
Joey Hernández

I'm aware of this though it's a bit more complex. Non-native msgpack implementations will find it hard to compete against native implementations.

I did a lot of research on binary serialization. Specifically writing several of my own, comparing to igbinary and msgpack for speed, size when uncompressed and size when compressed.

Two things were difficult:

  1. Making any difference in size, except for string deduplication (igbinary does quite well with this even post compression). Note that most improvements made to the binary to make it smaller really had much the same effect as pre-compressing so rarely made any difference after compression.
  2. Anything, and I mean getting anything much faster in JS be it in JS itself or C++ was very difficult to get the same performance gain you get compared to something like PHP C extensions. A lot of the overhead in JS is baked right into the objects used under the hood in C++ if I remember not just the interpretation.

msgpack being slower is a problem though I'm not sure why it should be so at least on the backend. I only ever considered that a problem on the frontend. It should have all the same limitations and advantages as the native JSON code does. Unless, which I can't be sure about as it was a while ago, there might have also been some barriers between the engine and extensions.

In theory a backend implementation of msgpack, at least minimal, would be little more than a copy and paste of the one for JSON (though if it uses something like YACC it's overkill). It should basically be the same but with less so faster.

It doesn't help that JS isn't entirely binary friendly (another area which it sorely hurt compared to PHP, which I mention a lot because a lot of infrastructure is a mix of the two and they need to talk to each other).

JSON does in some circumstances have size benefits but they're not often brought out.

It's a shame that msgpack isn't as fast at least frontend due to no native (tried that webasm stuff / cutting edge JS though, typed arrays, etc)?

Regardless, it's not always the main thing. It's still potentially much smaller which might be important if bandwidth or the amount written / read is a concern.

I haven't checked recently, but msgpack may still not support dictionaries as an extra option. You can however do this yourself. In the serialized stream you might look for if there are repeat strings and then store those only once in an array then the string type points to those instead.

That has an interesting benefit as you can reduce memory use as well as a side effect reducing copies of the same string. Though I don't know if JS engines are able to do that themselves in any situation.

A similar thing might be done with object arrays. IE if they're all the same, instead of [{a: 123, b: false}, {a: 321, b: true}] have ['a', 'b', 123, false, 321, false] though that kind of things you might want to better do yourself over the serialization. You can do the same for a string dictionary like igbinary has which is surprisingly effective even post compression but might have complications or limitations with streaming and memory usage patterns.

This is a similar concept to gson and interned strings. I've been using those successful and for proven gain for some time now.

There's probably a missing pre/post pack library out there for these cases. They can all take more or less CPU but it's basically the same trade off the same as with compression, spend more time but get better results.

I found it very annoying with very large things that you can't get the best of both worlds with msgpack. IE, you notice both the CPU time and net time.

It wouldn't surprise me if one of the API's somewhere sneaking into modern JS has some binary serialisation that might be native.

I'm seeing C++ for msgpack so it should at least in theory be possible to get it up to JSON speed on the backend. It's sometimes more important on the backend because that's where the contention is.

Thread Thread
trinly01 profile image
Trinmar Boado Author

Thanks for the detailed explanation ❤️

My goal is to be pure JS implementation with no binary dependencies.

I think streams are optimized to minimize CPU and memory overheads

Thread Thread
joeyhub profile image
Joey Hernández • Edited

There's always a lot to learn, I still learn every day.

No bins is good for supporting front and back as well as being hardware agnostic.

You should take a look at fs.open, and see if there's a flock, etc.

You might decide you don't need to implement flock but if you make that decision it's crucial to understand concurrency issues.

To test this might be easier than you think. See if there's a sleepSync. Then after writing each line sleep.

Then run a basic program filling the in memory store then saving it twice at the same time.

You'll see lines from both. Ok it doesn't matter you mgiht think they're the same and it's just the same op twice but initialise each database so that they have the same keys but different values.

If you really push it, just run the program thousands of times at the same time with no delays, then you can get something like the file will have one line that suddenly stops and then the line from another program continues meaning the lines wont even be parsable in JSON.

When people are starting out the first thought is "how can I make this work". The next step, the revolution, is thinking "how can I break this" :D.

Collapse
trinly01 profile image
Trinmar Boado Author • Edited

Thanks for the suggestion. <3
I already converted it from appendFileSync to fileWriteStream

Collapse
joeyhub profile image
Joey Hernández • Edited

These are concepts you might find it useful to study and learn as well as experiment with.

Non-blocking is a crucial concept in JS that gives it much of it's performance (async). While this write operation is happening everything is blocked and nothing can happen.

By also writing to the file like that, it's opening it everytime, writing the line then closing it. You can instead open it, write all the lines, then close it.

fs.writeFileSync(path, JSON.stringify(data)) is enough in this case or fs.writeFileSync(path, JSON.stringify(['create', data])).

You can probably also get away with just \n for lines in many cases. Line endings only matter these days in niche situations and \n works well as a standard.

You should check options to see if writeFileSync supports locking but otherwise you may need to look up the lock functions. If it's not concurrency safe and is non-blocking it might need to say that in the documentation otherwise using it can mess up someone's application either due to data corruption or routine freezes.

It might suit your app, IE, you only run it once at a time per path and either can tolerate delays or are working with small amounts of data but for others it would not be even half way stable.

Once you learn it and apply it once it's like riding a bicycle. It's much more enjoyable though if you write some minimal benchmarks and tests. It's then nice to see it doing better so you get a reward for it.

Collapse
thecodrr profile image
Abdullah Atta

Would love if you showed some benchmarks instead of copy-pasting the whole API here.

Collapse
trinly01 profile image
Trinmar Boado Author

Hi Atta,
Nice suggestion!
You may contribute to this project by adding a benchmark.
Will love to merge it ❤️

Collapse
thecodrr profile image
Abdullah Atta

No it's your job to provide benchmarks. :(

Thread Thread
rezanop profile image
Rezanop

To be honest, I get your point, but I don't like this wording when it comes to open source projects :)

Thread Thread
thecodrr profile image
Abdullah Atta

Projects are projects. You can't expect other people to provide benchmarks for your project. You are the one advertising it, so you need to provide benchmarks. So yes, it is your job. Open source is not an excuse.

Collapse
roelofjanelsinga profile image
Roelof Jan Elsinga

Looks very interesting! How is the performance when you have 100.000 - 1.000.000 records?

Collapse
trinly01 profile image
Trinmar Boado Author

Hi Roelof, Contributors are welcome.
Need help to test it using real world scenarios.

Collapse
patarapolw profile image
Pacharapol Withayasakpunt

Currently I am interested in JSON schema validation, perhaps with Ajv.

Collapse
trinly01 profile image
Trinmar Boado Author

All suggestions are being considered. Some are already implemented. Thank you to all of insights and inputs ❤️