DEV Community

Cover image for Bulletproof node.js project architecture πŸ›‘οΈ

Bulletproof node.js project architecture πŸ›‘οΈ

Sam on April 18, 2019

Originally posted on softwareontheroad.com Update 04/21/2019: Implementation example in a GitHub repository Introduction Express.js is...
Collapse
 
skyjur profile image
Ski

with express if you use async handler always wrap the code with try/catch otherwise in case if something happens express will never respond

route.post('/', async (req, res, next) {
   try {
     ... code
   } catch(e) {
      next(e)
   }
})
Enter fullscreen mode Exit fullscreen mode
Collapse
 
dimpiax profile image
Dmytro Pylypenko

Or use wrapper around callback.

const wrapCatch = fn => (...args) => fn(args).catch(args[2])
...
app.get('/ping', wrapCatch(async (req, res) => {
  throw Error('break')

  res.send('pong')
}));

app.use((err, req, res, next) => {
  console.error('NOOO')
})
Collapse
 
victorzamfir profile image
Victor Zamfir

I'd suggest you go even further, like I did here - github.com/oors/oors/blob/master/p...

You can see here how to use it to generate a CRUD experience using async / await - github.com/oors/oors/blob/master/p...

Thread Thread
 
dimpiax profile image
Dmytro Pylypenko

This code is probably for the specific use case, but looks not good.
If the response is undefined, there is no further handle, just a hang.

...
.then(response => {
  if (typeof response !== 'undefined') {
    res.json(response);
  }
})
...
Thread Thread
 
victorzamfir profile image
Victor Zamfir

Not really.

You can render the response however you like (using res.json or res.render etc) or you can just return something anything that will be JSON encoded and rendered as is.

Example:

app.get(
  '/',
  wrapHandler(async (req, res) => {
    const result = await ...
    res.json(result);
  }),
);

which is the same thing as:

app.get(
  '/',
  wrapHandler(async (req, res) => {
    const result = await ...
    return result
  }),
);

But in other cases you might want to do something like this:

app.get(
  '/',
  wrapHandler(async (req, res) => {
    const result = await ...
    res.render('template', result)
  }),
);

All of these work as expected.

Thread Thread
 
dimpiax profile image
Dmytro Pylypenko • Edited

You're right!

And this will make an error by negligence.

...
wrapHandler(async (req, res) => {
  const result = await ...
  res.json(result)

  return result
})
...
Thread Thread
 
victorzamfir profile image
Victor Zamfir • Edited

Whoever writes code like that is negligent :)

It's like saying that express.js is too permissive for letting you write code like this:

...
(req, res) => {
  const result = await ...
  res.json(result);
  res.json(someOtherThing);
  res.render('someView');
}
...

I think everyone knows that it's a bad practice to do other things (including returning something) after sending the response in express.js (there are very special cases though, but they don't include the one you shared).

Thread Thread
 
dimpiax profile image
Dmytro Pylypenko

Yes, my idea to hide real response in a route is not a good idea.
Cause you have mixed flow.
In one case, you can put return in other res.SOMETHING, but not both.
It makes harder to maintain and avoid potential fails in a development team.

Thread Thread
 
victorzamfir profile image
Victor Zamfir

Yep, I see your point and you're right in a way.

I only use express.js to build APIs (I never render a template or something). So it's more like a convention (shortcut) as returing JSONs is what I do in 90% of the cases.

Thread Thread
 
dimpiax profile image
Dmytro Pylypenko

Yep, this is why I started from this message: "This code is probably for the specific use case, but looks not good."
For example, if you return await ... with undefined you will go in hang situation.

Collapse
 
xayden profile image
Abdullah Gira

Or you can use express-async-error.

# terminal
npm -i express-async-error
Enter fullscreen mode Exit fullscreen mode
// index.js
require('express-async-error');
Enter fullscreen mode Exit fullscreen mode

and that's it, any error happens the express-async-error will catch it and passes it to the default express error handler.

Collapse
 
elie222 profile image
Elie

Prefer feature based file structure. Why group all your models and all your services together. They don’t really have anything in common. They just happen to be of the same type. Better to group posts model and posts service together.

Collapse
 
sweepyoface profile image
sweepyoface

This doesn't make much sense. A model will often apply to multiple request methods and/or routes.

Collapse
 
psfeng profile image
Pin-Sho Feng • Edited

I've been using this generic structure, which I find works pretty well in practice in several languages and domains:

/common
  /models (shared models are moved here)
  /...
/feature1
  /data
    /models (optional folder, if project is too small, not worth it
  /domain
    /models
  /presentation
/feature2
  /data
  /domain
  /presentation
Thread Thread
 
santypk4 profile image
Sam

That's a good architecture too! :)

Collapse
 
yawaramin profile image
Yawar Amin

We might be talking about different 'models' here. E.g. the types that are tightly coupled to a particular controller (request) or a service should probably live in the same file or folder. But types that are not obviously coupled could go in a shared 'models' folder, but then I would argue, why are they not strongly coupled to any controller or service? Another point here is that it's OK for a controller to import the types of a service that it uses, that is just normal layered architecture for a higher level to be aware of the lower level.

Collapse
 
elie222 profile image
Elie

So import it where it's needed. Same way as you do when putting models in its folder etc.

Collapse
 
madbence profile image
Bence DΓ‘nyi

i'm not sure about the file structure. i think that when you look at a good architecture, it should be obvious what's the purpose of the application, but if you look at folders like api, services, models, that'll tell you nothing. i usually organize files by their purpose, eg. user, product, order, etc.

Collapse
 
sohaibraza profile image
SohaibRaza

I agree with you components should be self contained.

Collapse
 
aidanbeale profile image
Aidan Beale

Great architecture and file structure! I also like your bit about the loaders. Very clean way of doing it.
Do you have a skeleton setup on githib or something? I'd love to play around with this.

Collapse
 
santypk4 profile image
Sam

I'm glad you like it :)

Here is the github repository github.com/santiq/bulletproof-nodejs

Collapse
 
aidanbeale profile image
Aidan Beale

Thanks! :)

Collapse
 
agmadt profile image
Adhityo Agam

yep, waiting for the skeleton too

Collapse
 
isakkeyten profile image
Isak Keyetn

This architecture is completely different from this one
github.com/i0natan/nodebestpractices
why would you say yours is better?

Collapse
 
thorstenhirsch profile image
Thorsten Hirsch

Are you sure? I don't see any conflicts. I would say that the "nodebestpractices" focus on different things than Santiago. Actually I only find these 2 points comparable to the advice given here:

1.1 Structure your solution by components
1.2 Layer your components, keep Express within its boundaries

This is very general advice. Santiago follows this advice, but goes way further and fills it with practical instructions.

Collapse
 
isakkeyten profile image
Isak Keyetn

But his structure is not by components - but by roles. He has a models folder that holds all components models and services with all components services, instead of having a componentName folder with the respectable model and service.

Thread Thread
 
santypk4 profile image
Sam

Oh yes, the good component base architecture, that's a good way to do it too!! :)

Collapse
 
tawsbob profile image
Dellean Santos

better is not the right question, because every architecture has a purpose... what fit yours?

Collapse
 
victorzamfir profile image
Victor Zamfir

Good points.

Another one would be to adopt and follow a plugin / module - based architecture. This way you can split responsibilities and reuse those modules with other projects as well.

A frameworks that pays good respect to these principles and other similar ones is github.com/oors/oors - a framework I created.

It's modules-based, integrates with express.js, promotes a layered architecture, DI is baked in, has great support for MongoDB and GraphQL, plus much more than that.
Feel free to check out the already existing plugins and ask me any questions about it.

Collapse
 
sweepyoface profile image
sweepyoface

Just some advice – your project doesn't have any kind of helpful readme or docs. I have no idea how to use it, so I won't. Please consider putting effort into that instead of advertising it on articles :(

Collapse
 
victorzamfir profile image
Victor Zamfir

It actually does - github.com/oors/oors/tree/master/docs - but it's a bit outdated and incomplete. Nonetheless, it highlights the general idea behind the framework.
But you're completely right and documentation is in the making. It hasn't been a priority so far because I was the one to instruct the people who have been using it so far.

The framework's been out for a while now and it's been used in production on some great products.

That being said, I do plan to improve the framework and write more quality extensions, so if you're into node.js, GraphQL, MongoDB... you might want to stick around :)

Collapse
 
jacobgoh101 profile image
Jacob Goh

Thanks for this enlightening post.

It reminds me of NestJS, which is a cool Typescript framework.

Collapse
 
_manojc profile image
Manoj Chandrashekar

This is great! I have worked on many nodejs projects and each time the project structure has been a little better than the previous time. Loaders is a nice touch - I hadn't thought of that. Also shout out to Agenda.js - makes job management super simple and it's very reliable!

Collapse
 
israelmuca profile image
Israel MuΓ±oz

Hey Santiago, already followed you on Twitter, and as I said there, I really like the points you made.
It'd be awesome if you have a GitHub repo with a basic project set up with this structure.

Btw, there's an error on your link here:

check my guide on using agenda.js the best task manager for node.js.

It's pointing to:
dev.to/nodejs-scalability-issues

I guess you want it to point to your blog:
softwareontheroad.com/nodejs-scala...

Collapse
 
santypk4 profile image
Sam

Thanks, I fixed that broken link :)

Here is the example repository github.com/santiq/bulletproof-nodejs

Have fun!

Collapse
 
kbariotis profile image
Kostas Bariotis

Well done for this article! Loved it. I would only advise you to remove the middle services layer as you are using Mongoose and Mongoose is able to provide with fully featured models that can do pretty much everything like validation, custom functions, hooks, dependencies on other models, etc. etc.. The middle Services layer will only create an unneeded abstraction that you gonna hate down the road, I certainly hated mine when I realised how much power Mongoose had and that I was trying to write stuff that were already existing.

Put your model's business logic inside Mongose models, handle dependencies on other models inside those models and then let the controllers handle the glue between these models and other pieces of your architecture like sending emails, etc.

Also, make sure to explore this repository which provides a great starting point for a Node.js project structure and covers most of what you covered already. Read the reasoning behind it here.

Collapse
 
xout profile image
xout • Edited

You have a typo:

export default class UserService() {

should be

export default class UserService {

Very useful article!
Exactly what I was looking for.

You could also link some of your project with those patterns, I think it would help wrap reader's head around it.

Collapse
 
santypk4 profile image
Sam

Thanks for noticing that!

There is a link to an example project with the patterns implemented, you can check it out here github.com/santiq/bulletproof-nodejs

Collapse
 
simlu profile image
Lukas Siemon

And then you go microservices + lambda and everything becomes different (and more decentralized). Hah :) Nice writeup!

Now we need an npm packages that configures and manages your project structure for you. Feel free to take a look at my robo-config. I'll get to managing the whole project structure dynamically eventually as well :P

Collapse
 
strahinjalak profile image
Strahinja Laktovic

Awesome post, mate, very nice and organized. I'm curious about your error handling, though. How do you do it, since you have no try catches in services, nor in route handlers ?

Collapse
 
santypk4 profile image
Sam

Thank you!

I didn't write too much on good practices about error handling because I wanted to keep the examples simple and concise.

But here you have the complete repository, with proper error handling and more details.
github.com/santiq/bulletproof-nodejs

Collapse
 
strahinjalak profile image
Strahinja Laktovic

Cool, i checked out your boilerplate, I'd maybe extract error handling logic from the middleware to separate module, maybe add logging. Also, have you thought about having StatusError extending the regular one ? So that you can have comprehensive errors with statuses even from within services, instead of response 500. I'd also add that is of high importance if one should extend regular error to attach stack trace from the super class.

Thread Thread
 
btsuhako profile image
Blake

All great points!! I was thinking about logging after reading this good guide. Centralized logging would be great. Common context such as user, request-id, timing, etc. can be added to all log output. Lots of log shipping programs like to parse structured logs, and formatting in JSON makes it super easy. Also console.log() is not performant for production

@Strahinja - love of the idea of a StatusError. We do something similar in our application. Controllers can throw new Error() which our Express error handler logs as a server error and responds with an HTTP 500 to the client. Controllers can also use a custom error object and throw new ResponseError(err, 400), which logs a warning and returns an HTTP 400 to the client.

Thread Thread
 
strahinjalak profile image
Strahinja Laktovic

@blake That sounds really nice. I made a discussion regarding this. You could maybe respond there and maybe put a bit of code so we can put exchanging of ideas in motion.

Collapse
 
scootcho profile image
Scott Yu

Awesome article! I like the structure of this project and IoC. I came from Ruby on Rails so I've been using a similar MVC (or just MC, depends if serving html) structure borrowed from RoR in my Node projects. I'm going to implement these patterns in my new project right now.

I'm curious on where you would put your helpers/utils files though? Do you have a separate directory for them? Or do you stick them under the services directory that uses them?

Collapse
 
santypk4 profile image
Sam

Hi!
I use these concepts as a base for my projects, I stick to the 3 layer pattern and sometimes I create a helper's folders and put there functions not related to any service, but other times those helpers were useful to a couple of microservices so I used a private npm package.

Collapse
 
aurelmegn profile image
Aurel

Hello and thank you for your post,

I would like to suggest that you add an emphasis on the repositories,
Your post talk about the domain isolation from the controller but I think it would be great if you dive into the directory structure of the domain logic. I am referring to DDD philosophy

Collapse
 
michi profile image
Michael Z

This sounds like a lot to setup. Have you tried Adonis.js? It has a similar structure to Laravel / RoR, no custom setup needed.

Collapse
 
jaymoretti profile image
Jay Moretti

Great write up.
For env variables that are injected in other ways (via CI or Lambda) you could move the dotenv line to run when you start your node app:

"node -r dotenv/config --inspect app"

That will avoid failing to load the .env file when it's only present in one of your environments (local).

Collapse
 
hosseinnedaee profile image
Hossein Nedaee • Edited

I went through your code and learned a lot, I didn't finish it yet.
Articles about structure are very enjoyable for me. I need to learn much more about software structure and design patterns.
Thank you.

Collapse
 
santypk4 profile image
Sam

I'm really happy that it helped you !! :)

Collapse
 
hallsamuel90 profile image
Sam Hall • Edited

Love the write up! One thing I'm struggling with is where to put more complex functionality in this structure.

Say for example I had some business need to pull data from an external source, transform it, and the push it to another source. In this scenario I would need something in the api and service layer to kick off the whole thing but from there I have trouble keeping things organized. I need all of the objects from data source A and B and another service to map from A to B. Would you create a module to house all of this logic or try to keep it flat in the structure you have shown here? Thanks!

Collapse
 
pandres95 profile image
Pablo AndrΓ©s Dorado SuΓ‘rez

Several years ago, I started out a project. Initially to tackle some issues I saw in Sails, it turned out into β€”more or less, some issues like env variables to manage secrets are still on the goβ€” this exactly.

If you want to check (also, I'm looking for some contributors and anyone willing to use it) the project, you can find it here

Collapse
 
clandau profile image
Courtney

Thanks for this! When I built my first app I kind of obsessed over file structure a bit too much and couldn't find any solid resources laying out the whys. I ended up looking at a lot of github projects to figure out mine. I wish I had this back then!

Collapse
 
wickedknock profile image
wickedknock

I am new to nodejs , I have learnt express and mongodb crud , now I want to learn more , please can you only tell me the topics I have to learn myself to understand and use your repo .

Things I saw in repo I need to learn first is typescript , something called eslint and etc

Thanks <3

Collapse
 
shindesharad71 profile image
SHARAD SHINDE • Edited

Great post! This is what I am actually looking for!

Can you please review my project structure?

github.com/shindesharad71/Anstagra...

Collapse
 
santypk4 profile image
Sam

Seems very good, keep coding! :)

Be careful with the file uploads to express, it can be a real problem in production

You may want to implement a direct upload solution, here is an example with AWS S3 but I saw that you are using GCP so look for something similar.

file uploads

Collapse
 
shindesharad71 profile image
SHARAD SHINDE

Thank you for your valuable comment and guidance.

I already implemented the direct upload to GCP, no server involved.

Thanks!

Collapse
 
noway profile image
Ilia • Edited

I wouldn't put stock into this. This is a collection of anti patterns. Quite over-dogmatic. None of these patterns make sense at a small scale, neither they do in a complex codebase where every one of those paradigms get in the way of clean code.

Start small. Organise code into functions. Practice TDD. Follow functional programming. Keep directory structure flat. Don't write custom Express middleware.

The rest will follow.

Collapse
 
santypk4 profile image
Sam

It’s seems like you are having a bad day.
I send you a virtual hug buddy :)

Collapse
 
al_karlssen profile image
Al Karlssen

Node.js is an open-source, JavaScript run-time environment used to execute JavaScript code on the server-side. Node.js itechcraft.com/node-js/ development process has changed the paradigm that JavaScript is used primarily on the client-side. That’s why Node.js has become one of the foundational elements of the β€œJavaScript everywhere” paradigm.

Collapse
 
quentin_mrt profile image
Mauret πŸ‘¨πŸΌβ€πŸ’»

Hi, nice work ! I just don't understand your unit test. I understood that you had mock UserModel with create method, but when I try to do the same in my test i have got an error :

Argument of type '{ find: () => IUser[]; }' is not assignable to parameter of type 'UserModel'.
Type '{ find: () => IUser[]; }' is missing the following properties from type 'UserModel': watch, translateAliases, bulkWrite, findById, and 58 more.

Any idea to fix this ? Maybe I have to do this differently

Collapse
 
melitus profile image
Aroh Sunday

Thanks, Sam, this is a wonderful and detailed explanation. I would appreciate seeing the full example of the pub/sub sample with eventemitter as demonstrated above. How can memory leaks be checked and avoided in a bigger nodejs app with EventEmitter? Once again, thanks

Collapse
 
codingstatus profile image
CodingStatus.com

This tutorial suggest me a basic & best way about MVC structure, Using this concept, I have developed MVC folder structure in Express with CRUD Example. So, Thanks for sharing..

Collapse
 
samrocksc profile image
Sam Clark

really nice breakdown! Thanks!

Collapse
 
thorstenhirsch profile image
Thorsten Hirsch

Awesome project structure. I especially like your addition of the pub/sub layer.

Collapse
 
pebcakerror profile image
Jonathan Worent

How might the architecture look if the data layer is a 3rd party API?

Collapse
 
santypk4 profile image
Sam

Should not be so different.

A wrapper class needs to be written for every API resource, with a compatible interface across all models.


class UserModel(){

   public UpdateOne(obj: { selector: any, update: any }):Promise<any> {
     // ...API Call
   }

}

Some methods, for example, UpdateMany may not be present in some third-party API resources and so a custom logic needs to be written.


class CompanyModel(){
   public UpdateOne(obj: { selector: any, update: any }):Promise<any> {
     // ...API Call
   }
   public UpdateMany(objs: Array<{ selector: any, update: any }>):Promise<Array<any>> {
     return Promise.all(
           updates.map(u => {
              return this.UpdateOne(u.selector, u.update);
          })
     )
   }

}

Collapse
 
andreasbergqvist profile image
Andreas Bergqvist

Great post! Thanks for sharing!

Collapse
 
becklin profile image
Beck Lin • Edited

Great article and lots of valuable points ! learn so much from your architecture while Im working on my on nodejs project.
Just want to know your opinion regarding the future of nodejs. It seems that the market of nodejs is not so popular as java or python.

Collapse
 
artoodeeto profile image
aRtoo

Question. What if I wanted to add react. where should I put it? should there be only one node modules for frontend and backend? and what if I wanted to dockerize each? thank you.

Collapse
 
fedemengo profile image
Federico Mengozzi • Edited

Hello, awesome article. Would it be possible to have a list of all patterns used in the project?

Collapse
 
santypk4 profile image
Sam

Sure.

The project follows the 3 tier (or layers) pattern for structuring the code and separate the controllers, services and data access layer.

3 tier pattern

Also, there are some events involved, following the Pub/Sub pattern and inversion of control

And of course, as other developers pointed, you can go nuts and implement something like the 'Clean Domain Driven Design' architecture but is a little overwhelming for little applications.

The list of patterns used here

  • 3 tiers pattern
  • Pub/Sub pattern
  • Inversion of control
Collapse
 
fedemengo profile image
Federico Mengozzi

Thanks! This is really helpful

Collapse
 
unruhschuh profile image
Thomas Leitz

Awesome post! In the app.js listing you have

await loaders.init({ expressApp: app });

yet, loaders/index.js doesn't define in init function or did I miss something?

Collapse
 
meet_zaveri profile image
Meet Zaveri • Edited

Though I am a frontend dev, this gave me good insights of structuring code architecture and seperating concerns. Great write up!

Collapse
 
rommik profile image
Roman Mikhailov

Loved it! Simple, clean, logical, and extendable!

Collapse
 
andersonjoseph profile image
Anderson. J

Awesome architecture! It looks really scalable.

Collapse
 
somedood profile image
Basti Ortiz

This was a really good read. Thanks for this! This is exactly what I needed to improve my "architecture skills". Keep it up!

Collapse
 
mkalatrash profile image
Mohammad Alatrash

Nice article and architecture, the loaders part is really good. Thank you!

Collapse
 
solianir profile image
Pedro Ivan Partida Galarza

Why use typedi instead of plain old higher-order functions? Functions are first-class citizens in JS/TS, so why not use HOCs instead of adding an additional dependency that applies a pattern that only makes sense in languages without HOCs?

Collapse
 
quentin_mrt profile image
Mauret πŸ‘¨πŸΌβ€πŸ’»

Hi, thank's for this architecture, i found it perfect for scale API with Node.js ! Adding Typescript & loaders are great ideas

Collapse
 
aslasn profile image
Ande • Edited

Why do we need two different api and services folder? The subscribers or pub/sub also falls under services.

Collapse
 
tusharshahi profile image
TusharShahi

Great article.

Maybe you want to add on to this :
An imperative call to a dependent service is not the best way of doing it.

Can you explain why?

Collapse
 
m07ameda7med profile image
Mohamed Ahmed

Thanks for the great article 🀩, I knew most of them but great to put all these in one place, but what about error handling it would be great if u discuss that too, thanks again πŸ™

Collapse
 
whydonti profile image
Nikhil Bhandarkar

Great post, great explanation.
But I can't figure out how to handle errors for different scenarios when I should not return any error message or http status from service layer.

Collapse
 
cthulhu profile image
Catalin Ciubotaru

Really nice article. I am more of a front end developer now but coming from a heavy .NET background and this made a lot of sense. Great job!

Collapse
 
shanshaji_8 profile image
Shan Shaji • Edited

Inspired by this, but didn't use typescript, DI
node-js-starter

Collapse
 
sraveend profile image
Sreeharsha

I love this thank you!

Collapse
 
prajwalch profile image
Prajwal Chapagain

Hey, but how should i load env using the config? should import on each module where it require?

Collapse
 
abidullah786 profile image
ABIDULLAH786

Well explained with simple examplesβ™₯️

Collapse
 
digen21 profile image
Digen More

How to create Database Access Layer/Object
In service based architecture?