DEV Community

loading...
Cover image for Bulletproof node.js project architecture 🛡️

Bulletproof node.js project architecture 🛡️

Sam
Node.js Developer, AWS Lover, JavaScript advocate, React.js is my friend.
Originally published at softwareontheroad.com Updated on ・11 min read

Originally posted on softwareontheroad.com

Update 04/21/2019: Implementation example in a GitHub repository

Introduction

Express.js is great frameworks for making a node.js REST APIs however it doesn't give you any clue on how to organizing your node.js project.

While it may sound silly, this is a real problem.

The correct organization of your node.js project structure will avoid duplication of code, will improve stability, and potentially, will help you scale your services if is done correctly.

This post is extense research, from my years of experience dealing with a poor structured node.js project, bad patterns, and countless hours of refactoring code and moving things around.

If you need help to align your node.js project architecture, just drop me a letter at santiago@softwareontheroad.com

Table of contents

The folder structure 🏢

Here is the node.js project structure that I'm talking about.

I use this in every node.js REST API service that I build, let's see in details what every component do.

  src
  │   app.js          # App entry point
  └───api             # Express route controllers for all the endpoints of the app
  └───config          # Environment variables and configuration related stuff
  └───jobs            # Jobs definitions for agenda.js
  └───loaders         # Split the startup process into modules
  └───models          # Database models
  └───services        # All the business logic is here
  └───subscribers     # Event handlers for async task
  └───types           # Type declaration files (d.ts) for Typescript
Enter fullscreen mode Exit fullscreen mode

It is more than just a way of ordering javascript files...

3 Layer architecture 🥪

The idea is to use the principle of separation of concerns to move the business logic away from the node.js API Routes.

3 layer pattern

Because someday, you will want to use your business logic on a CLI tool, or not going far, in a recurring task.

And make an API call from the node.js server to itself it's not a good idea...

3 layer pattern for node.js REST API

☠️ Don't put your business logic inside the controllers!! ☠️

You may be tempted to just use the express.js controllers to store the business logic of your application, but this quickly becomes spaghetti code, as soon as you need to write unit tests, you will end up dealing with complex mocks for req or res express.js objects.

It's complicated to distingue when a response should be sent, and when to continue processing in 'background', let's say after the response is sent to the client.

Here is an example of what not to do.

  route.post('/', async (req, res, next) => {

    // This should be a middleware or should be handled by a library like Joi.
    const userDTO = req.body;
    const isUserValid = validators.user(userDTO)
    if(!isUserValid) {
      return res.status(400).end();
    }

    // Lot of business logic here...
    const userRecord = await UserModel.create(userDTO);
    delete userRecord.password;
    delete userRecord.salt;
    const companyRecord = await CompanyModel.create(userRecord);
    const companyDashboard = await CompanyDashboard.create(userRecord, companyRecord);

    ...whatever...


    // And here is the 'optimization' that mess up everything.
    // The response is sent to client...
    res.json({ user: userRecord, company: companyRecord });

    // But code execution continues :(
    const salaryRecord = await SalaryModel.create(userRecord, companyRecord);
    eventTracker.track('user_signup',userRecord,companyRecord,salaryRecord);
    intercom.createUser(userRecord);
    gaAnalytics.event('user_signup',userRecord);
    await EmailService.startSignupSequence(userRecord)
  });

Enter fullscreen mode Exit fullscreen mode

# Use a service layer for your business logic 💼

This layer is where your business logic should live.

It's just a collection of classes with clear porpuses, following the SOLID principles applied to node.js.

In this layer there should not exists any form of 'SQL query', use the data access layer for that.

  • Move your code away from the express.js router

  • Don't pass the req or res object to the service layer

  • Don't return anything related to the HTTP transport layer like a status code or headers from the service layer.

Example

  route.post('/', 
    validators.userSignup, // this middleware take care of validation
    async (req, res, next) => {
      // The actual responsability of the route layer.
      const userDTO = req.body;

      // Call to service layer.
      // Abstraction on how to access the data layer and the business logic.
      const { user, company } = await UserService.Signup(userDTO);

      // Return a response to client.
      return res.json({ user, company });
    });
Enter fullscreen mode Exit fullscreen mode

Here is how your service will be working behind the scenes.

  import UserModel from '../models/user';
  import CompanyModel from '../models/company';

  export default class UserService {

    async Signup(user) {
      const userRecord = await UserModel.create(user);
      const companyRecord = await CompanyModel.create(userRecord); // needs userRecord to have the database id 
      const salaryRecord = await SalaryModel.create(userRecord, companyRecord); // depends on user and company to be created

      ...whatever

      await EmailService.startSignupSequence(userRecord)

      ...do more stuff

      return { user: userRecord, company: companyRecord };
    }
  }

Enter fullscreen mode Exit fullscreen mode

Visit the example repository

Use a Pub/Sub layer too 🎙️

The pub/sub pattern goes beyond the classic 3 layer architecture proposed here but it's extremely useful.

The simple node.js API endpoint that creates a user right now, may want to call third-party services, maybe to an analytics service, or maybe start an email sequence.

Sooner than later, that simple "create" operation will be doing several things, and you will end up with 1000 lines of code, all in a single function.

That violates the principle of single responsibility.

So, it's better to separate responsibilities from the start, so your code remains maintainable.

  import UserModel from '../models/user';
  import CompanyModel from '../models/company';
  import SalaryModel from '../models/salary';

  export default class UserService() {

    async Signup(user) {
      const userRecord = await UserModel.create(user);
      const companyRecord = await CompanyModel.create(user);
      const salaryRecord = await SalaryModel.create(user, salary);

      eventTracker.track(
        'user_signup',
        userRecord,
        companyRecord,
        salaryRecord
      );

      intercom.createUser(
        userRecord
      );

      gaAnalytics.event(
        'user_signup',
        userRecord
      );

      await EmailService.startSignupSequence(userRecord)

      ...more stuff

      return { user: userRecord, company: companyRecord };
    }

  }
Enter fullscreen mode Exit fullscreen mode

An imperative call to a dependent service is not the best way of doing it.

A better approach is by emitting an event i.e. 'a user signed up with this email'.

And you are done, now it's the responsibility of the listeners to do their job.

  import UserModel from '../models/user';
  import CompanyModel from '../models/company';
  import SalaryModel from '../models/salary';

  export default class UserService() {

    async Signup(user) {
      const userRecord = await this.userModel.create(user);
      const companyRecord = await this.companyModel.create(user);
      this.eventEmitter.emit('user_signup', { user: userRecord, company: companyRecord })
      return userRecord
    }

  }
Enter fullscreen mode Exit fullscreen mode

Now you can split the event handlers/listeners into multiple files.

  eventEmitter.on('user_signup', ({ user, company }) => {

    eventTracker.track(
      'user_signup',
      user,
      company,
    );

    intercom.createUser(
      user
    );

    gaAnalytics.event(
      'user_signup',
      user
    );
  })
Enter fullscreen mode Exit fullscreen mode
  eventEmitter.on('user_signup', async ({ user, company }) => {
    const salaryRecord = await SalaryModel.create(user, company);
  })
Enter fullscreen mode Exit fullscreen mode
  eventEmitter.on('user_signup', async ({ user, company }) => {
    await EmailService.startSignupSequence(user)
  })
Enter fullscreen mode Exit fullscreen mode

You can wrap the await statements into a try-catch block or you can just let it fail and handle the 'unhandledPromise' process.on('unhandledRejection',cb)

Dependency Injection 💉

D.I. or inversion of control (IoC) is a common pattern that will help the organization of your code, by 'injecting' or passing through the constructor the dependencies of your class or function.

By doing this way you will gain the flexibility to inject a 'compatible dependency' when, for example, you write the unit tests for the service, or when the service is used in another context.

Code with no D.I

  import UserModel from '../models/user';
  import CompanyModel from '../models/company';
  import SalaryModel from '../models/salary';  
  class UserService {
    constructor(){}
    Sigup(){
      // Caling UserMode, CompanyModel, etc
      ...
    }
  }
Enter fullscreen mode Exit fullscreen mode

Code with manual dependency injection

  export default class UserService {
    constructor(userModel, companyModel, salaryModel){
      this.userModel = userModel;
      this.companyModel = companyModel;
      this.salaryModel = salaryModel;
    }
    getMyUser(userId){
      // models available throug 'this'
      const user = this.userModel.findById(userId);
      return user;
    }
  }
Enter fullscreen mode Exit fullscreen mode

Now you can inject custom dependencies.

  import UserService from '../services/user';
  import UserModel from '../models/user';
  import CompanyModel from '../models/company';
  const salaryModelMock = {
    calculateNetSalary(){
      return 42;
    }
  }
  const userServiceInstance = new UserService(userModel, companyModel, salaryModelMock);
  const user = await userServiceInstance.getMyUser('12346');
Enter fullscreen mode Exit fullscreen mode

The amount of dependencies a service can have is infinite, and refactor every instantiation of it when you add a new one is a boring and error-prone task.

That's why dependency injection frameworks were created.

The idea is you declare your dependencies in the class, and when you need an instance of that class, you just call the 'Service Locator'.

Let's see an example using typedi an npm library that brings D.I to node.js

You can read more on how to use typedi in the official documentation

WARNING typescript example

  import { Service } from 'typedi';
  @Service()
  export default class UserService {
    constructor(
      private userModel,
      private companyModel, 
      private salaryModel
    ){}

    getMyUser(userId){
      const user = this.userModel.findById(userId);
      return user;
    }
  }
Enter fullscreen mode Exit fullscreen mode

services/user.ts

Now typedi will take care of resolving any dependency the UserService require.

  import { Container } from 'typedi';
  import UserService from '../services/user';
  const userServiceInstance = Container.get(UserService);
  const user = await userServiceInstance.getMyUser('12346');
Enter fullscreen mode Exit fullscreen mode

Abusing service locator calls is an anti-pattern

Using Dependency Injection with Express.js in Node.js

Using D.I. in express.js is the final piece of the puzzle for this node.js project architecture.

Routing layer

  route.post('/', 
    async (req, res, next) => {
      const userDTO = req.body;

      const userServiceInstance = Container.get(UserService) // Service locator

      const { user, company } = userServiceInstance.Signup(userDTO);

      return res.json({ user, company });
    });
Enter fullscreen mode Exit fullscreen mode

Awesome, project is looking great!
It's so organized that makes me want to be coding something right now.

Visit the example repository

An unit test example 🕵🏻

By using dependency injection and these organization patterns, unit testing becomes really simple.

You don't have to mock req/res objects or require(...) calls.

Example: Unit test for signup user method

tests/unit/services/user.js

  import UserService from '../../../src/services/user';

  describe('User service unit tests', () => {
    describe('Signup', () => {
      test('Should create user record and emit user_signup event', async () => {
        const eventEmitterService = {
          emit: jest.fn(),
        };

        const userModel = {
          create: (user) => {
            return {
              ...user,
              _id: 'mock-user-id'
            }
          },
        };

        const companyModel = {
          create: (user) => {
            return {
              owner: user._id,
              companyTaxId: '12345',
            }
          },
        };

        const userInput= {
          fullname: 'User Unit Test',
          email: 'test@example.com',
        };

        const userService = new UserService(userModel, companyModel, eventEmitterService);
        const userRecord = await userService.SignUp(teamId.toHexString(), userInput);

        expect(userRecord).toBeDefined();
        expect(userRecord._id).toBeDefined();
        expect(eventEmitterService.emit).toBeCalled();
      });
    })
  })

Enter fullscreen mode Exit fullscreen mode

Cron Jobs and recurring task ⚡

So, now that the business logic encapsulated into the service layer, it's easier to use it from a Cron job.

You should never rely on node.js setTimeout or another primitive way of delay the execution of code, but on a framework that persist your jobs, and the execution of them, in a database.

This way you will have control over the failed jobs, and feedback of those who succeed.
I already wrote on good practice for this so, check my guide on using agenda.js the best task manager for node.js.

Configurations and secrets 🤫

Following the battle-tested concepts of Twelve-Factor App for node.js the best approach to store API Keys and database string connections, it's by using dotenv.

Put a .env file, that must never be committed (but it has to exist with default values in your repository) then, the npm package dotenv loads the .env file and insert the vars into the process.env object of node.js.

That could be enough but, I like to add an extra step.
Have a config/index.ts file where the dotenv npm package and loads the .env file and then I use an object to store the variables, so we have a structure and code autocompletion.

config/index.js

  const dotenv = require('dotenv');
  // config() will read your .env file, parse the contents, assign it to process.env.
  dotenv.config();

  export default {
    port: process.env.PORT,
    databaseURL: process.env.DATABASE_URI,
    paypal: {
      publicKey: process.env.PAYPAL_PUBLIC_KEY,
      secretKey: process.env.PAYPAL_SECRET_KEY,
    },
    paypal: {
      publicKey: process.env.PAYPAL_PUBLIC_KEY,
      secretKey: process.env.PAYPAL_SECRET_KEY,
    },
    mailchimp: {
      apiKey: process.env.MAILCHIMP_API_KEY,
      sender: process.env.MAILCHIMP_SENDER,
    }
  }
Enter fullscreen mode Exit fullscreen mode

This way you avoid flooding your code with process.env.MY_RANDOM_VAR instructions, and by having the autocompletion you don't have to know how to name the env var.

Visit the example repository

Loaders 🏗️

I took this pattern from W3Tech microframework but without depending upon their package.

The idea is that you split the startup process of your node.js service into testable modules.

Let's see a classic express.js app initialization

  const mongoose = require('mongoose');
  const express = require('express');
  const bodyParser = require('body-parser');
  const session = require('express-session');
  const cors = require('cors');
  const errorhandler = require('errorhandler');
  const app = express();

  app.get('/status', (req, res) => { res.status(200).end(); });
  app.head('/status', (req, res) => { res.status(200).end(); });
  app.use(cors());
  app.use(require('morgan')('dev'));
  app.use(bodyParser.urlencoded({ extended: false }));
  app.use(bodyParser.json(setupForStripeWebhooks));
  app.use(require('method-override')());
  app.use(express.static(__dirname + '/public'));
  app.use(session({ secret: process.env.SECRET, cookie: { maxAge: 60000 }, resave: false, saveUninitialized: false }));
  mongoose.connect(process.env.DATABASE_URL, { useNewUrlParser: true });

  require('./config/passport');
  require('./models/user');
  require('./models/company');
  app.use(require('./routes'));
  app.use((req, res, next) => {
    var err = new Error('Not Found');
    err.status = 404;
    next(err);
  });
  app.use((err, req, res) => {
    res.status(err.status || 500);
    res.json({'errors': {
      message: err.message,
      error: {}
    }});
  });


  ... more stuff 

  ... maybe start up Redis

  ... maybe add more middlewares

  async function startServer() {    
    app.listen(process.env.PORT, err => {
      if (err) {
        console.log(err);
        return;
      }
      console.log(`Your server is ready !`);
    });
  }

  // Run the async function to start our server
  startServer();
Enter fullscreen mode Exit fullscreen mode

As you see, this part of your application can be a real mess.

Here is an effective way to deal with it.

  const loaders = require('./loaders');
  const express = require('express');

  async function startServer() {

    const app = express();

    await loaders.init({ expressApp: app });

    app.listen(process.env.PORT, err => {
      if (err) {
        console.log(err);
        return;
      }
      console.log(`Your server is ready !`);
    });
  }

  startServer();
Enter fullscreen mode Exit fullscreen mode

Now the loaders are just tiny files with a concise purpose

loaders/index.js

  import expressLoader from './express';
  import mongooseLoader from './mongoose';

  export default async ({ expressApp }) => {
    const mongoConnection = await mongooseLoader();
    console.log('MongoDB Intialized');
    await expressLoader({ app: expressApp });
    console.log('Express Intialized');

    // ... more loaders can be here

    // ... Initialize agenda
    // ... or Redis, or whatever you want
  }
Enter fullscreen mode Exit fullscreen mode

The express loader

loaders/express.js


  import * as express from 'express';
  import * as bodyParser from 'body-parser';
  import * as cors from 'cors';

  export default async ({ app }: { app: express.Application }) => {

    app.get('/status', (req, res) => { res.status(200).end(); });
    app.head('/status', (req, res) => { res.status(200).end(); });
    app.enable('trust proxy');

    app.use(cors());
    app.use(require('morgan')('dev'));
    app.use(bodyParser.urlencoded({ extended: false }));

    // ...More middlewares

    // Return the express app
    return app;
  })

Enter fullscreen mode Exit fullscreen mode

The mongo loader

loaders/mongoose.js

  import * as mongoose from 'mongoose'
  export default async (): Promise<any> => {
    const connection = await mongoose.connect(process.env.DATABASE_URL, { useNewUrlParser: true });
    return connection.connection.db;
  }
Enter fullscreen mode Exit fullscreen mode

See a complete example of loaders here

Conclusion

We deep dive into a production tested node.js project structure, here are some summarized tips:

  • Use a 3 layer architecture.

  • Don't put your business logic into the express.js controllers.

  • Use PubSub pattern and emit events for background tasks.

  • Have dependency injection for your peace of mind.

  • Never leak your passwords, secrets and API keys, use a configuration manager.

  • Split your node.js server configurations into small modules that can be loaded independently.

See the example repository here

✋ Hey ! Before you go 🏃‍

If you enjoy this article, I recommend you to subscribe to my email list so you never miss another one like this. ⬇️ ⬇️

Email List Form

I will not try to sell you anything, I promise

And don't miss my latest post, I believe you will love it :)

Read my research on the most downloaded frontend framework, the result will surprise you!

Don't forget to visit my blog to get more awesome posts like this softwareontheroad.com

Discussion (91)

Collapse
skyjur profile image
Ski

with express if you use async handler always wrap the code with try/catch otherwise in case if something happens express will never respond

route.post('/', async (req, res, next) {
   try {
     ... code
   } catch(e) {
      next(e)
   }
})
Enter fullscreen mode Exit fullscreen mode
Collapse
dimpiax profile image
Dmytro Pylypenko

Or use wrapper around callback.

const wrapCatch = fn => (...args) => fn(args).catch(args[2])
...
app.get('/ping', wrapCatch(async (req, res) => {
  throw Error('break')

  res.send('pong')
}));

app.use((err, req, res, next) => {
  console.error('NOOO')
})
Collapse
victorzamfir profile image
Victor Zamfir

I'd suggest you go even further, like I did here - github.com/oors/oors/blob/master/p...

You can see here how to use it to generate a CRUD experience using async / await - github.com/oors/oors/blob/master/p...

Thread Thread
dimpiax profile image
Dmytro Pylypenko

This code is probably for the specific use case, but looks not good.
If the response is undefined, there is no further handle, just a hang.

...
.then(response => {
  if (typeof response !== 'undefined') {
    res.json(response);
  }
})
...
Thread Thread
victorzamfir profile image
Victor Zamfir

Not really.

You can render the response however you like (using res.json or res.render etc) or you can just return something anything that will be JSON encoded and rendered as is.

Example:

app.get(
  '/',
  wrapHandler(async (req, res) => {
    const result = await ...
    res.json(result);
  }),
);

which is the same thing as:

app.get(
  '/',
  wrapHandler(async (req, res) => {
    const result = await ...
    return result
  }),
);

But in other cases you might want to do something like this:

app.get(
  '/',
  wrapHandler(async (req, res) => {
    const result = await ...
    res.render('template', result)
  }),
);

All of these work as expected.

Thread Thread
dimpiax profile image
Dmytro Pylypenko • Edited

You're right!

And this will make an error by negligence.

...
wrapHandler(async (req, res) => {
  const result = await ...
  res.json(result)

  return result
})
...
Thread Thread
victorzamfir profile image
Victor Zamfir • Edited

Whoever writes code like that is negligent :)

It's like saying that express.js is too permissive for letting you write code like this:

...
(req, res) => {
  const result = await ...
  res.json(result);
  res.json(someOtherThing);
  res.render('someView');
}
...

I think everyone knows that it's a bad practice to do other things (including returning something) after sending the response in express.js (there are very special cases though, but they don't include the one you shared).

Thread Thread
dimpiax profile image
Dmytro Pylypenko

Yes, my idea to hide real response in a route is not a good idea.
Cause you have mixed flow.
In one case, you can put return in other res.SOMETHING, but not both.
It makes harder to maintain and avoid potential fails in a development team.

Thread Thread
victorzamfir profile image
Victor Zamfir

Yep, I see your point and you're right in a way.

I only use express.js to build APIs (I never render a template or something). So it's more like a convention (shortcut) as returing JSONs is what I do in 90% of the cases.

Thread Thread
dimpiax profile image
Dmytro Pylypenko

Yep, this is why I started from this message: "This code is probably for the specific use case, but looks not good."
For example, if you return await ... with undefined you will go in hang situation.

Collapse
xayden profile image
Abdullah Osama

Or you can use express-async-error.

# terminal
npm -i express-async-error
Enter fullscreen mode Exit fullscreen mode
// index.js
require('express-async-error');
Enter fullscreen mode Exit fullscreen mode

and that's it, any error happens the express-async-error will catch it and passes it to the default express error handler.

Collapse
elie222 profile image
Elie

Prefer feature based file structure. Why group all your models and all your services together. They don’t really have anything in common. They just happen to be of the same type. Better to group posts model and posts service together.

Collapse
sweepyoface profile image
sweepyoface

This doesn't make much sense. A model will often apply to multiple request methods and/or routes.

Collapse
psfeng profile image
Pin-Sho Feng • Edited

I've been using this generic structure, which I find works pretty well in practice in several languages and domains:

/common
  /models (shared models are moved here)
  /...
/feature1
  /data
    /models (optional folder, if project is too small, not worth it
  /domain
    /models
  /presentation
/feature2
  /data
  /domain
  /presentation
Thread Thread
santypk4 profile image
Sam Author

That's a good architecture too! :)

Collapse
yawaramin profile image
Yawar Amin

We might be talking about different 'models' here. E.g. the types that are tightly coupled to a particular controller (request) or a service should probably live in the same file or folder. But types that are not obviously coupled could go in a shared 'models' folder, but then I would argue, why are they not strongly coupled to any controller or service? Another point here is that it's OK for a controller to import the types of a service that it uses, that is just normal layered architecture for a higher level to be aware of the lower level.

Collapse
elie222 profile image
Elie

So import it where it's needed. Same way as you do when putting models in its folder etc.

Collapse
madbence profile image
Bence Dányi

i'm not sure about the file structure. i think that when you look at a good architecture, it should be obvious what's the purpose of the application, but if you look at folders like api, services, models, that'll tell you nothing. i usually organize files by their purpose, eg. user, product, order, etc.

Collapse
sohaibraza profile image
SohaibRaza

I agree with you components should be self contained.

Collapse
aidanbeale profile image
Aidan Beale

Great architecture and file structure! I also like your bit about the loaders. Very clean way of doing it.
Do you have a skeleton setup on githib or something? I'd love to play around with this.

Collapse
santypk4 profile image
Sam Author

I'm glad you like it :)

Here is the github repository github.com/santiq/bulletproof-nodejs

Collapse
aidanbeale profile image
Aidan Beale

Thanks! :)

Collapse
agmadt profile image
Adhityo Agam

yep, waiting for the skeleton too

Collapse
isakkeyten profile image
Марио Рајчиќ

This architecture is completely different from this one
github.com/i0natan/nodebestpractices
why would you say yours is better?

Collapse
thorstenhirsch profile image
Thorsten Hirsch

Are you sure? I don't see any conflicts. I would say that the "nodebestpractices" focus on different things than Santiago. Actually I only find these 2 points comparable to the advice given here:

1.1 Structure your solution by components
1.2 Layer your components, keep Express within its boundaries

This is very general advice. Santiago follows this advice, but goes way further and fills it with practical instructions.

Collapse
isakkeyten profile image
Марио Рајчиќ

But his structure is not by components - but by roles. He has a models folder that holds all components models and services with all components services, instead of having a componentName folder with the respectable model and service.

Thread Thread
santypk4 profile image
Sam Author

Oh yes, the good component base architecture, that's a good way to do it too!! :)

Collapse
tawsbob profile image
Dellean Santos

better is not the right question, because every architecture has a purpose... what fit yours?

Collapse
_manojc profile image
Manoj Chandrashekar

This is great! I have worked on many nodejs projects and each time the project structure has been a little better than the previous time. Loaders is a nice touch - I hadn't thought of that. Also shout out to Agenda.js - makes job management super simple and it's very reliable!

Collapse
victorzamfir profile image
Victor Zamfir

Good points.

Another one would be to adopt and follow a plugin / module - based architecture. This way you can split responsibilities and reuse those modules with other projects as well.

A frameworks that pays good respect to these principles and other similar ones is github.com/oors/oors - a framework I created.

It's modules-based, integrates with express.js, promotes a layered architecture, DI is baked in, has great support for MongoDB and GraphQL, plus much more than that.
Feel free to check out the already existing plugins and ask me any questions about it.

Collapse
sweepyoface profile image
sweepyoface

Just some advice – your project doesn't have any kind of helpful readme or docs. I have no idea how to use it, so I won't. Please consider putting effort into that instead of advertising it on articles :(

Collapse
victorzamfir profile image
Victor Zamfir

It actually does - github.com/oors/oors/tree/master/docs - but it's a bit outdated and incomplete. Nonetheless, it highlights the general idea behind the framework.
But you're completely right and documentation is in the making. It hasn't been a priority so far because I was the one to instruct the people who have been using it so far.

The framework's been out for a while now and it's been used in production on some great products.

That being said, I do plan to improve the framework and write more quality extensions, so if you're into node.js, GraphQL, MongoDB... you might want to stick around :)

Collapse
jacobgoh101 profile image
Jacob Goh

Thanks for this enlightening post.

It reminds me of NestJS, which is a cool Typescript framework.

Collapse
strahinjalak profile image
Strahinja Laktovic

Awesome post, mate, very nice and organized. I'm curious about your error handling, though. How do you do it, since you have no try catches in services, nor in route handlers ?

Collapse
santypk4 profile image
Sam Author

Thank you!

I didn't write too much on good practices about error handling because I wanted to keep the examples simple and concise.

But here you have the complete repository, with proper error handling and more details.
github.com/santiq/bulletproof-nodejs

Collapse
strahinjalak profile image
Strahinja Laktovic

Cool, i checked out your boilerplate, I'd maybe extract error handling logic from the middleware to separate module, maybe add logging. Also, have you thought about having StatusError extending the regular one ? So that you can have comprehensive errors with statuses even from within services, instead of response 500. I'd also add that is of high importance if one should extend regular error to attach stack trace from the super class.

Thread Thread
btsuhako profile image
Blake

All great points!! I was thinking about logging after reading this good guide. Centralized logging would be great. Common context such as user, request-id, timing, etc. can be added to all log output. Lots of log shipping programs like to parse structured logs, and formatting in JSON makes it super easy. Also console.log() is not performant for production

@Strahinja - love of the idea of a StatusError. We do something similar in our application. Controllers can throw new Error() which our Express error handler logs as a server error and responds with an HTTP 500 to the client. Controllers can also use a custom error object and throw new ResponseError(err, 400), which logs a warning and returns an HTTP 400 to the client.

Thread Thread
strahinjalak profile image
Strahinja Laktovic

@blake That sounds really nice. I made a discussion regarding this. You could maybe respond there and maybe put a bit of code so we can put exchanging of ideas in motion.

Collapse
xout profile image
xout • Edited

You have a typo:

export default class UserService() {

should be

export default class UserService {

Very useful article!
Exactly what I was looking for.

You could also link some of your project with those patterns, I think it would help wrap reader's head around it.

Collapse
santypk4 profile image
Sam Author

Thanks for noticing that!

There is a link to an example project with the patterns implemented, you can check it out here github.com/santiq/bulletproof-nodejs

Collapse
kbariotis profile image
Kostas Bariotis

Well done for this article! Loved it. I would only advise you to remove the middle services layer as you are using Mongoose and Mongoose is able to provide with fully featured models that can do pretty much everything like validation, custom functions, hooks, dependencies on other models, etc. etc.. The middle Services layer will only create an unneeded abstraction that you gonna hate down the road, I certainly hated mine when I realised how much power Mongoose had and that I was trying to write stuff that were already existing.

Put your model's business logic inside Mongose models, handle dependencies on other models inside those models and then let the controllers handle the glue between these models and other pieces of your architecture like sending emails, etc.

Also, make sure to explore this repository which provides a great starting point for a Node.js project structure and covers most of what you covered already. Read the reasoning behind it here.

Collapse
simlu profile image
Lukas Siemon

And then you go microservices + lambda and everything becomes different (and more decentralized). Hah :) Nice writeup!

Now we need an npm packages that configures and manages your project structure for you. Feel free to take a look at my robo-config. I'll get to managing the whole project structure dynamically eventually as well :P

Collapse
israelmuca profile image
Israel Muñoz

Hey Santiago, already followed you on Twitter, and as I said there, I really like the points you made.
It'd be awesome if you have a GitHub repo with a basic project set up with this structure.

Btw, there's an error on your link here:

check my guide on using agenda.js the best task manager for node.js.

It's pointing to:
dev.to/nodejs-scalability-issues

I guess you want it to point to your blog:
softwareontheroad.com/nodejs-scala...

Collapse
santypk4 profile image
Sam Author

Thanks, I fixed that broken link :)

Here is the example repository github.com/santiq/bulletproof-nodejs

Have fun!

Collapse
hosseinnedaee profile image
Hossein Nedaee • Edited

I went through your code and learned a lot, I didn't finish it yet.
Articles about structure are very enjoyable for me. I need to learn much more about software structure and design patterns.
Thank you.

Collapse
santypk4 profile image
Sam Author

I'm really happy that it helped you !! :)

Collapse
michi profile image
Michael Z

This sounds like a lot to setup. Have you tried Adonis.js? It has a similar structure to Laravel / RoR, no custom setup needed.

Collapse
jaymoretti profile image
Jay Moretti

Great write up.
For env variables that are injected in other ways (via CI or Lambda) you could move the dotenv line to run when you start your node app:

"node -r dotenv/config --inspect app"

That will avoid failing to load the .env file when it's only present in one of your environments (local).

Collapse
aurelmegn profile image
Aurel

Hello and thank you for your post,

I would like to suggest that you add an emphasis on the repositories,
Your post talk about the domain isolation from the controller but I think it would be great if you dive into the directory structure of the domain logic. I am referring to DDD philosophy

Collapse
scootcho profile image
Scott Yu

Awesome article! I like the structure of this project and IoC. I came from Ruby on Rails so I've been using a similar MVC (or just MC, depends if serving html) structure borrowed from RoR in my Node projects. I'm going to implement these patterns in my new project right now.

I'm curious on where you would put your helpers/utils files though? Do you have a separate directory for them? Or do you stick them under the services directory that uses them?

Collapse
santypk4 profile image
Sam Author

Hi!
I use these concepts as a base for my projects, I stick to the 3 layer pattern and sometimes I create a helper's folders and put there functions not related to any service, but other times those helpers were useful to a couple of microservices so I used a private npm package.

Collapse
hallsamuel90 profile image
Sam Hall • Edited

Love the write up! One thing I'm struggling with is where to put more complex functionality in this structure.

Say for example I had some business need to pull data from an external source, transform it, and the push it to another source. In this scenario I would need something in the api and service layer to kick off the whole thing but from there I have trouble keeping things organized. I need all of the objects from data source A and B and another service to map from A to B. Would you create a module to house all of this logic or try to keep it flat in the structure you have shown here? Thanks!

Collapse
pandres95 profile image
Pablo Andrés Dorado Suárez

Several years ago, I started out a project. Initially to tackle some issues I saw in Sails, it turned out into —more or less, some issues like env variables to manage secrets are still on the go— this exactly.

If you want to check (also, I'm looking for some contributors and anyone willing to use it) the project, you can find it here

Collapse
shindesharad71 profile image
SHARAD SHINDE • Edited

Great post! This is what I am actually looking for!

Can you please review my project structure?

github.com/shindesharad71/Anstagra...

Collapse
santypk4 profile image
Sam Author

Seems very good, keep coding! :)

Be careful with the file uploads to express, it can be a real problem in production

You may want to implement a direct upload solution, here is an example with AWS S3 but I saw that you are using GCP so look for something similar.

file uploads

Collapse
shindesharad71 profile image
SHARAD SHINDE

Thank you for your valuable comment and guidance.

I already implemented the direct upload to GCP, no server involved.

Thanks!

Collapse
al_karlssen profile image
Al Karlssen

Node.js is an open-source, JavaScript run-time environment used to execute JavaScript code on the server-side. Node.js itechcraft.com/node-js/ development process has changed the paradigm that JavaScript is used primarily on the client-side. That’s why Node.js has become one of the foundational elements of the “JavaScript everywhere” paradigm.

Collapse
codingstatus profile image
codingStatus.com

This tutorial suggest me a basic & best way about MVC structure, Using this concept, I have developed MVC folder structure in Express with CRUD Example. So, Thanks for sharing..

Collapse
wickedknock profile image
wickedknock

I am new to nodejs , I have learnt express and mongodb crud , now I want to learn more , please can you only tell me the topics I have to learn myself to understand and use your repo .

Things I saw in repo I need to learn first is typescript , something called eslint and etc

Thanks <3

Collapse
melitus profile image
Aroh Sunday

Thanks, Sam, this is a wonderful and detailed explanation. I would appreciate seeing the full example of the pub/sub sample with eventemitter as demonstrated above. How can memory leaks be checked and avoided in a bigger nodejs app with EventEmitter? Once again, thanks

Collapse
noway profile image
Ilia • Edited

I wouldn't put stock into this. This is a collection of anti patterns. Quite over-dogmatic. None of these patterns make sense at a small scale, neither they do in a complex codebase where every one of those paradigms get in the way of clean code.

Start small. Organise code into functions. Practice TDD. Follow functional programming. Keep directory structure flat. Don't write custom Express middleware.

The rest will follow.

Collapse
santypk4 profile image
Sam Author

It’s seems like you are having a bad day.
I send you a virtual hug buddy :)

Collapse
clandau profile image
Courtney

Thanks for this! When I built my first app I kind of obsessed over file structure a bit too much and couldn't find any solid resources laying out the whys. I ended up looking at a lot of github projects to figure out mine. I wish I had this back then!

Collapse
quentin_mrt profile image
Mauret 👨🏼‍💻

Hi, nice work ! I just don't understand your unit test. I understood that you had mock UserModel with create method, but when I try to do the same in my test i have got an error :

Argument of type '{ find: () => IUser[]; }' is not assignable to parameter of type 'UserModel'.
Type '{ find: () => IUser[]; }' is missing the following properties from type 'UserModel': watch, translateAliases, bulkWrite, findById, and 58 more.

Any idea to fix this ? Maybe I have to do this differently

Collapse
pebcakerror profile image
Jonathan Worent

How might the architecture look if the data layer is a 3rd party API?

Collapse
santypk4 profile image
Sam Author

Should not be so different.

A wrapper class needs to be written for every API resource, with a compatible interface across all models.


class UserModel(){

   public UpdateOne(obj: { selector: any, update: any }):Promise<any> {
     // ...API Call
   }

}

Some methods, for example, UpdateMany may not be present in some third-party API resources and so a custom logic needs to be written.


class CompanyModel(){
   public UpdateOne(obj: { selector: any, update: any }):Promise<any> {
     // ...API Call
   }
   public UpdateMany(objs: Array<{ selector: any, update: any }>):Promise<Array<any>> {
     return Promise.all(
           updates.map(u => {
              return this.UpdateOne(u.selector, u.update);
          })
     )
   }

}

Collapse
somedood profile image
Basti Ortiz (Some Dood)

This was a really good read. Thanks for this! This is exactly what I needed to improve my "architecture skills". Keep it up!

Forem Open with the Forem app