Originally posted on softwareontheroad.com
Update 04/21/2019: Implementation example in a GitHub repository
Introduction
Express.js is...
For further actions, you may consider blocking this person and/or reporting abuse
with express if you use async handler always wrap the code with try/catch otherwise in case if something happens express will never respond
Or use wrapper around callback.
I'd suggest you go even further, like I did here - github.com/oors/oors/blob/master/p...
You can see here how to use it to generate a CRUD experience using async / await - github.com/oors/oors/blob/master/p...
This code is probably for the specific use case, but looks not good.
If the response is undefined, there is no further handle, just a hang.
Not really.
You can render the response however you like (using res.json or res.render etc) or you can just return something anything that will be JSON encoded and rendered as is.
Example:
which is the same thing as:
But in other cases you might want to do something like this:
All of these work as expected.
You're right!
And this will make an error by negligence.
Whoever writes code like that is negligent :)
It's like saying that express.js is too permissive for letting you write code like this:
I think everyone knows that it's a bad practice to do other things (including returning something) after sending the response in express.js (there are very special cases though, but they don't include the one you shared).
Yes, my idea to hide real response in a route is not a good idea.
Cause you have mixed flow.
In one case, you can put
return
in otherres.SOMETHING
, but not both.It makes harder to maintain and avoid potential fails in a development team.
Yep, I see your point and you're right in a way.
I only use express.js to build APIs (I never render a template or something). So it's more like a convention (shortcut) as returing JSONs is what I do in 90% of the cases.
Yep, this is why I started from this message: "This code is probably for the specific use case, but looks not good."
For example, if you
return await ...
withundefined
you will go in hang situation.Or you can use express-async-error.
and that's it, any error happens the express-async-error will catch it and passes it to the default express error handler.
Prefer feature based file structure. Why group all your models and all your services together. They donβt really have anything in common. They just happen to be of the same type. Better to group posts model and posts service together.
This doesn't make much sense. A model will often apply to multiple request methods and/or routes.
I've been using this generic structure, which I find works pretty well in practice in several languages and domains:
That's a good architecture too! :)
We might be talking about different 'models' here. E.g. the types that are tightly coupled to a particular controller (request) or a service should probably live in the same file or folder. But types that are not obviously coupled could go in a shared 'models' folder, but then I would argue, why are they not strongly coupled to any controller or service? Another point here is that it's OK for a controller to import the types of a service that it uses, that is just normal layered architecture for a higher level to be aware of the lower level.
So import it where it's needed. Same way as you do when putting models in its folder etc.
i'm not sure about the file structure. i think that when you look at a good architecture, it should be obvious what's the purpose of the application, but if you look at folders like
api
,services
,models
, that'll tell you nothing. i usually organize files by their purpose, eg.user
,product
,order
, etc.I agree with you components should be self contained.
Great architecture and file structure! I also like your bit about the loaders. Very clean way of doing it.
Do you have a skeleton setup on githib or something? I'd love to play around with this.
I'm glad you like it :)
Here is the github repository github.com/santiq/bulletproof-nodejs
Thanks! :)
yep, waiting for the skeleton too
This architecture is completely different from this one
github.com/i0natan/nodebestpractices
why would you say yours is better?
Are you sure? I don't see any conflicts. I would say that the "nodebestpractices" focus on different things than Santiago. Actually I only find these 2 points comparable to the advice given here:
1.1 Structure your solution by components
1.2 Layer your components, keep Express within its boundaries
This is very general advice. Santiago follows this advice, but goes way further and fills it with practical instructions.
But his structure is not by components - but by roles. He has a
models
folder that holds all components models andservices
with all components services, instead of having acomponentName
folder with the respectable model and service.Oh yes, the good component base architecture, that's a good way to do it too!! :)
better is not the right question, because every architecture has a purpose... what fit yours?
Good points.
Another one would be to adopt and follow a plugin / module - based architecture. This way you can split responsibilities and reuse those modules with other projects as well.
A frameworks that pays good respect to these principles and other similar ones is github.com/oors/oors - a framework I created.
It's modules-based, integrates with express.js, promotes a layered architecture, DI is baked in, has great support for MongoDB and GraphQL, plus much more than that.
Feel free to check out the already existing plugins and ask me any questions about it.
Just some advice β your project doesn't have any kind of helpful readme or docs. I have no idea how to use it, so I won't. Please consider putting effort into that instead of advertising it on articles :(
It actually does - github.com/oors/oors/tree/master/docs - but it's a bit outdated and incomplete. Nonetheless, it highlights the general idea behind the framework.
But you're completely right and documentation is in the making. It hasn't been a priority so far because I was the one to instruct the people who have been using it so far.
The framework's been out for a while now and it's been used in production on some great products.
That being said, I do plan to improve the framework and write more quality extensions, so if you're into node.js, GraphQL, MongoDB... you might want to stick around :)
Thanks for this enlightening post.
It reminds me of NestJS, which is a cool Typescript framework.
This is great! I have worked on many nodejs projects and each time the project structure has been a little better than the previous time. Loaders is a nice touch - I hadn't thought of that. Also shout out to Agenda.js - makes job management super simple and it's very reliable!
Hey Santiago, already followed you on Twitter, and as I said there, I really like the points you made.
It'd be awesome if you have a GitHub repo with a basic project set up with this structure.
Btw, there's an error on your link here:
It's pointing to:
dev.to/nodejs-scalability-issues
I guess you want it to point to your blog:
softwareontheroad.com/nodejs-scala...
Thanks, I fixed that broken link :)
Here is the example repository github.com/santiq/bulletproof-nodejs
Have fun!
Well done for this article! Loved it. I would only advise you to remove the middle services layer as you are using Mongoose and Mongoose is able to provide with fully featured models that can do pretty much everything like validation, custom functions, hooks, dependencies on other models, etc. etc.. The middle Services layer will only create an unneeded abstraction that you gonna hate down the road, I certainly hated mine when I realised how much power Mongoose had and that I was trying to write stuff that were already existing.
Put your model's business logic inside Mongose models, handle dependencies on other models inside those models and then let the controllers handle the glue between these models and other pieces of your architecture like sending emails, etc.
Also, make sure to explore this repository which provides a great starting point for a Node.js project structure and covers most of what you covered already. Read the reasoning behind it here.
You have a typo:
export default class UserService() {
should be
export default class UserService {
Very useful article!
Exactly what I was looking for.
You could also link some of your project with those patterns, I think it would help wrap reader's head around it.
Thanks for noticing that!
There is a link to an example project with the patterns implemented, you can check it out here github.com/santiq/bulletproof-nodejs
And then you go microservices + lambda and everything becomes different (and more decentralized). Hah :) Nice writeup!
Now we need an npm packages that configures and manages your project structure for you. Feel free to take a look at my
robo-config
. I'll get to managing the whole project structure dynamically eventually as well :PAwesome post, mate, very nice and organized. I'm curious about your error handling, though. How do you do it, since you have no try catches in services, nor in route handlers ?
Thank you!
I didn't write too much on good practices about error handling because I wanted to keep the examples simple and concise.
But here you have the complete repository, with proper error handling and more details.
github.com/santiq/bulletproof-nodejs
Cool, i checked out your boilerplate, I'd maybe extract error handling logic from the middleware to separate module, maybe add logging. Also, have you thought about having StatusError extending the regular one ? So that you can have comprehensive errors with statuses even from within services, instead of response 500. I'd also add that is of high importance if one should extend regular error to attach stack trace from the super class.
All great points!! I was thinking about logging after reading this good guide. Centralized logging would be great. Common context such as user, request-id, timing, etc. can be added to all log output. Lots of log shipping programs like to parse structured logs, and formatting in JSON makes it super easy. Also
console.log()
is not performant for production@Strahinja - love of the idea of a
StatusError
. We do something similar in our application. Controllers canthrow new Error()
which our Express error handler logs as a server error and responds with an HTTP 500 to the client. Controllers can also use a custom error object andthrow new ResponseError(err, 400)
, which logs a warning and returns an HTTP 400 to the client.@blake That sounds really nice. I made a discussion regarding this. You could maybe respond there and maybe put a bit of code so we can put exchanging of ideas in motion.
Awesome article! I like the structure of this project and IoC. I came from Ruby on Rails so I've been using a similar MVC (or just MC, depends if serving html) structure borrowed from RoR in my Node projects. I'm going to implement these patterns in my new project right now.
I'm curious on where you would put your helpers/utils files though? Do you have a separate directory for them? Or do you stick them under the services directory that uses them?
Hi!
I use these concepts as a base for my projects, I stick to the 3 layer pattern and sometimes I create a helper's folders and put there functions not related to any service, but other times those helpers were useful to a couple of microservices so I used a private npm package.
Hello and thank you for your post,
I would like to suggest that you add an emphasis on the repositories,
Your post talk about the domain isolation from the controller but I think it would be great if you dive into the directory structure of the domain logic. I am referring to DDD philosophy
This sounds like a lot to setup. Have you tried Adonis.js? It has a similar structure to Laravel / RoR, no custom setup needed.
Great write up.
For env variables that are injected in other ways (via CI or Lambda) you could move the dotenv line to run when you start your node app:
"node -r dotenv/config --inspect app"
That will avoid failing to load the .env file when it's only present in one of your environments (local).
I went through your code and learned a lot, I didn't finish it yet.
Articles about structure are very enjoyable for me. I need to learn much more about software structure and design patterns.
Thank you.
I'm really happy that it helped you !! :)
Love the write up! One thing I'm struggling with is where to put more complex functionality in this structure.
Say for example I had some business need to pull data from an external source, transform it, and the push it to another source. In this scenario I would need something in the api and service layer to kick off the whole thing but from there I have trouble keeping things organized. I need all of the objects from data source A and B and another service to map from A to B. Would you create a module to house all of this logic or try to keep it flat in the structure you have shown here? Thanks!
Several years ago, I started out a project. Initially to tackle some issues I saw in Sails, it turned out into βmore or less, some issues like env variables to manage secrets are still on the goβ this exactly.
If you want to check (also, I'm looking for some contributors and anyone willing to use it) the project, you can find it here
Thanks for this! When I built my first app I kind of obsessed over file structure a bit too much and couldn't find any solid resources laying out the whys. I ended up looking at a lot of github projects to figure out mine. I wish I had this back then!
I am new to nodejs , I have learnt express and mongodb crud , now I want to learn more , please can you only tell me the topics I have to learn myself to understand and use your repo .
Things I saw in repo I need to learn first is typescript , something called eslint and etc
Thanks <3
Great post! This is what I am actually looking for!
Can you please review my project structure?
github.com/shindesharad71/Anstagra...
Seems very good, keep coding! :)
Be careful with the file uploads to express, it can be a real problem in production
You may want to implement a direct upload solution, here is an example with AWS S3 but I saw that you are using GCP so look for something similar.
Thank you for your valuable comment and guidance.
I already implemented the direct upload to GCP, no server involved.
Thanks!
I wouldn't put stock into this. This is a collection of anti patterns. Quite over-dogmatic. None of these patterns make sense at a small scale, neither they do in a complex codebase where every one of those paradigms get in the way of clean code.
Start small. Organise code into functions. Practice TDD. Follow functional programming. Keep directory structure flat. Don't write custom Express middleware.
The rest will follow.
Itβs seems like you are having a bad day.
I send you a virtual hug buddy :)
Node.js is an open-source, JavaScript run-time environment used to execute JavaScript code on the server-side. Node.js itechcraft.com/node-js/ development process has changed the paradigm that JavaScript is used primarily on the client-side. Thatβs why Node.js has become one of the foundational elements of the βJavaScript everywhereβ paradigm.
Hi, nice work ! I just don't understand your unit test. I understood that you had mock UserModel with create method, but when I try to do the same in my test i have got an error :
Argument of type '{ find: () => IUser[]; }' is not assignable to parameter of type 'UserModel'.
Type '{ find: () => IUser[]; }' is missing the following properties from type 'UserModel': watch, translateAliases, bulkWrite, findById, and 58 more.
Any idea to fix this ? Maybe I have to do this differently
Thanks, Sam, this is a wonderful and detailed explanation. I would appreciate seeing the full example of the pub/sub sample with eventemitter as demonstrated above. How can memory leaks be checked and avoided in a bigger nodejs app with EventEmitter? Once again, thanks
This tutorial suggest me a basic & best way about MVC structure, Using this concept, I have developed MVC folder structure in Express with CRUD Example. So, Thanks for sharing..
really nice breakdown! Thanks!
Awesome project structure. I especially like your addition of the pub/sub layer.
How might the architecture look if the data layer is a 3rd party API?
Should not be so different.
A wrapper class needs to be written for every API resource, with a compatible interface across all models.
Some methods, for example,
UpdateMany
may not be present in some third-party API resources and so a custom logic needs to be written.Great post! Thanks for sharing!
Great article and lots of valuable points ! learn so much from your architecture while Im working on my on nodejs project.
Just want to know your opinion regarding the future of nodejs. It seems that the market of nodejs is not so popular as java or python.
Question. What if I wanted to add react. where should I put it? should there be only one node modules for frontend and backend? and what if I wanted to dockerize each? thank you.
Hello, awesome article. Would it be possible to have a list of all patterns used in the project?
Sure.
The project follows the 3 tier (or layers) pattern for structuring the code and separate the controllers, services and data access layer.
Also, there are some events involved, following the Pub/Sub pattern and inversion of control
And of course, as other developers pointed, you can go nuts and implement something like the 'Clean Domain Driven Design' architecture but is a little overwhelming for little applications.
The list of patterns used here
Thanks! This is really helpful
Awesome post! In the
app.js
listing you haveyet,
loaders/index.js
doesn't define in init function or did I miss something?Though I am a frontend dev, this gave me good insights of structuring code architecture and seperating concerns. Great write up!
Loved it! Simple, clean, logical, and extendable!
Awesome architecture! It looks really scalable.
This was a really good read. Thanks for this! This is exactly what I needed to improve my "architecture skills". Keep it up!
Nice article and architecture, the loaders part is really good. Thank you!
Why use typedi instead of plain old higher-order functions? Functions are first-class citizens in JS/TS, so why not use HOCs instead of adding an additional dependency that applies a pattern that only makes sense in languages without HOCs?
Hi, thank's for this architecture, i found it perfect for scale API with Node.js ! Adding Typescript & loaders are great ideas
Why do we need two different api and services folder? The subscribers or pub/sub also falls under services.
Great article.
Maybe you want to add on to this :
An imperative call to a dependent service is not the best way of doing it.
Can you explain why?
Thanks for the great article π€©, I knew most of them but great to put all these in one place, but what about error handling it would be great if u discuss that too, thanks again π
Great post, great explanation.
But I can't figure out how to handle errors for different scenarios when I should not return any error message or http status from service layer.
Really nice article. I am more of a front end developer now but coming from a heavy .NET background and this made a lot of sense. Great job!
Inspired by this, but didn't use typescript, DI
node-js-starter
I love this thank you!
Hey, but how should i load env using the config? should import on each module where it require?
Well explained with simple examplesβ₯οΈ
How to create Database Access Layer/Object
In service based architecture?