Project structure for an Express REST API when there is no "standard way"

Originally published at coreycleary.me. This is a cross-post from my content blog. I publish new content every week or two, and you can sign up to my newsletter if you'd like to receive my articles directly to your inbox! I also regularly send cheatsheets, links to great tutorials by other developers, and other freebies!

Do any GitHub or Google search for REST API structures using Node + Express and you'll find very few of them follow the same organization.

What's even worse is, while there are lots of good tutorials out there, many of them have no structure at all. They just put everything into a 200-line server.js file and call it a day...

One of the best things about Node can also one of the most painful - there are few (if any) true conventions.

Sure, there are recommend ways of doing things. But it's such a flexible platform that you can often pick any way of doing something and it will likely work.

But even without conventions, developers want to know the best way to do things. And when it comes to REST API's (and Node projects in general...), everyone seems to feel like they're flying blind when it comes to structuring the project!

No "right way"

Ultimately, I don't believe there is one best project structure for Express projects.

Instead of asking:

"What’s the best way to structure my files and folders?"

I think it’s better to ask:

"What places do my different types of logic go?"

There are clearer answers to that question, and things we can follow.

And by doing a good job of separating our logic by layers, a project structure will naturally emerge. A structure that is flexible to how you choose to organize it, whether that is by more traditional MVC (or MVC-ish) or by the cool new kid, components. That's because these structures will be a layered approach anyways! You can simply group the routes, controllers, services, etc. into a component folder.

As long as logic is in the "right" place, the structure/organization becomes not that big a deal.

It's easier to refactor without having logic in weird places, it's easier to test without having logic in weird places, and once you've got your code checked into source control, it's easy to make changes anyways! Want to try out a components structure instead? Easy to make the changes!

"OK, I'm on board... but now what?"

A quick note on scope of this post: obviously all projects can include many different things. In order to make this post understandable and not overwhelm you, I'm going to leave out request/model validation and authentication. Both of those are animals on their own, but expect posts in the future addressing them.

Also, this is not intended to be a "boilerplate" project where you copy the repo, enter npm start in the console and instantly have a full-fledged application. Although, you will get a running app if you follow along. But a boilerplate wouldn't really achieve the goal of explaining layers in an app, where to put logic, and how to arrive at a structure based on that.

Since we're dealing with a REST API and an API receives a request and returns a response, let's follow that request from the point it hits our application, travels through our layers, and a response gets returned by the app. Along the way, we’ll figure out where our different types of logic should go.

Layers?

Let's take a look at an architecture diagram first:

Ever peeled an onion? After you peel off the first outer layer, there are a couple layers underneath that.

"Layers" in this context is much the same, meaning we've got:

a HTTP layer --> which is "outside" the service layer --> which is "outside" the database access layer --> which is... you get the picture

Where does the logic go?

We'll be using an example of a blog application to demonstrate logic separation and our resulting structure.

When I mentioned "types of logic" I was referring to the two "main" categories REST API logic falls into - HTTP logic and business logic. Of course, you can split "types of logic" down as much as you’d like, but these two are the main categories.

**The data access layer logic is often the more "technical" business logic, and I’ve grouped it in with the business logic as requirements often drive the queries you’ll need to write and the reports you’ll need to generate.

Routes

const express = require('express')

const { blogpost } = require('../controllers')

const router = express.Router()

router.post('/blogpost', blogpost.postBlogpost)

module.exports = router

As you can see from the code above, no logic should go in your routes/routers. They should only chain together your controller functions (in this case, we only have one). So routes are pretty simple. Import your controller(s) and chain together the functions.

I usually just have one controller per route, but there are exceptions, of course. If you have a controller that handles authentication, and have routes that need authentication, you would obviously need to import that as well and hook it up to your route.

Unless you have a ton of routes, I usually put them all in one index.js file. If you do have a ton of routes, you can put them into individual route files, import them all into one index.js file and export that.

If you want to understand how to avoid manually prepending '/api' to each individual route, check out this other post I wrote on that.

Controllers

const { blogService } = require('../services')

const { createBlogpost } = blogService

/*
 * call other imported services, or same service but different functions here if you need to
*/
const postBlogpost = async (req, res, next) => {
  const {user, content} = req.body
  try {
    await createBlogpost(user, content)
    // other service call (or same service, different function can go here)
    // i.e. - await generateBlogpostPreview()
    res.sendStatus(201)
    next()
  } catch(e) {
    console.log(e.message)
    res.sendStatus(500) && next(error)
  }
}

module.exports = {
  postBlogpost
}

I think of controllers as "orchestrators". They call the services, which contain more "pure" business logic. But by themselves,controllers don't really contain any logic other than handling the request and calling services. The services do most of the work, while the controllers orchestrate the service calls and decide what to do with the data returned.

And if it’s not obvious already, they take the HTTP request forwarded from the route and either return a response, or keep the chain of calls going. They handle the HTTP status codes as part of this response too.

Why Express/HTTP context should end here

Something I see fairly frequently is the Express req object (which is our HTTP "context") passed through beyond the routes and controllers to the services or even database access layer. But the problem with that is that, now the rest of the application depends on not only the request object, but on Express as well. If you were to swap out frameworks, it would be more work to find all the instances of the req object and remove them.

It also makes testing more difficult and this does not achieve a separation of concerns that we strive for in designing our applications.

Instead, if you use destructuring to pull out what pieces of data you need from req, you can simply pass those on to the services. The Express logic "ends" right there in the controllers.

If you need to make a call to an external API from one of your services, that is ok though, and we'll discuss that more when we cover what logic goes in services. But for now know that those calls are outside the HTTP context of your application.

And with that, we know where to put our "initial" logic that the REST API will handle (routes + controllers). On to the business logic layer...

Services

const { blogpostDb } = require('../db')

/*
  * if you need to make calls to additional tables, data stores (Redis, for example), 
  * or call an external endpoint as part of creating the blogpost, add them to this service
*/
const createBlogpost = async (user, content) => {
  try {
    return await blogpostDb(user, content)
  } catch(e) {
    throw new Error(e.message)
  }
}

module.exports = {
  createBlogpost
}

Services should contain the majority of your business logic: - logic that encapsulates your business requirements, calls your data access layer or models, calls API's external to the Node application. And in general, contains most of your algorithmic code.

You can certainly call external API's from within your controllers as well, but think about if that API is returning something that should be part of a "unit". Services should ultimately return a cohesive resource, and so if what that external API call returns is needed to augment your business logic, keep the logic there.

For example, if part of creating the blogpost was also posting the link to Twitter (an external API call), you would put it in the service above.

Why not call the models/data layer directly from the controllers if that's all this service is doing?

While our example above is simple in that all it does is access the database through our data access layer function - blogpostDb - as more business requirements get added, you add that Twitter API call, requirements change, etc. it will get complex fast.

If your controller handled all that logic, plus the request handling logic it already takes care of, it would start to become really hard to test, really fast. And remember, controllers can make multiple different service calls. So if you pulled all that logic out of other services and put it in the same controller, it would get even more unmanageable. You'd end up with the dreaded "fat controller" nightmare.

Data Access Layer/Models

const blogpostDb = (user, content) => {
  /*
   * put code to call database here
   * this can be either an ORM model or code to call the database through a driver or querybuilder
   * i.e.-
    INSERT INTO blogposts (user_name, blogpost_body)
    VALUES (user, content);
  */
  return 1 //just a dummy return as we aren't calling db right now
}

module.exports = {
  blogpostDb
}

In the code above, rather than set up a full database connection, I just pseudo-coded it but adding it is easy enough. When you have your logic isolated like this, it's easy to keep it limited to just data access code.

If it's not obvious, "Data Access Layer" means the layer that contains your logic for accessing persistent data. This can be something like a database, a Redis server, Elasticsearch, etc. So whenever you need to access such data, put that logic here.

"Models" is the same concept but used as part of an ORM.

Even though both are different they contain the same type of logic, which is why I recommend putting either kind in a db folder so that its general enough. Whether you’re using models from an ORM or you’re using a query builder or raw SQL, you can put the logic there without changing the name of directory.

Utils

The last type of logic we'll cover is that of common logic functions that are not necessarily specific to your business logic or domain, or even a REST API in general. A good example of a utility function would be a function that converts milliseconds to minutes and/or seconds, or one that checks two arrays to see if they contain similar items. These are general enough - and reusable enough - that they deserve to go in their own folder.

My preferred method is just putting these all into an index.js file and exporting each function. And I leave it at that as they don't really have a bearing on the rest of the project structure.

app.js / server.js

const express = require('express')
const bodyParser = require('body-parser')
const app = express()
const routes = require('./routes')

app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: true }));

app.get('/', (req, res) => res.send('App is working'))

app.use('/api', routes)

app.listen(3000, () => console.log('Example app listening on port 3000!'))

module.exports = {
  app
}

And just to tie it all together, I included an example entrypoint (usually named app.js or server.js) that goes in the root of your project structure. You can add middleware here (such as bodyParser) and import your routes file.

Now, the structure that emerges

When you separate your logic like I've described in this post, the structure seems to "naturally" fall into place like the above. This is why I like separating my logic in Express applications like this, it's easy to figure out where to put things!

And you can, of course, add more directories to the structure as you see fit (maybe a config folder, for example). But this is a great base to start from and 90% of your code will fall into one of these folders if you separate your logic as described.

Last but not least, tests!

Now that we've covered the structure that following this pattern will emerge out of, it's worth pointing out where tests go. I don't think this is as strict a rule, but I tend to keep my tests in one root tests folder and mimic the structure of the rest of the application.

In case you noticed, routes is missing! That's because if you separate out your logic like I've done, you don't really need to test the routes. You can use something like supertest if you want to, but the core logic - things that can break more easily with code changes! - will already be tested in your controllers, services, etc.

As an alternative, you could also add a tests folder within each "layer" folder, i.e. - a tests directory within the controllers, one within the services, etc. Just depends on preference, don't fret about it.

Also, some developers like to separate test directories by unit tests and by integration tests. My thoughts on that are that if you have a an application where there is a clear delineation and you have a lot of integration tests, it might be good to separate them. But more often than not I include them in the same directory.

Wrapping up

Like I stated at the beginning of this post, I don't believe there is a "best structure". It's much more helpful to make sure you've separated your logic into roles instead. THAT will give you the structure as a by-product, as well as giving you the flexibility you need to easily make changes later.

So if you're starting a new Express project and wasting time deciding which folders you should create, what you should name them, what should go in them - or if you're refactoring an existing Express app - use the approach I've described here to get you unstuck and get the ball rolling. And don't worry about it from there.

Remember, you can always change it later as long as your separation of logic is sound!

One more thing!

There's only so much I could cover here without it being overwhelming and you closing the window fast. I'm going to supplement this with additional structure/separation of logic articles coming soon.

If you want those additional articles emailed directly to you, here's that link again to subscribe to my newsletter! I send out new articles every week or two, in addition to cheatsheets, quick tips, and more.