ViitorCloud Technologies

Posted on Sep 6 • Originally published at viitorcloud.com

Building Scalable APIs with Node.js and Express

#webdev #javascript #beginners #programming

Node.js and Express have become essential tools in building scalable APIs, crucial for maintaining high performance and ensuring seamless user experiences. With its event-driven architecture, Node.js is highly efficient in managing the growing demands for real-time applications, making it a top choice for backend development. A study found that 68% of developers reported increased productivity with Node.js, largely due to its capability to use JavaScript on both the client and server sides, thus simplifying the development process. In this blog, we'll provide a step-by-step guide, complete with precise code examples, to help you build scalable APIs using Node.js and Express.

Key Principles for Building Scalable APIs

Asynchronous Programming

Node.js is inherently asynchronous, allowing it to handle multiple requests simultaneously without blocking the event loop. This feature is crucial for scalability, as it enables efficient resource management, particularly under high load conditions.

Modular Architecture

Adopting a modular approach is essential. By breaking down the application into smaller, self-contained modules, developers can manage complexity and facilitate independent scaling. Each module should focus on a specific functionality, which allows for easier testing, maintenance, and scaling of individual components as needed.

Microservices

Utilizing a microservices architecture can significantly enhance scalability. This approach allows different parts of the application to be scaled independently, optimizing resource use and improving overall performance. Each microservice can be developed, deployed, and scaled without affecting other services.

Effective Routing and Middleware

Express.js simplifies the creation of routes and the integration of middleware. Middleware functions can handle tasks such as authentication, logging, and error handling, making the API more robust and easier to manage. Properly organizing routes and controllers is vital for maintaining clarity and efficiency in the codebase.

Error Handling and Logging

Implementing comprehensive error handling and logging mechanisms is crucial for maintaining API stability. This ensures that issues can be quickly identified and resolved, minimizing downtime and enhancing user experience.

API Documentation

Using tools like Swagger for API documentation is recommended. Well-documented APIs not only facilitate easier onboarding for new developers but also ensure that the API is user-friendly for consumers, enabling better integration and usage.

Performance Optimization

Optimizing code for performance is critical. Techniques include minimizing synchronous operations, managing resource consumption, and leveraging caching strategies (e.g., using Redis) to reduce database load and improve response times.

Load Testing

Regular load testing is essential to identify potential bottlenecks and ensure the API can handle anticipated traffic spikes. Monitoring key metrics such as response times and error rates can guide necessary adjustments and scaling strategies.

How to Build Scalable API with Node.js and Express

1. Setup Your Project

We created folder scalable-api. Open this folder inside your code editor.

then follow below instruction.

To Initialize a new Node.js project run below command.

npm init -y

Once you run above command, your package.json file will be created inside the folder.

now, you have to run below command to install express.

npm install express

once you run above command, node_modules folder & package-lock.json file will be created in root folder.

2. Creating the Server

Now that you already installed Express, your next step is to create your server file and set up a basic express server.

You have to create a file name app.js in the root folder.

Now, you created your file app.js, set up basic Express server in app.js.

const express = require('express');
const app = express();
const port = process.env.PORT || 3000;

// Middleware to parse JSON bodies
app.use(express.json());

// Basic route
app.get('/', (req, res) => {
    res.send('Hello World!');
});

// Start the server
app.listen(port, () => {
    console.log(`Server is running on port ${port}`);
});

Now, once you done, run your server by executing following command in your terminal.

node app.js

Once you run above command, you’ll see below message in your terminal

Starting the server...
Middleware set up.
Route set up.
Server is running on port 3000

and to access the server, open your chrome browser or whichever you’re using, Navigate to http://localhost:3000 , You’ll see the “Hello World” message.

as shown in above screen.

The next steps for building scalable APIs with Node.js and Express typically involve:

3. Defining & Adding More Routes:

Adding More Routes:

You have to define additional routes to handle different endpoints. For example, let’s add routes for /api and /users.

// ... existing code ...

// New route for /api
app.get('/api', (req, res) => {
    res.send('API endpoint');
});

// New route for /users
app.get('/users', (req, res) => {
    res.send('Users endpoint');
});

// ... existing code ...

You to add above code in your app.js file and update your app.js file.

app.get('/api', (req, res) => { ... }): This defines a new route that listens for GET requests on the /api endpoint. When a request is received, it sends back the response "API endpoint".
app.get('/users', (req, res) => { ... }): This defines another route that listens for GET requests on the /users endpoint. When a request is received, it sends back the response "Users endpoint".

Now, let’s test the new routes.

start your server by running:

node app.js

Now, open your browser, and go to http://localhost:3000/api , you’ll see the message “API endpoint”

Now, you should navigate to **http://localhost:3000/users,** you’ll see the message “Users endpoint”, shown in below screen

Now, your next step will be to connect to a database. For this, we’ll use we'll use MongoDB with Mongoose, a popular ODM (Object Data Modeling) library for MongoDB and Node.js.

4. Connecting to a Database.

Install a mongoose.

You’ve to run the following command to install the Mongoose.

npm install mongoose

now, you to update your app.js to Connect to MongoDB:

Let’s see how you can update your app.js file to connect to MongoDB database.

   const mongoose = require('mongoose');

   // Connect to MongoDB
   mongoose.connect('mongodb://localhost:27017/mydatabase', { useNewUrlParser: true, useUnifiedTopology: true })
       .then(() => console.log('Connected to MongoDB'))
       .catch(err => console.error('Could not connect to MongoDB', err));

   // ... existing code ...

Connect to MongoDB: Use mongoose.connect to connect to your MongoDB database. Replace 'mongodb://localhost:27017/mydatabase' with your actual MongoDB connection string if it's different.

Now, that for that, You must first have set up your MongoDB, If you haven’t, let’s set it up now. Use MongoDB Atlas (Cloud Database). You have to create a MongoDB Atlas account, Go to MongoDB Atlas and sign up for free account.

once you sign up via or Google or GitHub Account, You’ll see below screen after login.

You’ll to click on visit MongoDB Atlas, Click on Create a New Cluster,

Once you click on Create as shown in above screen, you’ll be redirect to below screen, select M0 free version.

you have to keep your configuration as it is by default, now click on Create Deployment.

once, you click on that, you’ll come to this screen.

You’ve to notedown, your username and password, once you click on “Create Database User”,

Once you click on “Choose a connection method”,

now, choose, Choose “Connect to your application”, You’ll see below screen.

Now, you’ll see connection string like this - mongodb+srv://<username>:<password>@cluster0.mongodb.net/myFirstDatabase?retryWrites=true&w=majority

replace it with your username and password you noted down earlier.

Now that, you to copy your MongoDB string and have to replace in your app.js code, and you have to update it.

now, you to run your node app.js file again in your terminal. You’ll see the message “Connected to MongoDB”

5. Implementing Middleware

So, now you see above successful message, now, next step is to add simple logging middleware using Morgan which is a popular HTTP request logger middleware for Node.js.

To install Morgan, run below command.

npm install morgan

now, you’ve to update, your app.js file to use Morgan.

   const express = require('express');
   const morgan = require('morgan'); // Import morgan
   const app = express();
   const port = process.env.PORT || 3000;

   console.log('Starting the server...');

   const mongoose = require('mongoose');

   // Replace with your actual MongoDB connection string
   const mongoURI = 'mongodb+srv://<username>:<password>@cluster0.mongodb.net/myFirstDatabase?retryWrites=true&w=majority';

   mongoose.connect(mongoURI, { useNewUrlParser: true, useUnifiedTopology: true })
       .then(() => console.log('Connected to MongoDB'))
       .catch(err => console.error('Could not connect to MongoDB', err));

   // Middleware to parse JSON bodies
   app.use(express.json());

   // Use morgan for logging
   app.use(morgan('tiny'));

   console.log('Middleware set up.');

   // Basic route
   app.get('/', (req, res) => {
       console.log('Received a request on /');
       res.send('Hello World!');
   });

   // New route for /api
   app.get('/api', (req, res) => {
       res.send('API endpoint');
   });

   // New route for /users
   app.get('/users', (req, res) => {
       res.send('Users endpoint');
   });

   console.log('Route set up.');

   // Start the server
   app.listen(port, () => {
       console.log(`Server is running on port ${port}`);
   });

now, run your node app.js, go to http://localhost:3000/ and you’ll see log entry for request in terminal. This step will add basic logging to your application.

Your next step is to define a schema and model for your MongoDB collections using Mongoose. This will allow you to interact with your database in a structured way.

6. Define a Schema and Model

You have to create a new file named user.js in a models directory you may need to create the models directory if it doesn't exist). You’ve define a User Schema and model for user.

   const mongoose = require('mongoose');

   // Define the User schema
   const userSchema = new mongoose.Schema({
       name: {
           type: String,
           required: true
       },
       email: {
           type: String,
           required: true,
           unique: true
       },
       password: {
           type: String,
           required: true
       }
   });

   // Create the User model
   const User = mongoose.model('User', userSchema);

   module.exports = User;

Now, you’ve to update the app.js to Use the User Model:

In app.js, import the User model and create a route to add a new user:

   const express = require('express');
   const morgan = require('morgan');
   const mongoose = require('mongoose');
   const User = require('./models/user'); // Import the User model
   const app = express();
   const port = process.env.PORT || 3000;

   console.log('Starting the server...');

   // Replace with your actual MongoDB connection string
   const mongoURI = 'mongodb+srv://<username>:<password>@cluster0.mongodb.net/myFirstDatabase?retryWrites=true&w=majority';

   mongoose.connect(mongoURI, { useNewUrlParser: true, useUnifiedTopology: true })
       .then(() => console.log('Connected to MongoDB'))
       .catch(err => console.error('Could not connect to MongoDB', err));

   // Middleware to parse JSON bodies
   app.use(express.json());

   // Use morgan for logging
   app.use(morgan('tiny'));

   console.log('Middleware set up.');

   // Basic route
   app.get('/', (req, res) => {
       console.log('Received a request on /');
       res.send('Hello World!');
   });

   // New route for /api
   app.get('/api', (req, res) => {
       res.send('API endpoint');
   });

   // New route for /users
   app.get('/users', (req, res) => {
       res.send('Users endpoint');
   });

   // Route to add a new user
   app.post('/users', async (req, res) => {
       try {
           const user = new User(req.body);
           await user.save();
           res.status(201).send(user);
       } catch (error) {
           res.status(400).send(error);
       }
   });

   console.log('Route set up.');

   // Start the server
   app.listen(port, () => {
       console.log(`Server is running on port ${port}`);
   });

now, that you’ve to open postman, you can also use desktop postman agent, click on new request, select request type to “Post”, Enter the URL - http://localhost:3000/users , now, select body tab, select row and json there.

enter the following json in text area.

     {
         "name": "John Doe",
         "email": "john.doe@example.com",
         "password": "password123"
     }

and once you send the request, it will reflect in your MongoDB Atlas account, you have to go database and have to select cluster0, find your database which you’ve create, go to the user, and here you’ll found the information, you send via a request. just like below screen.

7. Implemented CRUD Operations:

As we successfully added users to your MongoDB database, the next step is to implement additional CRUD (Create, Read, Update, Delete) operations for your User model. This will allow you to manage users more effectively.

Let's start by adding routes for reading, updating, and deleting users.

1. Read (GET) Users

Add a route to get all users and a route to get a user by ID.

// ... existing code ...

// Route to get all users
app.get('/users', async (req, res) => {
    try {
        const users = await User.find();
        res.send(users);
    } catch (error) {
        res.status(500).send(error);
    }
});

// Route to get a user by ID
app.get('/users/:id', async (req, res) => {
    try {
        const user = await User.findById(req.params.id);
        if (!user) {
            return res.status(404).send('User not found');
        }
        res.send(user);
    } catch (error) {
        res.status(500).send(error);
    }
});

// ... existing code ...

2. Update (PUT) Users

Add a route to update a user by ID.

// ... existing code ...

// Route to update a user by ID
app.put('/users/:id', async (req, res) => {
    try {
        const user = await User.findByIdAndUpdate(req.params.id, req.body, { new: true, runValidators: true });
        if (!user) {
            return res.status(404).send('User not found');
        }
        res.send(user);
    } catch (error) {
        res.status(400).send(error);
    }
});

// ... existing code ...

3. Delete (DELETE) Users

Add a route to delete a user by ID.

// ... existing code ...

// Route to delete a user by ID
app.delete('/users/:id', async (req, res) => {
    try {
        const user = await User.findByIdAndDelete(req.params.id);
        if (!user) {
            return res.status(404).send('User not found');
        }
        res.send(user);
    } catch (error) {
        res.status(500).send(error);
    }
});

// ... existing code ...

once, you update your above code in your app.js file,

start your server using node app.js command,

now, send request to get all users: You’ll see below screen, in your http://localhost:3000/users

when you run get user by id, you’ll see in terminal as well on local host as well.

when you update any information, you’ll able to see it here. we changed name from john doe to Ethan lee,

when you run a delete request, one user will be deleted.

so, we successfully implemented and tested all the basic CRUD Operations for your API.

8. Implement Error Handling Middleware

Centralized error handling helps manage errors gracefully and provides consistent error responses.

Add Error Handling Middleware

   // ... existing code ...

   // Error handling middleware
   app.use((err, req, res, next) => {
       console.error(err.stack);
       res.status(500).send({ error: 'Something went wrong!' });
   });

   // Start the server
   app.listen(port, () => {
       console.log(`Server is running on port ${port}`);
   });

9. Use Environment Variables

Using environment variables helps manage configuration settings securely.

Install dotenv

npm install dotenv

2. Create a .env File:

PORT=3000
MONGODB_URI=mongodb+srv://<username>:<password>@cluster0.mongodb.net/myFirstDatabase?retryWrites=true&w=majority

this is your updated app.js

   require('dotenv').config();
   const express = require('express');
   const morgan = require('morgan');
   const mongoose = require('mongoose');
   const cors = require('cors');
   const User = require('./models/user');
   const app = express();
   const port = process.env.PORT || 3000;

   console.log('Starting the server...');

   mongoose.connect(process.env.MONGODB_URI, { useNewUrlParser: true, useUnifiedTopology: true })
       .then(() => console.log('Connected to MongoDB'))
       .catch(err => console.error('Could not connect to MongoDB', err));

   // Middleware to parse JSON bodies
   app.use(express.json());

   // Use morgan for logging
   app.use(morgan('tiny'));

   // Use cors for handling CORS issues
   app.use(cors());

   console.log('Middleware set up.');

   // Basic route
   app.get('/', (req, res) => {
       console.log('Received a request on /');
       res.send('Hello World!');
   });

   // CRUD routes
   // ... existing CRUD routes ...

   // Error handling middleware
   app.use((err, req, res, next) => {
       console.error(err.stack);
       res.status(500).send({ error: 'Something went wrong!' });
   });

   console.log('Route set up.');

   // Start the server
   app.listen(port, () => {
       console.log(`Server is running on port ${port}`);
   });

10. Add Basic Authentication

For a more secure API, you might want to add basic authentication. Here’s a simple example using HTTP Basic Auth:

Install Basic Auth Middleware:

npm install express-basic-auth

2. Update app.js to Use Basic Auth:

   const basicAuth = require('express-basic-auth');

   // ... existing code ...

   // Use basic auth for all routes
   app.use(basicAuth({
       users: { 'admin': 'supersecret' },
       challenge: true
   }));

   // ... existing code ...

11. Document Your API

Using tools like Swagger can help you document your API endpoints.

Install Swagger UI

npm install swagger-ui-express swagger-jsdoc

Create a Swagger Configuration File

create a swagger.js file in root folder, you’ve to add following code to your file.

const swaggerJsDoc = require('swagger-jsdoc');
const swaggerUi = require('swagger-ui-express');

const swaggerOptions = {
    swaggerDefinition: {
        openapi: '3.0.0',
        info: {
            title: 'User API',
            version: '1.0.0',
            description: 'A simple Express User API'
        },
        servers: [
            {
                url: 'http://localhost:3000'
            }
        ]
    },
    apis: ['./app.js'] // Path to the API docs
};

const swaggerDocs = swaggerJsDoc(swaggerOptions);

module.exports = (app) => {
    app.use('/api-docs', swaggerUi.serve, swaggerUi.setup(swaggerDocs));
};

Update app.js to Use Swagger

Add the following code to your app.js file to set up Swagger:

update your app.js file

require('dotenv').config();
const express = require('express');
const morgan = require('morgan');
const mongoose = require('mongoose');
const cors = require('cors');
const User = require('./models/user');
const setupSwagger = require('./swagger'); // Import the Swagger setup function
const app = express();
const port = process.env.PORT || 3000;

console.log('Starting the server...');

mongoose.connect(process.env.MONGODB_URI, { useNewUrlParser: true, useUnifiedTopology: true })
    .then(() => console.log('Connected to MongoDB'))
    .catch(err => console.error('Could not connect to MongoDB', err));

// Middleware to parse JSON bodies
app.use(express.json());

// Use morgan for logging
app.use(morgan('tiny'));

// Use cors for handling CORS issues
app.use(cors());

console.log('Middleware set up.');

// Basic route
app.get('/', (req, res) => {
    console.log('Received a request on /');
    res.send('Hello World!');
});

// CRUD routes
// ... existing CRUD routes ...

// Setup Swagger
setupSwagger(app);

// Error handling middleware
app.use((err, req, res, next) => {
    console.error(err.stack);
    res.status(500).send({ error: 'Something went wrong!' });
});

console.log('Route set up.');

// Start the server
app.listen(port, () => {
    console.log(`Server is running on port ${port}`);
});

Add Swagger Comments to Your Routes

Add comments to your routes in app.js to document them with Swagger. Here’s an example for the GET /users route:

/**
 * @swagger
 * /users:
 *   get:
 *     summary: Retrieve a list of users
 *     responses:
 *       200:
 *         description: A list of users
 *         content:
 *           application/json:
 *             schema:
 *               type: array
 *               items:
 *                 type: object
 */
app.get('/users', async (req, res) => {
    try {
        const users = await User.find();
        res.send(users);
    } catch (error) {
        res.status(500).send(error);
    }
});

/**
 * @swagger
 * /users/{id}:
 *   get:
 *     summary: Retrieve a single user by ID
 *     parameters:
 *       - in: path
 *         name: id
 *         required: true
 *         schema:
 *           type: string
 *         description: The user ID
 *     responses:
 *       200:
 *         description: A single user
 *         content:
 *           application/json:
 *             schema:
 *               type: object
 *       404:
 *         description: User not found
 */
app.get('/users/:id', async (req, res) => {
    try {
        const user = await User.findById(req.params.id);
        if (!user) {
            return res.status(404).send('User not found');
        }
        res.send(user);
    } catch (error) {
        res.status(500).send(error);
    }
});

// Add similar comments for other routes...

now, when you go to http://localhost:3000/api-docs , use your username and password which is in app.js

You’ll get this, you should be able to access the Swagger UI and see the documentation for your API.

You have successfully built a scalable API with Node.js and Express, complete with CRUD operations, basic authentication, and API documentation using Swagger. This should provide a comprehensive demo for your technical article. You can deploy your API over cloud which make it accessible for all people on the internet, you can use services like Heroku for deployment.

Conclusion

Building scalable APIs with Node.js and Express requires a strategic approach, including modular architecture, optimized performance with non-blocking I/O, clustering, efficient database management, and robust security measures. By implementing caching, monitoring tools, and auto-scaling, your API can handle thousands of requests per second, ensuring reliability and performance under heavy loads. Ready to scale your API with Node.js and Express? Contact us today to build a future-proof API solution for your business needs!

FAQs

1. What are the main advantages of using Node.js for building APIs?

Node.js offers several advantages for API development, including its asynchronous, non-blocking architecture that allows for handling multiple requests simultaneously. This leads to improved performance and scalability. Additionally, Node.js uses JavaScript, enabling developers to work across both the client and server sides, which streamlines the development process.

2. Why should I use Express.js with Node.js?

Express.js is a minimal and flexible web application framework that simplifies the process of building APIs with Node.js. It provides robust routing, middleware support, and easy integration with various databases, making it an excellent choice for developing RESTful APIs quickly and efficiently.

3. How can I ensure my API is scalable?

To ensure your API is scalable, consider implementing a microservices architecture, which allows different components to be scaled independently. Additionally, optimize your code for performance, use caching strategies, and conduct regular load testing to identify and address potential bottlenecks.

4. What are some best practices for error handling in Node.js APIs?

Best practices for error handling in Node.js APIs include using middleware to catch errors globally, logging errors for monitoring and debugging, and providing meaningful error messages to clients. It's also essential to handle different types of errors (e.g., validation errors, database errors) appropriately to enhance user experience.

5. How can ViitorCloud help you build scalable APIs with Node.js and Express?

ViitorCloud offers comprehensive development services tailored to your business needs, specializing in building scalable APIs using Node.js and Express. Our team of experienced developers utilizes best practices in API design and architecture, ensuring that your application can handle high loads efficiently. With a focus on performance optimization and a robust development process, ViitorCloud can help you create APIs that not only meet current demands but also adapt to future growth, providing a strong foundation for your digital solutions.

DEV Community