Introduction
Rate limiting is a method used for controlling network traffic. It limits the number of actions a user can make per unit of time 1. In this tutorial, we will rate limit a login route to help protect it from brute force attacks. This limits the number of password guesses that can be made by an attacker. We'll use the npm package node-rate-limiter-flexible to count and limit the number of login attempts by key. Each key will have a points value that will count the number of failed login attempts. The keys will expire after a set amount of time. The key-value pairs will be stored in Redis, which is an open-source in-memory data structure store. It has many different use cases. We will use it as a simple database. Redis is simple to use and very fast. We'll create an online instance of Redis, connect it to an express application, and then use the Redis command-line interface (redis-cli) to view the database. A prerequisite for this tutorial is an ExpressJS application with a login route and user authentication.
We will use two types of keys to count the number of failed logins. One will be a string made using the user's IP address. The other will be a string made by joining the user's email address and IP address. When a user attempts to log in, if the user exists and the password is not correct, the two keys will be created for the user.
For example, the keys stored in Redis may look like this after a failed login attempt where the password was incorrect:
key 1: "login_fail_ip-192.168.1.1" : 1
key 2: "login_fail_username_and_ip-example@example.com_192.168.1.1" : 1
Prerequisites
Express app with login route and login authentication (login with username or email)
Registered users stored in a database
Set up the rate-limiting middleware
Middleware used that is not necessary for rate-limiting
This example is from an Express application that uses MongoDB as a database to store the users' data. The following libraries, which will be used in this example, are not necessarily required to set up login rate limiting.
- passport - authentication middleware
- util.promisify() - a method defined in the utilities module of the Node.js standard library. It converts methods that return responses using a callback function to instead return responses in a promise object. The syntax is much cleaner.
- connect-flash - middleware for flash messages notifying a user if the login was successful or not
Submitted data on the request.body is parsed as a JSON object by the built-in middleware function in Express: Express.json(). The data is stored in JSON format as it is a commonly used, organized, and easily accessible text-based format 2.
These were added as application-level middleware in app.js using app.use().
Rate-limiting middleware
The rate-limiting middleware used is a modification of the node-rate-limiter-flexible library example of how to protect a login endpoint. This rate-limiting middleware is written for an Express application using a Redis store, but the same idea can be applied to rate-limiting middleware with other Node.js frameworks such as Koa, Hapi, and Nest or a pure NodeJS application 3. We'll create 2 rate limiters. The first blocks the login route, for one hour, after 10 consecutive failed login attempts. The failed login counts are reset after a successful login. Rate limiting is based on the user's email address and IP address. The second blocks the login route, for one day, after 100 failed login attempts. Rate limiting is based on the user's IP address. After this middleware is set up, we will set up the Redis database.
You can simply rate limit based on IP address only, the problem with this is that IP addresses are not always unique 4. A user in a network that shares a public IP address could block other users in that network. If you limit based on email address only, then a malicious user could block someone's access to the application by simply sending many requests to log in. Blocking by email address and IP address adds some flexibility. A user may be blocked using one IP address but could try login from another device. It is important to note that most devices use a dynamic IP address that changes over time and that IP addresses can be modified 5, 6. Rate-limiting aims to minimize brute force attacks to guess a user's password. When rate limiting, user experience also needs to be considered. Being too strict by blocking users after only a few attempts is not good for the user experience. You need to make a trade-off between security and user experience.
npm packages required for Redis connection and rate-limiting
Rate limit controller
Create a file for the rate-limiting middleware. For example, rateLimitController.js.
In this controller that will handle the login route POST request, a connection to Redis will be set up. Then a rate limiter instance that counts and limits the number of failed logins by key will be set up. The storeClient property of the rate limiter instance will link the rate limiter instance to a Redis database (redisClient) that will be set up later. A points property on the rate limiter instance determines how many login attempts can be made. Keys are created on the instance by using the IP address of the login request or the IP address and email address. When a user fails to log in, points are consumed. This means the count for the key increases. When this count exceeds the points property value, which is the maximum number of failed login attempts allowed, then a message is sent to the user that states that too many login attempts have been made. The keys only exist for a defined amount of time, after this time the rate-limiting is reset. A variable, retrySecs, will be created to determine when a user can try to log in again. The time remaining until another login can be attempted is determined by using the msBeforeNext() method on the rate limiter instance.
If the login route is not being rate-limited, then we will authenticate the user. In this tutorial, Passport is used. If the authentication fails and the user's email exists, then a point will be consumed from each rate limiter instance. If authentication is successful the key for the current user, based on IP address and email address, will be deleted and the user will be logged in. A login session is established using the Passport.js method logIn().
const redis = require('redis');
const { RateLimiterRedis } = require('rate-limiter-flexible');
const passport = require('passport');
// create a Redis client - connect to Redis (will be done later in this tutorial)
const redisClient = redis.createClient(process.env.REDIS_URL, {
enable_offline_queue: false
});
// if no connection, an error will be emitted
// handle connection errors
redisClient.on('error', err => {
console.log(err);
// this error is handled by an error handling function that will be explained later in this tutorial
return new Error();
});
const maxWrongAttemptsByIPperDay = 100;
const maxConsecutiveFailsByEmailAndIP = 10;
// the rate limiter instance counts and limits the number of failed logins by key
const limiterSlowBruteByIP = new RateLimiterRedis({
storeClient: redisClient,
keyPrefix: 'login_fail_ip_per_day',
// maximum number of failed logins allowed. 1 fail = 1 point
// each failed login consumes a point
points: maxWrongAttemptsByIPperDay,
// delete key after 24 hours
duration: 60 * 60 * 24,
// number of seconds to block route if consumed points > points
blockDuration: 60 * 60 * 24 // Block for 1 day, if 100 wrong attempts per day
});
const limiterConsecutiveFailsByEmailAndIP = new RateLimiterRedis({
storeClient: redisClient,
keyPrefix: 'login_fail_consecutive_email_and_ip',
points: maxConsecutiveFailsByEmailAndIP,
duration: 60 * 60, // Delete key after 1 hour
blockDuration: 60 * 60 // Block for 1 hour
});
// create key string
const getEmailIPkey = (email, ip) => `${email}_${ip}`;
// rate-limiting middleware controller
exports.loginRouteRateLimit = async (req, res, next) => {
const ipAddr = req.ip;
const emailIPkey = getEmailIPkey(req.body.email, ipAddr);
// get keys for attempted login
const [resEmailAndIP, resSlowByIP] = await Promise.all([
limiterConsecutiveFailsByEmailAndIP.get(emailIPkey),
limiterSlowBruteByIP.get(ipAddr)
]);
let retrySecs = 0;
// Check if IP or email + IP is already blocked
if (
resSlowByIP !== null &&
resSlowByIP.consumedPoints > maxWrongAttemptsByIPperDay
) {
retrySecs = Math.round(resSlowByIP.msBeforeNext / 1000) || 1;
} else if (
resEmailAndIP !== null &&
resEmailAndIP.consumedPoints > maxConsecutiveFailsByEmailAndIP
) {
retrySecs = Math.round(resEmailAndIP.msBeforeNext / 1000) || 1;
}
// the IP and email + ip are not rate limited
if (retrySecs > 0) {
// sets the response’s HTTP header field
res.set('Retry-After', String(retrySecs));
res
.status(429)
.send(`Too many requests. Retry after ${retrySecs} seconds.`);
} else {
passport.authenticate('local', async function(err, user) {
if (err) {
return next(err);
}
if (!user) {
// Consume 1 point from limiters on wrong attempt and block if limits reached
try {
const promises = [limiterSlowBruteByIP.consume(ipAddr)];
// check if user exists by checking if authentication failed because of an incorrect password
if (info.name === 'IncorrectPasswordError') {
console.log('failed login: not authorized');
// Count failed attempts by Email + IP only for registered users
promises.push(
limiterConsecutiveFailsByEmailAndIP.consume(emailIPkey)
);
}
// if user does not exist (not registered)
if (info.name === 'IncorrectUsernameError') {
console.log('failed login: user does not exist');
}
await Promise.all(promises);
req.flash('error', 'Email or password is wrong.');
res.redirect('/login');
} catch (rlRejected) {
if (rlRejected instanceof Error) {
throw rlRejected;
} else {
const timeOut =
String(Math.round(rlRejected.msBeforeNext / 1000)) || 1;
res.set('Retry-After', timeOut);
res
.status(429)
.send(`Too many login attempts. Retry after ${timeOut} seconds`);
}
}
}
// If passport authentication successful
if (user) {
console.log('successful login');
if (resEmailAndIP !== null && resEmailAndIP.consumedPoints > 0) {
// Reset limiter based on IP + email on successful authorisation
await limiterConsecutiveFailsByEmailAndIP.delete(emailIPkey);
}
// login (Passport.js method)
req.logIn(user, function(err) {
if (err) {
return next(err);
}
return res.redirect('/');
});
}
})(req, res, next);
}
};
Extra notes
Within the RedisClient, the property enable_offline_queue is set to false. This is done to prevent issues such as slowing down servers if many requests are queued due to a Redis connection failure. The author of node-rate-limiter-flexible recommends this setting unless you have reasons to change it 7.
req.ip contains the remote IP address of the request 8. If you are using the Express app behind a reverse proxy, such as Cloudflare CDN, then you should set the Express apps trust proxy setting to true and provide the IP address, subnet, or an array of these that can be trusted as a reverse proxy. If you do not do this, the value of req.ip will be the IP address of the reverse proxy 9. Also note that running your application locally during development, req.ip will return 127.0.0.1 if you are using IPv4 or ::1, ::fff:127.0.0.1 if you are using IPv6 10. These describe the local computer address.
In index.js, the file with all of your routes. The following route is defined:
router.post('/login', catchErrors(rateLimitController.loginRouteRateLimit));
catchErrors is an error handling function that is used to catch any async-await errors in the controller. This error handling method is from the Wes Bos course Learn Node.
The errors for a Redis connection failure are handled as follows: Node Redis returns a NR_CLOSED error code if the client's connection dropped. ECONNRESET is a connection error. You can also set up a retry strategy for Node Redis to try and reconnect if the connection fails 11.
if (err.code === 'NR_CLOSED' || err.code === 'ECONNRESET') {
req.flash('error', 'There was a connection error');
res.redirect('back');
Set up Redis
The code above will not work yet as there is no Redis database set up. We will create a Redis database in the cloud using Redis Labs. We will use the free plan. Then we will connect to this database through our Express app. To view the database, we will download Redis locally so that we can use the built-in client redis-cli (command-line interface). We will download and use Redis using Windows Subsystem for Linux (WSL), which allows you to use a Linux terminal in Windows. Other methods are described on the Redis website download page.
Create an account with Redis Labs
Create an account on the Redis Labs website. Follow the instructions in the documentation to learn how to create a database.
Connect the Redis instance on Redis Labs with your Express application
In your express application variables.env add the REDIS_URL:
REDIS_URL=redis://<password>@<Endpoint>
Your Endpoint and password can be found in the database in the Configuration details of the View Database screen:
- The Endpoint setting shows the URL for your database and the port number.
- The Access Control & Security setting shows the password.
In the rate limit controller from the previous section, the following code connects the cloud Redis instance, hosted on Redis Labs, to the Express application:
const redisClient = redis.createClient(process.env.REDIS_URL, {
// if no connection, an error will be emitted
enable_offline_queue: false
});
The rate limiter instances connect to the cloud Redis instance as follows (also from the rate limit controller):
const limiterSlowBruteByIP = new RateLimiterRedis({
storeClient: redisClient,
...
const limiterConsecutiveFailsByUsernameAndIP = new RateLimiterRedis({
storeClient: redisClient,
...
Set up WSL and download Redis
You will be able to rate limit your login route now, the next step is to set up Redis locally so that we can view the Redis database using the Redis command-line interface (redis-cli). Redis works best with Linux. Linux and OS X are the two operating systems where Redis is developed and tested the most. Linux is recommended for deployment 12, 13.
You can follow this article on how to set up WSL, download and install a supported Linux distro and install Redis locally. Install Redis somewhere outside of your application. The Linux distro used in this tutorial is Ubuntu 18.04.
Connect the redis-cli to the Redis instance on Redis Labs
We will use the redis-cli locally to see the key-value pairs created. Run your Express application and in a WSL terminal run the redis-cli:
- cd into the Redis folder that you downloaded
cd redis-6.2.3
- make sure the server is running
sudo service redis-server start
to stop the server:
sudo service redis-server stop
If you run redis-cli, you will connect to the local instance of Redis and will run locally on the Localhost (127.0.0.1:6379). To exit, run quit. To connect the redis-cli to the cloud instance of the Redis Labs database that we created, we'll use the URL-based connection method from the Redis Labs docs. This connects to the Redis database using an endpoint URL and port number. Check the database Configuration details in the View Database screen to find the endpoint url and password.
$ redis-cli -h redis-19836.c9.us-east-1-2.ec2.cloud.redislabs.com
-p 19836 -a astrongpassword
h is the host: add your endpoint, without the port number
p is the port, which is shown at the end of the endpoint url
a is access control. Add your password
You can test if the connection worked by typing PING. If the connection worked redis-cli will return PONG.
if the response is NOAUTH Authentication required - check that you typed the password correctly. You can run quit to exit the redis-cli so that you can try again.
Basic Redis commands
There are many commands available as shown in the docs. For our use case, we only need to know a few simple commands. You can try them in the redis-cli that is connected to your Redis Labs Redis instance. Note that the commands are all uppercase in the Redis docs, but the commands are not case-sensitive. However, key names are case-sensitive.
PING
Checks the connection to the Redis database. If there is a connection, PONG will be returned.
SET
Set the string value of a key. It is used to create a key-value pair or change the value of an existing key.
> SET job teacher
OK
This sets the key "job" to the value "teacher". The response OK means that the command was successful.
MSET
Like SET, but its sets the values of multiple keys.
> MSET job "teacher" AGE "50" TITLE "Mr."
OK
GET
Get the value for a key.
> GET job
"teacher"
MGET
Get the value of multiple keys.
> MGET job age title
1) "teacher"
2) "50"
3) "Mr."
DEL
Deletes a specific key.
> DEL job
(integer) 1 -> this means that it found a key with the name "job" and deleted it.
If you try :
> GET job
(nil) -> this means that no key with the name "job" exists.
SCAN
View all keys. It iterates over a collection of keys. It is a cursor-based iterator. If you want to view all entries then run
> SCAN 0
1) "0"
2) "age"
3) "title"
The first value returned is "0", which indicates that a full iteration occurred. This means that all of the keys in the database were scanned. For more details, you can read up the description of the SCAN command in the docs.
If you want to view all keys, excluding the first key then run SCAN 1.
FLUSHALL
This deletes all of the keys in the database.
CLEAR
Clears the terminal.
Test the rate-limiting
We are going to test one of the rate limiters. Run your application locally and connect to Redis labs via the redis-cli in a WSL terminal. Before starting, make sure all of the keys in your database are deleted by running the command FLUSHALL. In your rate limit controller middleware (rateLimitController.js.), set maxConsecutiveFailsByEmailAndIP to 3. Set the options duration and blockDuration of limiterConsecutiveFailsByEmailAndIP to 60. This will allow us to test the rate-limiting quickly.
...
const maxConsecutiveFailsByEmailAndIP = 3;
...
const limiterConsecutiveFailsByEmailAndIP = new RateLimiterRedis({
storeClient: redisClient,
keyPrefix: 'login_fail_consecutive_email_and_ip',
points: maxConsecutiveFailsByEmailAndIP,
duration: 60
blockDuration: 60
});
...
Failed login with an account that does not exist
Try login using an email (or another user identifier, such as user name, used in your app) that does not exist (not registered).
After this, in the redis-cli, that is connected to your cloud Redis instance hosted on Redis Labs, view all of the keys.
yourRedisLabsEndpoint> SCAN 0
1)"0"
2) "login_fail_ip_per_day:::1"
On localhost,
req.ipwill return 127.0.0.1 if you are using IPv4 or ::1, ::fff:127.0.0.1 if you are using IPv6.
You can now check the number of consumed points (number of failed logins) of the limiterSlowBruteByIP rate limiter for the IP that tried to log in.
yourRedisLabsEndpoint> GET login_fail_ip_per_day:::1
"1"
Failed login with an account that does exist
Now try log in with an existing account and use the wrong password. Then view all of the keys in your Redis database.
yourRedisLabsEndpoint> SCAN 0
1)"0"
2) "login_fail_ip_per_day:::1"
3) "login_fail_consecutive_username_and_ip:realuser@example.com_::1"
You can now check the number of points consumed for the IP that tried to log in for the limiterSlowBruteByIP rate limiter key.
yourRedisLabsEndpoint> GET login_fail_ip_per_day:::1
"2"
Check the number of consumed points for the limiterConsecutiveFailsByEmailAndIP rate limiter key.
yourRedisLabsEndpoint> GET login_fail_consecutive_username_and_ip:realuser@example.com_::1
"1"
Try logging in more than 3 times within 1 minute. After this, you will get this message displayed in your browser:
Too many requests. Retry after 60 seconds.
The login route for the given IP and username pair will be blocked for 60 seconds. This is because the blockDuration that we set for the limiterConsecutiveFailsByEmailAndIP rate limiter is 60 seconds. After 60 seconds, check the number of consumed points for the key again:
yourRedisLabsEndpoint> GET login_fail_ip_per_day:::1
(nil)
It does not exist anymore as we set the duration property to 60. The key is deleted after 60 seconds.
Now try login using an existing account with the wrong password. After this, log in with the correct password. This will delete the limiterConsecutiveFailsByEmailAndIP rate limiter key for the given user and IP pair. This occurs once the login is successful, as can be seen in the rate limit controller:
...
if (resEmailAndIP !== null && resEmailAndIP.consumedPoints > 0) {
// Reset on successful authorisation
await limiterConsecutiveFailsByEmailAndIP.delete(emailIPkey);
}
...
You can do more thorough testing of POST requests using services such as Postman, which is a tool used to build and test APIs.
Conclusion
This is a basic example of how to rate limit a login route in an Express app using node-rate-limiter-flexible and Redis. node-rate-limiter-flexible was used to count and limit the number of login attempts by key. Redis was used to store the keys. We created a rate limiter middleware in an existing application with a login route and authentication. Two rate limiters were created. The first rate limiter rate-limited based on IP. The second rate-limited based on IP and the user's email address. Redis Labs was set up to create an online instance of Redis. The Redis Labs instance was connected to the Express app using an endpoint URL. Redis was installed locally and was connected to the online instance of Redis. Rate-limiting was tested by viewing the database keys, using the redis-cli, after attempted logins.
Here are some useful links for further study:
1) Redis Crash Course Tutorial - Learn the basics of Redis
2) Redis Caching in Node.js - Learn how to cache API calls using Redis.
3) API Rate Limiting with Node and Redis
Top comments (1)
Hi, great article, it really helped me. But I found a small mistake in your code related to passport.js.
Instead of this:
passport.authenticate('local', async function(err, user) {there should be this:
passport.authenticate('local', async function(err, user, info) {