Top 10 Security Best Practices we learned the hard way

Our users entrusted us with their data and it's our duty to keep this data secure as they use our application. Unfortunately, security best practices aren't the first things that developers learn when they start out. It's usually learned through experience, when the incident already happened.

In this article, I'll share our team's experiences and research on how to make your applications secure. It is by no means an exhaustive list, but these are the most common things to miss.

Let's dive in with an example eCommerce Store

For this article, let's imagine Fred is an application developer for GoodProducts Manila (GP Manila). It's an eCommerce company that has a website gpmanila.com. One of his ex-workers Nate is trying to do some damage. Here are a few ways he might do it:

[1] Users can access the data of others by URL hijacking

This is by far the simplest security exploit to do, (and thankfully), the simplest to remediate.

Let's dive into an example:

Users of GP Manila can register for an account and log in. They can view products and place orders for these products. There is also an orders page where she can track her current and past orders, the URL looks like this:

https://gpmanila.com/orders?user_id=12345 or https://gpmanila.com/users/12345/orders

As you may infer, this URL allows users to place the user_id of another user and possibly be able to access their order data. Nothing is stopping Nate from accessing a URL like https://gpmanila.com/orders?user_id=66666 to try and see the data of other users. Even if the user_id is random, I can create an automated script to keep trying different combinations until I get something.

Another variation of this exploit is when the frontend URL is safe, something like https://gpmanila.com/orders but the backend URL being called is something like:

https://api.gpmanila.com/orders?user_id=12345

This made it a little difficult for non-tech users to see this vulnerability. But our more techy friends will just have to see open the web browser's console and see this waiting for them.

[2] Users can access the data of others by Form hijacking

Another similar vulnerability is when our form submissions are too lax. For example, our "Checkout Order" API endpoint for GP Manila contains the following request body:

POST: https://gpmanila.com/orders/:checkout_order

{
  "items": [
    {"name": "silver bag", "price": 100, "quantity": 1}
  ],
  "total": 100,
  "user_id": 12345,
  "address": "8 Somewhere Street, Manila City, Philippines"
}

While this method is more secure than #1, it is merely security by obscurity. By hiding the user_id inside the API request data, it is harder to find. But it is still there.

Nate the hacker can just change the user_id of the API request so that his order will be charged to another user but will be delivered to his address.

Resolve [1] and [2] by using JWT tokens

In this case, we are using an OAuth service called Amazon Cognito which handles the authentication of our app separately from our backend.

When a user logs in, he should receive an ID token and a Refresh token from Cognito. The frontend then adds the ID token to the Authorization header of every API request being sent to the backend.

The ID token expires every hour. Once it expires, the backend throws an "IdTokenExpired" error to the FE. The FE takes this as a signal to use the refresh token to get another ID token from Cognito. This ID token is valid for another hour. And is renewed for every hour since.

Both the ID token and the Refresh token are in JWT token format. Before Cognito sends these tokens to the frontend on login, it signs the JWT token with a private key. The process of signing makes sure that if the JWT body gets tampered with, the verification process will fail.

Once the frontend sends the ID token back on every API request, the backend uses the public key to verify that the JWT token body has not been tampered with.

So even when Nate tries to change the user_id in the ID token, the backend's verification of the signature will fail.

Learn more about JWT tokens here

[3] Form submissions allow XSS attacks

At GoodProducts Manila, we have 1000 products. And we let our users review these products. Since its contents are not moderated, the reviews are posted right away.

Nate the hacker can exploit this by adding JavaScript runnable content on one of our most famous products.

<script> alert("this is a popup") </script>

This automatically gets saved in the database and becomes immediately visible to our other users. Since the review is JavaScript code, your frontend may recognize this as executable JavaScript. Hence, when other users view this review, the JS code executes.

Now, my sample JS code looks safe. It simply displays a popup for every user who sees this review. But imagine the more sinister things that Nate can do when he can execute JS code on other users' browsers. He can access the user's session cookies and send them to a remote server he controls. That way, he can log in as that user and create transactions on their behalf.

This exploit is called Cross-Site Scripting (XSS).

Resolve this by adding AWS WAF or any web firewall

Adding web firewalls like AWS WAF to your frontend and backend deployment can help remediate this. Before the review even gets created to your BE, the firewall takes a quick look at the request body and compares it with the filter rules you have set up. Most web firewalls have anti-XSS capability covered by default, helping you filter our requests with JavaScript code inside.

[4] Uploaded files have a virus

We're also looking for good developers for Good Products Manila. So we created a "Contact Us" form where you can upload your resume along with other contact information.

Nate the hacker can upload his resume in PDF format and add a virus along with it. When your HR opens the file on their end, the virus is also executed and goes on to infect his laptop.

Resolve this by adding anti-virus

One way to remediate this is to add anti-virus to our application. In our case, we upload the resume to the S3 bucket and just save an S3 object URL to the database.

We can build our own lambda function that scans every object that gets uploaded to the S3 bucket... Or we can buy one from the AWS Marketplace. Here are the top two options we have tried.

For both options, once you have subscribed through the marketplace, you will be redirected to AWS CloudFormation to provision the service to your AWS account.

Cloud Storage Security

Cloud Storage Security is the top option in terms of anti-virus. It provides access to an account in their web application. You can also add other users/groups to this account, and manage what type of access they have.

It also can protect EBS and EFS volumes. And it integrates nicely with AWS Security Hub, AWS CloudTrail Lake, and AWS Transfer Family.

However, the downside of this product is the cost. It charges 99USD per month for the first 100GB scanned, and 0.80USD per GB after that. All of this is on top of the cost of the AWS resources provisioned, which is usually at 40 USD (for a basic EC2 instance).

bucketAV powered by ClamAV - Antivirus for Amazon S3

BucketAV is cheaper but it does the basic protection job just as well. But it skimps on more advanced features that Cloud Storage offers.

It charges 0.05 USD per hour on instance sizes up to m5.large, but you have to shell out for the costs of running the container infrastructure on ECS.

Its UI is built on top of a CloudWatch dashboard.

[5] Error messages are too specific

In the spirit of having UI that's helpful to the customer, we tell them where they went wrong. Example error notices are:

• User with this email does not exist
• The password entered for this email is wrong
• The password used has already been expired
• The user is inactive

But these errors give Nate a lot more information when doing a brute-force attack. He can brute force 1000 emails and be able to know which emails have user accounts on our side. For each user, he can also brute force the password and know the password is wrong. He can keep trying until he gets it right. That's why the best error notification we can give is:

• User credentials are invalid

It provides little context for Nate, just that he didn't get the password right.

[6] Password policies are too simple

We often have to protect users against themselves. Simple passwords can be cracked instantly. By forcing them to a minimum character count, and adding symbols/letters/numbers, we are increasing the complexity of the password, making it harder to crack:

[7] No API or FE rate-limiting

The easiest way to brute force a password is by accessing the APIs. One way to prevent it is by adding basic rate-limiting to your APIs and frontend assets. Essentially, we limit each IP address to having a maximum of 500 calls per 5 minutes (for example). When the quota is exceeded, the FE or BE returns an error.

Rate-limiting is also the easiest way to protect against DDoS attacks. In this blog post, we tell our experience of how we survived a DDoS attack and why rate-limiting was central to our strategy.

By experience, we have to fine-tune the number. If the limit is too small, we will end up rate-limiting normal users, and end up penalizing heavy users of our site. If the limit is too big, it offers no real protection against brute force attacks. We also figured out that frontends needs a higher rate limit. Because we also serve fonts, images, and other static assets. This eats up on our limit, and we can reach it quicker compared to the backend.

[8] API logs contain the entire request body

There is no doubt that having some sort of API request/response logging is very helpful for developers to debug issues in production. But if devs aren't careful, they may be creating more headaches for themselves later on.

If we apply this logging blindly, we'll end up saving passwords, credentials, and other PII data on API logs. These logs tend to be less secure than the actual database. These logs usually sit as files inside our server, waiting for a potential hacker to penetrate the server.

Resolve this issue by truncating valuable data from the logs as you write them

# instead of
{ "email": "raphael.jamby@gmail.com", "password: "imhandsome12"}

# do instead:
{ "email": "XXXX", "password: "XXXX"}

[9] Credentials are pushed on the frontend

One final issue is adding hardcoded credentials on the frontend. Sometimes, the frontend needs to access AWS resources directly: upload files to S3, access Cognito APIs for login, etc.

The fastest way to do this is to use the AWS JavaScript SDK and add the AKID and secret key onto the frontend. This is a big mistake, especially if the frontend is a single-page application. There are a lot of web crawlers out there that scour the internet for these keys. And since SPAs are uploaded and delivered to the client in their entirety, the keys will be visible. In a matter of hours after deployment, you'll feel the effects of being hacked. And if the keys you uploaded are admin access, The Armageddon is coming your way.

The best way to resolve this is not to use AWS AKID and Secret, but instead use AWS Amplify with Cognito. Amplify is a library that helps you call Cognito when your users log in and register. When your user logs in, Cognito and Amplify grant temporary access to specified AWS resources.

[10] Not adding API request sanitation

When you are developing your application, you know exactly what each of your API endpoints expects for it to work: attribute name, the data type, the range of valid values, etc. You should use tools like Pydantic or Cerberus. These tools check request data against the rules you these expectations to make sure your app gets only what it expects, otherwise, it throws an error.

When you don't do request sanitation, you allow users to enter values in the API request that may potentially break the system.

[BONUS] Your security is only effective if all of your team knows it too

This final tip is not a technical one. We can have all of these best practices but if at the end of the day, your team doesn't understand why they are doing it, they probably won't end up implementing it.

Want to share other security best practices? Comment them below!

Photo by FlyD on Unsplash