Read this before Deploying VPC-Endpoints and S3 with Terraform.

In this article, you will learn how to securely limit access to your organization's private AWS-S3 bucket production workload from within a specific VPC using VPC endpoints with Terraform.

Case

As part of a security plan that aims to reduce organizational cost, your manager reaches out to you to talk about re-structuring the sensitive data flow from your private VPC to your private s3 bucket and from the s3 bucket back to the VPC.

Current infrastructure layout

The current infrastructure allows traffic over the internet and this requires a lot of configurations with firewall rules which we all know can be extremely complex if it is not properly implemented. The data transfer cost is higher than planned due to the use of a NatGateway device. Grain access control using SSL and TLS is excessive and complex. Finally, multiple resources across regions results in increased latency.

Your manager asked you

What can you do to ease this transformation but at the same time meet what is required for our business to keep running securely and with minimal cost?

Immediately you see the opportunity of using VPC endpoints for this use case. But before answering the above question let’s first consider the following:

Do you know what VPC endpoints are?

VPC endpoints are redundant virtual devices that scale horizontally and are highly available VPC components.

There are currently 3 different types of VPC endpoints (known to date) and each one is used depending on the supported service.

1. Interface endpoints
2. Gateway Load Balancer endpoints
3. Gateway endpoints

Now that we have a brief summary of what VPCe are we need to understand when and why we use them.

When you use a VPCe, you eliminate the need of using Internet-Gateway or a Nat-Gateway device. VPCe runs exclusively using the AWS backbone network. In this article, we will be focusing on Gateway endpoints.

How do VPC-Gateway endpoints work?

VPC-Gateway endpoint is used to communicate privately and securely with public resources such S3 and DynamoDB.

Let’s analyze this scenario a little deeper first. Envision that you have multiple services hosted within your VPC and you want to give them access to these public services and you do not necessarily want traffic flowing all over the Internet and do not want to set up a VPN connection. Well, as you will experience at the end of this article VPC-Gateway endpoint could help you to avoid all the unnecessary configurations very effectively.

What do you need to set up a VPC-Gateway endpoint connection with an S3 bucket and how does it work?

First of all, you will need a VPC with one to two subnets and a route table. You will then need to create a VPCe type Gateway endpoint per region to connect with the S3 bucket. The connection occurs at the subnet level, a prefix list (pl-xxxxxxx) is attached to the route table associated with your subnet(s) and an AWS prefix list name uses the form "com.amazonaws.region.service". One important fact to mention about VPCe is that AWS updates the prefix list on your behalf. Do you remember the cumbersome work of updating multiple firewalls? That is now gone 😊

Route table configuration example

You might be thinking ok this definitely makes the connection easier but how can I make this fully secure?

VPC endpoints offers an out of the box endpoint policy by default. This option allows full connection to each S3 bucket in our account which we do not necessarily want. Luckily you can also create your own robust custom policy. See example below.

In this example custom policy all requests or traffic outside your VPC will be denied

resource "aws_s3_bucket_policy" "allow_access_to_specific_vpce_only" {
  vpc_endpoint_id = aws_vpc_endpoint.gw_endpoint.id
  policy = jsonencode({
    "Version": "2012-10-17",
    "Statement": [
        {
        "Sid": "Access-to-specific-VPCE-only",
        "Principal": "*",
        "Action": "s3:*",
        "Effect": "Deny",
        "Resource": ["${aws_s3_bucket.work_load_bucket.arn}",
                    "${aws_s3_bucket.work_load_bucket.arn}/*"],
        "Condition": {
            "StringNotEquals": {
            "aws:SourceVpce": "${aws_vpc.vpc_end_point.id}"
            }
        }
    }
    ]
  })
}

Although Gateway endpoint seems a perfect fit in this case it has some current limitations (again, known to date):

• Endpoints are supported within the same Region only. You cannot create an endpoint between a VPC and a service in a different Region.
• Endpoints support IPv4 traffic only.
• You cannot transfer an endpoint from one VPC to another, or from one service to another.
• You have a quota on the number of endpoints you can create per VPC.
• Endpoint connections cannot be extended out of a VPC.

Documentation here

VPCe diagram

AWS data transfer cost

credits to The Open Guide to Amazon Web Services

Conclusion

I hope this article has given you a better understanding of the power of VPC endpoints and its limitations. Now go get some real work deployed using Terraform!

Functions, arguments and expressions of Terraform that were used in this project:

provider
variables
resources

Find the Terraform repo and directions for this project here