Ingest VPC Flow Logs into NewRelic

There are many use cases where we wanted to monitor the VPC Flow Logs to view the data going IN / OUT into our VPC. These network traces helps us to troubleshoot many network-related issues.

We do have a choice in AWS to save VPC Flow Log either into

• AWS CloudWatch or

• AWS S3 Buckets.

However, both of these solutions don't provide a good user-friendly view and can become cumbersome when trying to find a specific IP Address or Port.

Well thanks to Kinesis Data Firehose to provide us pretty much option to cope up these situation. We can ingest the data from many possible AWS services into Kinesis Data Firehose and send that to 3rd party monitoring solution to create some AWSome custom Dashboard and monitor the logs.

I will walk you through step by step to configure this solution in this blog. We can divide this into 3 parts

1. Create Kinesis Data Firehose
2. Create the VPC Flow Logs
3. Transform the Log using Lambda function (Optional)
4. Send the Logs to NewRelic Monitoring Solution

Create Kinesis Data Firehose

Create a Kinesis Data firehouse and select Source as Direct PUT and Destination as New Relic. Please note that Kinesis Data Firehose is near real time solution but not the real time solution as Kinesis.

Under Destination Setting - Select HTTP Endpoint URL as NEW Relic Log - US. Enter the API KEY (Copy the API Key form New Relic)

Click on following URL, this will land you to NewRelic API-Key screen, as shown below

https://one.newrelic.com/admin-portal/api-keys/home?

Create VPC Flow Log

Go to VPC --> Action and click on Create flow log

Under Filter, select weather you only want to monitor the ACCEPTED Traffic, REJECTED Traffic or ALL Traffic.

Under Destination, select Send to Kinesis Data Firehose in same account and select the Kinesis Data Firehose

This will take few seconds and then you will start seeing the data into NewRelic platform

If you like this article then don't forget to hit the like and share with others :)