Hello everyone, in this post I will be demonstrating how you can run localstack with Terraform and Docker and give you a proof of concept go application so you can tweak it according to your logic and follow anything you want to do such as integration/system tests for AWS services in your own CI/CD or localhost.
Github Repository for PoC(proof of concept):
hotdog-PoC-repository
Requirements:
- Docker
- docker-compose
- Terraform
- Go
- aws CLI
- A bit of lambda, dynamodb and kinesis knowledge
Intro
Localstack is a testing/mocking framework for developing Cloud applications locally. Where in theory, you can stick any AWS service and emulate them in localhost without ever needing the real AWS account.
Localstack’s primary goal to make integration/system testing less painful for developers.
What was built?
I built an imaginary hotdog food chain! (Note: No dogs were harmed in this process). Essentially PoC logic was I had 1 dogs dynamodb table which consist a dog model with 3 attributes ID, name, isAlive and isEaten. Then I had 3 lambdas dogCatcher, dogProcessor and hotDogDespatcher. dog catcher's responsibility is to get alive dogs via external API requests(I generated data for simplicity) with unique IDs and different names. Dog processor's responsibility is to kill the dogs and persist the data that was sent from dog catcher. Hot dog despatcher's responsibility is to give processed dogs(hot dogs) to people and observe which ones were eaten via external API requests(I assumed hot dogs get eaten if their name has case-insensitive "e" or "a" letter)
Aside from lambdas, I had 3 kinesis streams and 3 kinesis triggers in order to make lambdas talk to each other. The named kinesis streams is as follows; caughtDogs, hotDogs, eatenHotDogs.
Starting Localstack docker container with docker-compose
version: '3.8'
services:
localstack:
container_name: "localstack_main"
image: localstack/localstack:latest
environment:
- SERVICES=dynamodb,lambda,kinesis
- LAMBDA_EXECUTOR=docker_reuse
- DOCKER_HOST=unix:///var/run/docker.sock
- DEFAULT_REGION=ap-southeast-2
- DEBUG=1
- DATA_DIR=/tmp/localstack/data
- PORT_WEB_UI=8080
- LAMBDA_DOCKER_NETWORK=localstack-tutorial
- KINESIS_PROVIDER=kinesalite
ports:
- "53:53"
- "53:53/udp"
- "443:443"
- "4566:4566"
- "4571:4571"
- "8080:8080"
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- localstack_data:/tmp/localstack/data
networks:
default:
volumes:
localstack_data:
networks:
default:
external:
name: localstack-tutorial
docker-compose up -d --build
Bootstrapping our infra with Terraform
provider "aws" {
region = "ap-southeast-2"
access_key = "fake"
secret_key = "fake"
skip_credentials_validation = true
skip_metadata_api_check = true
skip_requesting_account_id = true
endpoints {
dynamodb = "http://localhost:4566"
lambda = "http://localhost:4566"
kinesis = "http://localhost:4566"
}
}
// DYNAMODB TABLES
resource "aws_dynamodb_table" "dogs" {
name = "dogs"
read_capacity = "20"
write_capacity = "20"
hash_key = "ID"
attribute {
name = "ID"
type = "S"
}
}
// KINESIS STREAMS
resource "aws_kinesis_stream" "caught_dogs_stream" {
name = "caughtDogs"
shard_count = 1
retention_period = 30
shard_level_metrics = [
"IncomingBytes",
"OutgoingBytes",
]
}
resource "aws_kinesis_stream" "hot_dogs_stream" {
name = "hotDogs"
shard_count = 1
retention_period = 30
shard_level_metrics = [
"IncomingBytes",
"OutgoingBytes",
]
}
resource "aws_kinesis_stream" "eaten_hot_dogs_stream" {
name="eatenHotDogs"
shard_count = 1
retention_period = 30
shard_level_metrics = [
"IncomingBytes",
"OutgoingBytes",
]
}
// LAMBDA FUNCTIONS
resource "aws_lambda_function" "dog_catcher_lambda" {
function_name = "dogCatcher"
filename = "dogCatcher.zip"
handler = "main"
role = "fake_role"
runtime = "go1.x"
timeout = 5
memory_size = 128
}
resource "aws_lambda_function" "dog_processor_lambda" {
function_name = "dogProcessor"
filename = "dogProcessor.zip"
handler = "main"
role = "fake_role"
runtime = "go1.x"
timeout = 5
memory_size = 128
}
resource "aws_lambda_function" "hot_dog_despatcher_lambda" {
function_name = "hotDogDespatcher"
filename = "hotDogDespatcher.zip"
handler = "main"
role = "fake_role"
runtime = "go1.x"
timeout = 5
memory_size = 128
}
// LAMBDA TRIGGERS
resource "aws_lambda_event_source_mapping" "dog_processor_trigger" {
event_source_arn = aws_kinesis_stream.caught_dogs_stream.arn
function_name = "dogProcessor"
batch_size = 1
starting_position = "LATEST"
enabled = true
maximum_record_age_in_seconds = 604800
}
resource "aws_lambda_event_source_mapping" "dog_processor_trigger_2" {
event_source_arn = aws_kinesis_stream.eaten_hot_dogs_stream.arn
function_name = "dogProcessor"
batch_size = 1
starting_position = "LATEST"
enabled = true
maximum_record_age_in_seconds = 604800
}
resource "aws_lambda_event_source_mapping" "hot_dog_despatcher_trigger" {
event_source_arn = aws_kinesis_stream.hot_dogs_stream.arn
function_name = "hotDogDespatcher"
batch_size = 1
starting_position = "LATEST"
enabled = true
maximum_record_age_in_seconds = 604800
}
./zip-it.sh
terraform init
terraform plan
terraform apply --auto-approve
Checking with aws CLI if everything is setup correctly
To see if everything was working correctly, I invoke dogCatcher and check out the dynamodb table;
aws lambda invoke --function-name dogCatcher --endpoint-url=http://localhost:4566 --payload '{"quantity": 2}' output.txt
aws dynamodb scan --endpoint-url http://localhost:4566 --table-name dogs
Result
I had pretty much great experience with Localstack. I think even though Localstack is quite new, it seems like it can be used for learning AWS SDKs as a developer without actually using live AWS services and getting billed for it. This can also speed up developer's integration tests(along with CI/CD) and debugging processes if configured properly because there are many services Localstack provides and I have only configured and used 3 of them here. This also saves lots of costs for any companies.
Also don't forget to check out Localstack's slack channel, they are really helpful for any issues you run into and follow me on Twitter for further questions!
localstack-slack
my-twitter
Top comments (2)
Great article!!!
One question: Why are there no lambdas?
the array should not be empty
additional info:
seems the lambdas were created accordingly to terraform
Also, in your post you also receive an empty array. What am I missing?
Found the issue! I was using eu-west-1 as my default aws region. Once I change it everything worked perfectly.
Great article man!! Thank you so much!!