DEV Community

Marcin Piczkowski
Marcin Piczkowski

Posted on

How to conditionally upload Lambda artifact to s3 with Terraform?

I have 2 projects on Gitlab. One with terraform to provision infrastructure, another with Lambda code. Lambda is configured to be deployed from S3 bucket.

In terraform I have a dummy zip file uploaded to s3. I had to give any ZIP otherwise Terraform complained during apply.

I use a separate project to keep Lambda code, which has separate build pipeline and deploy to the same s3 bucket. It should overwrite the dummy ZIP on deploy.

The problem is that whenever Terraform executes the real ZIP is overwritten with the dummy one again, so I need to deploy the real ZIP from another project again.

I don't want to keep function code together with Terraform project.
I also don't want my Terraform build to always trigger another project build (containing function source code), because most often Lambda does not change but other resources are modified in Terraform configuration.

I thought about having another Lambda triggered when ZIP is uploaded to S3 which would verify if it is a real or dummy one.
In case of dummy one it would trigger another project deployment using Gitlab API.

Is there any easy solution to my problem?

Top comments (8)

Collapse
 
jesusgollonet profile image
jesús gollonet • Edited

Can I ask about the purpose of the dummy zip? Is it intended to provision an empty lambda?

I'm not sure I understand the specific problem but I'll tell you that the boundaries between what is infra (generally terraform) and what is business logic (usually serverless/lambda) are far from clear cut for me.

Collapse
 
piczmar_0 profile image
Marcin Piczkowski

yes, the dummy zip is just to allow provisioning of lambda with Terraform. I may be wrong, but when I configured Lambda to be deployed from s3 I had to give s3 key and it had to exist, otherwise Terraform complained, so I had to put anything there.

Collapse
 
jesusgollonet profile image
jesús gollonet • Edited

What I would do is either move the terraform part to the lambda repo or get rid of it if you already have a way to deploy the lambda.

In my mind there's no such thing as "provisioning" a lambda and then "deploying" the code as one does in a serverful approach, and I believe keeping that mindset is what leads to the awkwardness you're dealing with. You don't gain anything by having an empty lambda placeholder waiting for your code, as the code and the infra are almost one and the same.

Hope that's helpful (and I'm making sense). Feel free to share code if you want more specific advice.

Thread Thread
 
piczmar_0 profile image
Marcin Piczkowski

This makes sense. Unfortunately the infrastructure is hybrid and consists of some ECS services inside vpc. Lambdas need to have access to VPC, so it needs subnet IDs, security group IDs and also URLs to the ECS services and if I keep all in one project then I can use references, otherwise I would have to pass the values of provisioned resources as input arguments to the separate Terraform configuration.

In addition, separate team is responsible for infrastructure and Terraform and separate for coding.

So in your opinion we should split the Terraform config so that both teams are responsible for their own parts?

Thread Thread
 
jesusgollonet profile image
jesús gollonet

Ok i see how keeping it in an infra repo makes sense in that case. I usually deal with much smaller stuff so take anything I say with a grain of salt.

A more workable approach might be adding the lambda code as a dependency of the infra. Either as a git submodule or grabbing with a shell script from terraform.

But again, far from an expert on that kind of situation so I'll stop here :P

Thread Thread
 
piczmar_0 profile image
Marcin Piczkowski

I'll think about your suggestion as well. Thanks :)

Collapse
 
piczmar_0 profile image
Marcin Piczkowski

I just found this serverless.com/blog/definitive-gui...

This makes a lot of sense to separate app infrastructure from shared infrastructure.

Collapse
 
piczmar_0 profile image
Marcin Piczkowski • Edited

This discussion gave me some more ideas on how to build separate TF configurations dependent on each other. github.com/hashicorp/terraform/iss...