Notification 1:All code related to this blog post can be found at
https://github.com/turjachaudhuri/AWS-Serverless/tree/SAMRepositoryApp1
Feel free to fork and let me know what wonderful things you have done with my crappy code.
What is AWS Serverless Application Repository?
The AWS Serverless Application Repository makes it easy for developers and enterprises to quickly find, deploy, and publish Serverless applications in the AWS Cloud . For more details check out - https://docs.aws.amazon.com/serverlessrepo/latest/devguide/what-is-serverlessrepo.html
Basically , this repo is a collection of Serverless applications that have been contributed by AWS teams and developers across the world . These applications can directly be deployed into your own AWS account with so much as a click of a button , and setting some parameters (more on this later).
Any application that you may want to submit to these repo needs to follow a few rules , one of which is that your app needs to have a valid AWS Serverless Application Model (AWS SAM) template file that defines the AWS resources which are used by your app.
What is AWS SAM?
For those of you who have some experience in working on Serverless applications , you must be aware that after a while , manually maintaining a Serverless app is quite difficult .Reason being there are so many different components in an app , mostly when it starts to grow complicated and new components are added over time . It is quite easy for an app to have something around 20+ lambdas , their associated events , IAM roles and policies associated those lambdas , DynamoDB tables , S3 buckets and so on . It is extremely difficult to maintain such solutions without some sort of framework . That is where AWS SAM comes in .
AWS SAM has a template file(either in JSON/YAML) that describes everything your app needs to have , starting from the lambdas , where their code is located , DynamoDB tables , associated IAM roles and policies. It is extremely similar to a CloudFormation template , and internally the SAM engine actually converts the SAM template into a CloudFormation(CF) template and then deploys it onto your AWS setup using AWS CF.
SAM makes it very easy to get started with creating Serverless apps and maintain them as they start getting more and more complex and involved. For more information on SAM , check this out - https://github.com/awslabs/serverless-application-model
What did I hope to accomplish when I started?
Basically , I wanted to create a pretty basic Serverless app and publish that to the AWS SAM Repository so that other users can deploy it . Also , I wanted to understand the whole flow , because I had not used SAM earlier .
Previously , I mostly relied on the Serverless Framework for this type of work . If you want to check out Serverless framework , and what it can do for you , please visit this site - https://serverless.com/
What does my app do?
My app is pretty simple by any standard . It simply consists of a lambda function that is triggered every time some object is inserted in a configured S3 bucket , and then loads the filename and timestamp in a DynamoDB table . My objective was not to create a complex Serverless app for a business scenario , I simply wanted to check out how SAM works and how I can leverage it for easier maintenance of my Serverless projects.
Does using SAM mean I don't have to code?
Not at all . SAM is simply a template file that is a technical description of all the components of your app , the events interacting between those components , IAM roles , and so on . It is not an alternative for the code that goes into your lambda functions . You still have to write buildable code , however SAM will link your lambda functions with your code effortlessly.
Also, the SAM engine helps you validate whether the SAM template is proper , and can help you deploy the whole app effortlessly from the SAM-cli using a few basic commands.
Enough ! How to get started?
To install SAM-cli , the easiest way is to get started via pip. SAM-cli is written on python , so python needs to be installed on your system as a prerequisite.
pip install aws-sam-cli
Gotcha 1 : I faced a lot of problem in this otherwise simple installation due to the conflicting versions and dependencies errors between aws-cli , sam-cli , boto3, boto-core and so on. However , they are not fatal errors , and if you simply have patience , and take care of the dependencies one by one , this will be a no-brainer to solve.
Follow this link for detailed instructions - https://docs.aws.amazon.com/lambda/latest/dg/sam-cli-requirements.html
Gotcha 2 : No you don't need docker to install sam-cli . Docker is only needed if you want to do unit testing on your local machine using SAM Local , which will be a different blog post altogether (if I ever figure it out ).
Setting up the project
Go to your working directory and run this command-
sam init --runtime dotnetcore2.0
You can choose any runtime out of the supported ones as per your comfort level . I am more comfortable with C# , so I chose dotnetcore . The above command will create a HelloWorld project for you with a sample template.yaml file and some sample C# code.
Now you can open the solution in your favorite IDE , and code as much as you want.
Gotcha 3 :The template file need not only be in YAML. JSON is also supported . I prefer JSON to YAML as of now . So I simply used template.json instead of template.yaml . However , in that case in all your subsequent commands in the SAM cli , you need to explicitly mention that your template file name is template.json.
Gotcha 4 :Most of the examples in the internet on SAM templates are YAML ones . Being able to convert them to JSON quickly and without any error is a must if you need to proceed . I used the online tool https://codebeautify.org/yaml-to-json-xml-csv a lot during my development for this.
What to do with the template.json file?
Getting started with SAM is a tad difficult if you are not familiar with CloudFormation or any IaC (Infrastructure as Code) scripting tool/language of any sorts . But once you get a hang of it , it is quite intuitive and easy to use . You can describe resources like S3 Bucket , DynamoDB tables , use variables/parameters in your script to make them more dynamic , use some clever CF transformations to help you along the way . Also , there are a lot of examples on the web to help you out in case you get stuck.
Gotcha 5: For people who are familiar with CF templates , they know that in CF we can define roles , policies , associate policies with roles and then associate those roles to other resources like lambda functions or kinesis streams. All the same can be done in SAM , but the Serverless repository will not accept all SAM policies . As of now AWS has a list of policy templates that are only supported in AWS Serverless repository. You can not create custom roles and use them . Even though the SAM template will validate , and you can deploy them to your account using the SAM deploy command , but those SAM templates will not be accepted by the Serverless application repository. More on this can be found here : https://docs.aws.amazon.com/serverlessrepo/latest/devguide/using-aws-sam.html#serverlessrepo-policy-templates
Coding is done , can i test locally?
AWS SAM Local can be used to test locally . I have personally not tinkered with it yet , though I hope to one day . From what I understood , it spawns up a docker instance with your code and you can play with it .
However , I do unit tests via Visual Studio . At the end of the day , even though the code you are writing is for a lambda function , it is also a function within a class , and as such can be tested also . However , the difficult thing is to mock the S3 Events or the API Request events . AWS SDK for .NET has classes that you can use to create sample events and then call your lambda function code with these test events and a test execution context.
Check my code in the GitHub repo mentioned at the top of this blog . I have a separate unit test folder which contains a single unit test , that mocks a S3 event and calls the lambda function with the test event to validate that all calls can be done properly.
Gotcha 6 : When the actual lambda function runs in the AWS ecosystem it will run with the privileges of the role that it is associated with . However , what happens when you are unit testing . What privileges will your code have or not have , will it be able to make the AWS API calls , or will they fail?
The way to proceed in this case is to use AWS profiles . More info on this here - https://docs.aws.amazon.com/cli/latest/userguide/cli-multiple-profiles.html. You can create on or more profiles you want and choose which one to use when you are in Visual Studio . If you are in a Microsoft shop , download the AWS Toolkit for VS 2017 , configure your profiles and choose the one you want . Also , profiles can be set via code like this :
Amazon.Runtime.AWSCredentials credentials = new Amazon.Runtime.StoredProfileAWSCredentials("[PUT YOUR PROFILE NAME HERE]");
More info on this can be found here : https://aws.amazon.com/blogs/developer/referencing-credentials-using-profiles/
Everything works fine , how do I deploy to my AWS setup?
To deploy to AWS , simply use a few basic commands . Go to your working directory where the source code is and type the following:
Command 1-
sam validate --template template.json --profile Hackathon
Here template.json can be any file (json/yaml) where your SAM template is defined , and Hackathon is the aws profile that corresponds to the AWS account you want to use. This command will tell you whether the SAM template is valid or not . Always do this before the actual deployment.
Gotcha 7: Don't forget to mention the profile . If no profile is mentioned , it will consider the Default profile . This might not be what to want , or maybe some other AWS account .
Command 2:
aws s3 mb s3://aws-sam-test-1 --region ap-south-1 --profile Hackathon
This is simply creating a bucket . This is the bucket where our whole app details (code) will be zipped and stored . When , in a future step we will deploy our app to AWS , the CloudFormation engine will create all the required resources and populate the AWS Lambda functions from the zip in this bucket .
Gotcha 8: Bucket names need to be unique across all of AWS (all regions , all users , everything)
Command 3:
sam package --profile Hackathon --template-file template.json --output-template-file serverless-output.yaml --s3-bucket aws-sam-test-1 --force-upload
This command will package the whole app into a zip and upload it into the s3 bucket mentioned , and also create a output SAM template file serverless-output.yaml that will reference the code from the s3 bucket zip .
Command 4:
sam deploy --profile Hackathon --template-file serverless-output.yaml --stack-name aws-sam-trial-1 --capabilities CAPABILITY_IAM --region ap-south-1
This basically calls CloudFormation deploy command to actually create a CF stack with the name specified above and creates all the resources mentioned in the SAM template file. While this command is running , you can actually navigate to AWS console , and check CloudFormation . You can see that your stack is getting created , and all its associated resources are getting created one by one.
Gotcha 9 : Your AWS SAM template will most of the time ,have some sorts of parameters defined to make your template dynamic and your infrastructure setup configurable . These parameters can usually be configured by users when the stack is deployed . However , when SAM deploy command runs it does not ask for input from the user , it simply deploys the stack with the default values for the parameters mentioned in the SAM template file . So , if you have a parameter for the bucket name you want to create , be sure that it is unique , otherwise the stack creation will fail and it will rollback.
Gotcha 10: If stack creation fails for some reason , then the stack transitions to a ROLLBACK_COMPLETE . If at that point , you make some changes to your template.json and run SAM deploy again , it will fail . So , you need to delete your stack first , and then proceed in that case.
If all of the above commands run without any errors , it means that all your AWS resources have been created and configured as per your template file , the lambda code has also been linked to the code zip mentioned earlier. At this point , you can go into your AWS console , and test and check whether everything is okay or not.
Gotcha 11: It is highly unlikely that you will get all your code right the first time . You will make multiple changes to your actual code(not the template.json , pure C# or java code) , and will want to push those changes into the AWS ecosystem . The SAM deploy command will fail in this case , because , since there are no stack changes in terms of infrastructure , CF thinks there is nothing to deploy . In such cases it is better to use separate commands or say the AWS Toolkit for VS to deploy . You can simply select your project and deploy it to to AWS with a single click . This will simply deploy your latest code changes to the lambda function , and nothing else.
My App rocks . How to publish it to the SAM repo now?
Publishing applications to the SAM repo is extremely simple . Just follow the steps mentioned here - https://docs.aws.amazon.com/serverlessrepo/latest/devguide/serverless-app-publishing-applications.html
However there are a few things to keep in mind .
If you want to make your application public , it needs to have an open-source license and the code needs to be pushed into some sort of a public repository like git , and the link referenced in the app details. This is needed when the app is public , and not in all cases.
During publishing the app , you are asked to select the SAM template file , do not select the template.json file that you wrote yourself . You need to select the serverless-output.yaml file created as an output of Command 3 in the last section. This is because template.json references your local code , while serverless-output.yaml references the actual code Uri of the S3 bucket where the packaged code resides.
Be sure to add the bucket policy granting the SAM repository service access to your code . This is required so that when an user tries to deploy your app in their AWS account , the SAM repository service can setup the CF stack in your account by referencing the packaged code that is stored in your bucket. This is also mentioned here- https://docs.aws.amazon.com/serverlessrepo/latest/devguide/serverless-app-publishing-applications.html
My app is published . Now what?
Now go to https://serverlessrepo.aws.amazon.com/applications and search for your app by the app name/ author name . Click on deploy next to the app and deploy it in some other region to test it out . You will simply need to configure the parameter names , and let CloudFormation do its magic . CF will spawn up the resources one by one , and you are ready to go .
Where to go from here?
The future is full of endless possibilities . AWS SAM templates provide a whole new dimension to the world of Serverless applications .Just like CF helped you maintain huge infrastructure setups in a single CF file , that could be version-controlled , SAM templates lets you do the same for Serverless apps .
Replicating your Serverless components that make up your app in a different account or in a separate region will not take a day , rather will be a matter of 1-5 minutes. From disaster recovery standpoint , this is a huge plus also.
Also , I believe that AWS will develop the Serverless repository into a marketplace of sorts , full of useful apps that other users can leverage as a starting point for their next big thing or a crucial component for their complex , multi-layered app.
As of today , there are 335 apps in the serverless repository . Go ahead , make one . And don't forget to take a look at the simple app that I developed . Search "FileLogTracker" or "turja" and give it a go.
Please feel free to reach out to me personally at my email or drop me a message in LinkedIn . Would love to know what you guys are up to nowadays.
Thanks.
Top comments (0)