Building a DevSecOps Pipeline to sell secured Products as AMI on AWS Marketplace

• DevSecOps Highlights
• Ready to take your product to AWS Marketplace?
• Guidelines for Building an AMI
• Use-case
  • Source
  • Build
  • Deploy
  • Build AMI
• Embrace security in DevSecOps pipeline
• Static Code Analysis
  • Important Selection Criteria
  • Tools
• Security Scanning
  • Threat Investigation
  • Vulnerability Assessment
• Resource
• Conclusion

DevSecOps aims to embed security in every part of the development process for the early detection and continuous mitigation of risks. This blog discusses best practices for taking a product to sell on AWS Marketplace. It looks at a CI/CD Pipeline implemented using AWS Developer tools to create and configure an EC2 instance and then bake an AMI using a Lambda function.
The blog focuses on three important aspects of incorporating safety and security measures into the CI/CD pipeline:
• Static Code Analysis for early identification of vulnerabilities in static code.
• Checking common vulnerabilities and exposures with Amazon Inspector.
• Threat Investigation using Amazon GuardDuty to monitor and detect unusual API calls or potentially unauthorized deployments.

DevSecOps Highlights

DevOps is a holistic approach integrating cultural philosophies, practices, and tools to enhance an organization's capacity to deliver applications and services swiftly. It serves as a bridge between Development and Operations, aiming to streamline the software development lifecycle.

Here is the DevOps principles

If you want to make the most of the agility and responsiveness that DevOps offers, IT security must become an integral part of every single stage of the development process.

DevSecOps is an extension of the DevOps philosophy, integrating security practices into every phase of the software development lifecycle. It emphasizes collaboration between development, operations, and security teams to ensure that security is prioritized and integrated from the beginning. This approach aims to build secure software and infrastructure, reduce vulnerabilities, and enhance overall security posture.

DevSecOps is to emphasize security as a shared responsibility that must be ingrained in every single stage of the development process.

In a nutshell, DevOps is for Seamless Application Delivery whereas DevSecOps is for Secured Seamless Application Delivery.

Ready to take your product to AWS Marketplace?

If you're a product company developing innovative solutions and looking to reach AWS cloud users, delivering your products through the AWS Marketplace can be a strategic move. One effective method to make your products accessible to buyers is by using Amazon Machine Images (AMIs). These AMIs contain all the necessary information to launch an Amazon Elastic Compute Cloud (Amazon EC2) instance pre-configured with your product, making it easy for customers to deploy and use your offering.
You may refer to the AWS Marketplace User Guide for more details on how you can get started as a seller, define pricing policy etc.

Guidelines for Building an AMI

(excerpt from Best practices for building AMIs)

• Ensure that your AMI meets all AWS Marketplace policies, including disabling root login.
• Create your AMI in the US East (N. Virginia) Region.
• Create products from existing, well-maintained AMIs backed by Amazon Elastic Block Store (Amazon EBS) with a clearly defined lifecycle provided by trusted, reputable sources such as AWS Marketplace.
• Build AMIs using the most up-to-date operating systems, packages, and software.
• Ensure that all AMIs must start with a public AMI that uses hardware virtual machine (HVM) virtualization and 64-bit architecture.
• Develop a repeatable process for building, updating, and republishing AMIs.
• Use a consistent operating system (OS) user name across all versions and products. AWS recommends ec2-user.
• Configure a running instance from your final AMI to the end-user experience you want, and test all installation methods, features, and performance before submission to AWS Marketplace.
• Check port settings as follows:
  • Linux-based AMIs – Ensure that a valid SSH port is open. The default SSH port is 22.
  • Windows-based AMIs – Ensure that an RDP port is open. The default RDP port is 3389. Also, the WinRM port (5985 by default) must be open to 10.0.0.0/16 and 10.2.0.0/16.

This blog will focus on developing a repeatable process for building, updating, and republishing AMIs and also bring in security elements at every stage of the pipeline as a key element.

Use-case

Imagine your product as a Java Web Application, with two separate teams handling the application and infrastructure code. They use CloudFormation templates and commit to different repositories - the WebApp Repo and the Infra Repo. Let's explore the DevSecOps pipeline implemented with AWS Developer Tools for this Java Web Application, focusing on the steps to securely and reliably bake the AMI.
In the scenario outlined:

• The Web Application and Infrastructure code reside in AWS CodeCommit.
• AWS CodeBuild is employed to compile the Web Application code and produce the artifact.
• The artifact is stored in an Amazon S3 bucket.
• AWS CodeDeploy is set up to deploy the code to the existing EC2 instance and subsequently create a Golden AMI using a Lambda function.
• AWS CodePipeline is utilized to model, visualize, and automate the steps from source to build to deploy to creating the Golden AMI.

There are 3 main components of CodePipeline

• Stages – The pipeline consists of multiple stages. In your organization, you can have multiple stages such as source, build, test, staging and production.
• Action – Each stage consists of a series of actions which specify what must be done on the artifact. Each stage can only perform one revision of the source code at a time. All of the artifacts are stored in an S3 bucket. For example, the code which needs to be built will be copied to the S3 bucket for the pipeline. This becomes the input artifact.
• Transition – A transition is an action which is performed when one stage is moved to the next in the pipeline. An approval action can also be put in place so that manual approval is required before it goes to the next stage.

Source

• A source stage in a pipeline typically watches a git repository and branch for changes.
• When a change is detected in the source (meaning a developer has merged a feature branch for example) a pipeline is executed.
• The action in the source stage will zip up all of the code in that repositories branch, and place it into a temporary S3 bucket.
• The use-case here has two actions in-parallel at source stage to pull the source code from WebApp and Infra repo.
• Then the next stage is invoked with a pointer to the zip file so it can perform a build.

Build

• A build stage in a pipeline 'compiles' or prepares the application’s source code into a deployable format.
• The application in this use-case simple Java Web application. This is built using a series of commands and artifact is uploaded to S3 bucket.
• Infra code being CloudFormation template, no action is needed at build stage.

Deploy

• A deploy stage in a pipeline ‘creates’ necessary infrastructure and / or ‘deploys’ the artifact on to the newly created or existing AWS resource.
• In this use-case there are two sequential actions
  • First action creates an EC2 instances or updates if exists
  • Second action deploys the web application into the existing EC2
• After deploy, the application detailed test suites like functional validation, security testing, performance testing etc can be executed.

Build AMI

• The above pipeline has an approval action so that after all the validations when the quality and security gates are passed, the release manager can approve AMI creation.
• The Lambda function builds the AMI, which can be published onto AWS Marketplace or it can also be used in dev, test, pre-prod and prod workloads.

Embrace security in DevSecOps pipeline

Fair enough... Is our AMI Secured for production workload or to be a candidate in AWS Marketplace?

A big No.

So how can we secure that…
AWS Marketplace maintains a set of policies for all products and offerings on AWS Marketplace to promote a safe, secure, and trustworthy platform for AWS customers.
The foremost perquisites AMIs must not contain any known vulnerabilities, malware, or viruses as detected by AWS Security.

Here you go with 3 important aspects of inculcating the safety and security measures into our CI / CD pipeline viz
• Static Code Analysis for early identification of vulnerabilities in non-running static code.
• Vulnerability Assessment with Amazon Inspector to set constant outlook for CVE (common vulnerabilities and exposures)
• Threat Investigation using Amazon GuardDuty to monitor & detect unusual API calls or potentially unauthorized deployments

Static Code Analysis

• Source code analysis tools, also referred to as Static Application Security Testing (SAST) Tools, are designed to analyze source code and/or compiled versions of code to help find security flaws. SAST addresses weaknesses in source code that might lead to vulnerabilities.
• Static analysis is commonly used to comply with coding guidelines — such as MISRA or ISO 26262
• Static analysis identifies defects before you run a program - Dynamic analysis identifies defects after you run a program (e.g., during unit testing).

There are plenty of tools available in the market and how would you choose the right one?

Important Selection Criteria

• Requirement: The tool must support your language, but not usually a key factor once it does.
• Types of Vulnerabilities it can detect (The OWASP Top Ten?) (more?)
• Does it require a fully buildable set of sources?
• Can it run against binaries instead of source?
• Can it be integrated into the developer’s IDE?
• License cost for the tool. (Some are sold per user, per org, per app, and per line of code analyzed. Consulting licenses are frequently different than end-user licenses.)
• Does it support Object-oriented programming (OOP)?

Tools

OpenSource: FindBugs, SpotBugs, PMD, DevBug (php)
Commercial: Fortify, Veracode, CodeSonar, CodeSec

Security Scanning

• In a DevSecOps pipeline, Amazon GuardDuty and Amazon Inspector play crucial roles in ensuring the security of your infrastructure and applications. Both services need to be enabled in your account to pick up events.
• When you use GuardDuty or Inspector with an AWS organization, the management account of that organization can designate any account within the organization as the delegated GuardDuty or Inspector administrator account.
  • The tasks executed by the delegated Inspector administrator account include activating or deactivating scans for member accounts, viewing aggregated finding data from the entire organization, and creating and managing suppression rules.
  • Similarly, the delegated GuardDuty administrator account also has the permission to enable and manage.

Threat Investigation

1. Amazon GuardDuty is a threat detection service that continuously monitors malicious activity and unauthorized behaviour in your AWS account.
2. Example: In a DevSecOps pipeline, GuardDuty can be used to detect unusual API calls, potentially compromised instances, or unauthorized deployments. For instance, GuardDuty might alert you if an EC2 instance in your environment is communicating with known malicious IP addresses.

1. When you enable Amazon GuardDuty for the first time or enable a protection type within GuardDuty, GuardDuty starts processing the corresponding foundational data sources, such as VPC flow logs, DNS logs, and AWS CloudTrail event and management logs within your AWS environment. GuardDuty uses these data sources to process a stream of events and then analyzes these events to identify potential security threats and generates findings in your account.
2. In addition to log data sources, GuardDuty can use additional data from other AWS services in your AWS environment to monitor and analyze for potential security threats.

Vulnerability Assessment

• Amazon Inspector is a vulnerability assessment service that helps improve the security and compliance of applications deployed on AWS by running security assessments against them.
• Example: In your pipeline, Inspector can be used to perform security assessments on your EC2 instances or other resources. For example, the Inspector can check for common vulnerabilities and exposures (CVEs) or security best practices, providing you with detailed findings and recommendations for remediation. When the Inspector identifies a CVE in your environment, it will flag it as a security finding, allowing you to take action to mitigate the vulnerability.

By integrating Amazon GuardDuty and Amazon Inspector into your DevSecOps pipeline, you can enhance the security posture of your applications and infrastructure, ensuring that potential threats and vulnerabilities are identified and addressed early in the development process.

Resources

• AWS Deveveloper Tools
• Amazon GuardDuty
• Amazon Inspector
• AMI-based products on AWS Marketplace

Conclusion

In conclusion, implementing a DevSecOps pipeline is crucial for selling your AMI on the AWS Marketplace. By embedding security practices into every stage of your development process, you can ensure that your AMI is secure, reliable, and compliant with industry standards. With the right tools and practices in place, you can build trust with your customers and differentiate your product in the marketplace. Embrace DevSecOps to unlock new opportunities and drive success in the cloud.