How to Automate AWS Snapshots Using the Life Cycle Policy

What Are AWS EBS Snapshots?

EBS snapshots are point-in-time copies of EBS volumes stored on Amazon S3. These snapshots are incremental, meaning each snapshot only stores data which has changed since the last snapshot, reducing necessary storage space. Snapshots are chained together into a sort of version history and the data is consolidated by Amazon, meaning individual snapshots can be deleted without affecting recovery capabilities. AWS snapshots can be accessed from the AWS portal to restore EBS volumes in case of loss but are not downloadable or exportable.

How Snapshots Were Created Before Native Automation

Initially, there was no automated process for snapshot creation and they had to either be taken manually, created using API calls, or a script or had to be run with Cron, a time-based scheduler, for each instance. The script that you created, or chose to use, had to be run through AWS CLI and was reliant on AWS not altering its code structure or denying third-party access. The introduction of Lambda simplified this process by allowing you to run code on AWS administrated instead of self-managed servers, but you still had to provide the code to be run.

Automation Process with Data Lifecycle Manager (DLM)

Amazon’s DLM allows you to create and manage your AWS snapshots through native automation directed by policies you create. In order to use DLM, you must first tag your resources and then create a backup policy. Policies define which resources are managed, which tags are targeted, the schedule of the backup, and how many snapshots are retained. Currently, EBS volumes and EC2 instances are the available resources. Tags are defined by you and should be created with careful consideration. Multiple tags can be added to each EBS volume to allow targeting by multiple policies and multiple policies can target the same tags. The policies you define will run according to their schedule, with up to an hour variance, until you deactivate or modify them.

Some things to be aware of:

Each AWS account is limited to 100 lifecycle policies per region, 50 tags per resource, and one schedule per policy. Once a policy is changed, it will no longer manage retained snapshots or target tags to which it no longer applies, so it is up to you to manage these manually, through the creation of a new policy, or with custom scripts. DLM is a per-region service so policies will only apply to the region for which they are created. There is no built-in option for copying snapshots from one region to another, such as for the purpose of disaster recovery, so you may still need to use custom scripts or do this process manually. DLM is a free service but you are still charged for storage used by snapshots and instance load and I/O usage will increase while snapshots are created.

Using the Lifecycle Policy to Automate AWS Snapshots

Life cycle policies can be created and managed through the EC2 console, the CLI, or APIs. All three methods require the creation of an IAM role to grant permissions necessary for DLM to manage your snapshots.

Permissions for Amazon DLM

The IAM role is created automatically, as AWSDataLifecycleManagerDefaultRole, the first time you create a policy through the AWS management console or you can create the role manually. If you choose the manual route you will need to assign permissions and add a trust relationship. You will also need to grant permissions for DLM use to your IAM users.

Using the EC2 Console

Using the EC2 console is probably the simplest user interface available. To create lifecycle policies navigate to the lifecycle manager in the EBS and provide the information outlined below:

• Description—description of the policy
• Target with tags—tags identifying volumes or instances to be backed up
• Schedule Name—name for the backup schedule
• Create snapshots every n Hours—time between repetitions, supported values are 2, 3, 4, 6, 8, 12, and 24
• Snapshot creation start time hh:mm UTC—time policy is scheduled to start
• Retention rule—set number of snapshots to be retained in the range of 1-1000
• Copy tags—Copy all user-defined tags on a source volume to snapshots of the volume created by this policy.
• Tag created snapshots—policy id and schedule name tags are automatically added but you can add additional tags
• IAM role—either the default created by DLM or your custom creation
• Policy status after creation—enable to run, disable to not

After a policy has been created you can view the policy in the details tab and modify settings or delete the policy through actions.

Using the CLI

If you are already familiar with the CLI, you can now use this method to work with lifecycle policies through AWS specified commands. To create policies, use create-lifecycle-policy and its appropriate parameters. An example of this can be seen below and AWS has produced a whitepaper with more detail.

aws dlm create-lifecycle-policy --description "My first policy" --state ENABLED --execution-role-arn arn:aws:iam::12345678910:role/AWSDataLifecycleManagerDefaultRole --policy-details file://policyDetails.json

After a policy has been created, use the following commands for management:

• To display—get-lifecycle-policy --policy-id policy-xxxxxxxxxxxxxxxxx
• To modify—update-lifecycle-policy (with changes to parameters used in creation as necessary)
• To delete—delete-lifecycle-policy --policy-id policy-xxxxxxxxxxxxxxxxx ### Other options If using the EC2 console or the CLI doesn’t suit you, policies can also be created and managed through either the Amazon DLM query API or by using AWS SDKs to access an API appropriate to your needs. ## Conclusion The introduction of native automation for AWS snapshot management makes the process much simpler and hopefully more reliable but it is important to understand the limitations of DML to make the benefits work for you. This automation can save you time and effort if used correctly but must still be monitored, particularly if there are multiple users with the ability to create or manage policies. Be sure to consider the cost of running and storing all your newly created backups when determining your automation policies so you aren’t trading time value for that of resources. If you need extended capabilities that go beyond what is currently offered natively, you can try a lifecycle management software.