DEV Community

Arun Kumar for AWS Community Builders

Posted on

Spot Instance Scenarios

Scenario

  • Instance stopped by AWS due to Insufficient Capacity but not started automatically by AWS when Capacity is available again.
  • No issue when user start the instance manually.

Reason

  • Service role is not able to access the KMS key that is cross account and the instance is using this KMS for their volume.

Troubleshooting

  • Look at the configuration changes and you will see “Client error on launch”.

1

  • Check CloudTrail logs and you can see the Access Denied error on KMS.

2

Solution

On the account whereby the instance is launched, run the following command to grant the KMS permission to service role.

aws kms create-grant — region <region> –key-id <arn of the KMS> — grantee-principal <arn of the Spot Service Role> — operations “Decrypt” “Encrypt” “GenerateDataKey” “GenerateDataKeyWithoutPlaintext” “CreateGrant” “DescribeKey” “ReEncryptFrom” “ReEncryptTo”
Enter fullscreen mode Exit fullscreen mode

Example:

aws kms create-grant — region ap-southeast-1 — key-id arn:aws:kms:ap-southeast-1:123456789:key/479d6414-e442–4873–9b10-123dwdas343 — grantee-principal arn:aws:iam::987654321:role/aws-service-role/spot.amazonaws.com/AWSServiceRoleForEC2Spot — operations “Decrypt” “Encrypt” “GenerateDataKey” “GenerateDataKeyWithoutPlaintext” “CreateGrant” “DescribeKey” “ReEncryptFrom” “ReEncryptTo”
Enter fullscreen mode Exit fullscreen mode

Result:

3

Note:
Monitor the situation to ensure that instance starts up whenever spot instance is reclaimed by AWS due to insufficient capacity.

Oldest comments (0)