In our last post, we learned about LitmusChaos, how to get started with a simple experiment, IAM Role for Service Account and more. In this blog post, we are going to see how you can run chaos experiments using LitmusChaos on AWS, if you have two different AWS accounts. Let us get started.
The deployment to production happens at a fast pace in a carefully constructed value stream. Primarily, we (performance engineers) work in a non-production environment. Though we run experiments in production, lower environments such as dev, QA, and performance environments can be your playground for your chaos experiments. Often, most of the organization will have more than one account, just to isolate the engineering units, environments, and more, just to avoid stepping on each other's toe.
As a chaos engineer, you may need to run experiments on the applications which are hosted in different AWS accounts.
Consider a simple application hosted on an EC2 instance in the AWS Account marketing-team
. LitmusChaos on AWS EKS runs on a performance-team
AWS account. In this case, we need to enable cross - AWS account access.
Prerequisites
The following are the prerequisites for implementing this setup:
- Two AWS Accounts with IAM privileges
- LitmusChaos on AWS Account A
- Sample application on AWS Account B
- eksctl
- kubectl
Creating two AWS accounts might be tricky as you need two different credit cards to sign up. I have borrowed an account from my friend, just to demonstrate this implementation.
Also, make sure you have valid privileges to play around with the IAM, Policy, Trust Relationships and more.
Let us name our AWS accounts to understand better. You are a Chaos Engineer who provides Chaos-as-a-Service. You run LitmusChaos on your AWS account called Chaos-Account
.
A customer has approached you to perform chaos experiments on their application. Let us call their AWS account a Customer-Account
.
On Chaos-Account
On Chaos-Account
make sure you have LitmusChaos is up and running successfully and able to run experiments either using Self-Agent or external agent.
Create a new IAM Role using the below policy and trust relationship.
Before creating IAM Role, make sure you have the valid OIDC provider URL.
To validate OIDC URL, enter the below command
aws iam list-open-id-connect-providers
The above command must return the OIDC entry e.g. oidc.eks.us-east-2.amazonaws.com/id/F25CB34E2ACAA3CE669BA4E0XXXXXXXX"
If not, create an OIDC using the below command. In this case, our cluster name is chaos-cluster
.
eksctl utils associate-iam-oidc-provider --cluster chaos-account --approve
Create a policy Chaos-Account-Policy
using the below JSON. Please follow the Zero Trust framework while creating a policy and trust relationship. I am just being playful as it is my demo account.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "ec2:*",
"Resource": [
"*"
]
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": "sts:*",
"Resource": [
"*"
]
}
]
}
Create a new IAM role called Chaos-Account-Role
and attach the above policy and add the below trust relationship. Make sure you replace the OIDC ID, account ID, region etc.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::<account_id>:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/F25CB34E2ACAA3CE669BA4EXXXXXXX"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"oidc.eks.us-east-1.amazonaws.com/id/F25CB34E2ACAA3CE669BA4EXXXXXXX:aud": "sts.amazonaws.com"
}
}
}
]
}
On Customer-Account
Login into your Customer-Account
or ask them to create a new policy called Customer-Account-Policy
using the below JSON. Again, please follow the Zero Trust policy.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"ec2:*",
"sts:*"
],
"Resource": "*"
}
]
}
Before creating a new role, let us create Identity provider in this account. Go to IAM, under Access management, click Identity providers.
Then, click Add provider button.
Select OpenID Connect
In Provider URL text box, enter the Chaos-Account
identity provider URL, and then click Get thumbprint button.
In Audience text box, enter sts.amazonaws.com
.
Click on Add provider button.
Create a new role by selecting Web identity as shown below and select your Identity provider and Audience [i.e. Chaos-Account
] and then click Next
In the next page, select the Customer-Account-Policy
then create a role Customer-Account-Role
.
Copy the Customer-Account-Role
ARN to the clipboard as we need it to annotate in the LitmusChaos
service account.
Annotation
Use the below command to annotate your LitmusChaos service account in the default namespace litmus
.
kubectl annotate serviceaccount -n litmus litmus-admin eks.amazonaws.com/role-arn=arn:aws:iam::<customer-account-id>:role/Customer-Account-Role --overwrite
To validate the annotation, use the below command:
kubectl describe sa -n litmus litmus-admin
Chaos Experiment
Now it is time to create a chaos experiment to stop the EC2 instance in Customer-Account
.
Copy the instance ID and the region which we need to feed it in the experiment YAML.
Create a ec2-terminate-by-id
experiment and configure the instance ID and the region. Make sure you remove the secrets from the YAML and schedule it.
If all is well, it should stop and start the EC2 instance in your customer account.
Conclusion
As you learned, cross - account implementation is easy and straightforward. It looks complex, but once you understand the OIDC concept, how role, policy, and trust relationships works, it is quick to implement. The above steps can be automated by leveraging AWS CLI and eksctl.
Top comments (0)