How to set up AWS Session Manager logging cross account

Setting up and managing access to your EC2 instances can be challenging. There are a lot of things you need to consider. How do you rotate your keys? Who logged in using that key? How do you limit the risk of thread actors?
And what did they do? What if you could log the complete session in a different AWS Account? And encrypt it using a customer managed KMS key? That would give you complete visibility and auditability on your shell sessions.
In this blog post I will address those questions using AWS Session Manager.

What is the AWS Session Manager?

Taken from the documentation:

Session Manager is a fully managed AWS Systems Manager capability. With Session Manager, you can manage your Amazon Elastic Compute Cloud (Amazon EC2) instances, edge devices, and on-premises servers and virtual machines (VMs).

So, what does that mean? It means that you can open an SSH shell using the AWS Console in your browser. Select the EC2 instance. Click the connect button and a shell session loads in the browser. You can also use CLI commands to do this.
The interesting part is that you grant access based on your IAM credentials. So you do not have to share private SSH keys across the team. No need to rotate them. No need to rotate them when a team member leaves. Because you don’t need a private key at all.
But this solution comes with another advantage. You do not need to open port 22 to connect. In fact, you do not need to open any inbound ports.

The SSM Agent comes pre-installed with the Amazon Linux 2 AMI.

Instead, it uses an SSM Agent. And this agent will register itself to the AWS Systems Manager. It will use the attached instance profile to do that. From that point on when you request a shell from the Session Manager. The agent will react on this response and establish the connection.

What are the other advantages?

Yes, there is more! Session Manager has some extra features!
You can track session activity using Amazon EventBridge. This means that you can react on these events. This creates the possibility to proactively react on these events. You could for instance: terminate the session if no access is allowed.
And that brings me to the main topic of this blog post. What if you suspect that some commands where executed on a server using a shell session? How do you prove it happened or did not happen?

Logging your session activity

The Session Manager gives you the ability to log the session to S3. Setting this up is described on the documentation pages. But, if you want to use a centralized logging account. And you need to use a customer managed KMS key, the documentation is not that helpful.

Depending on your use-case, you might want to limit the access. You can do this per account. Or within an AWS Organization.
For this example, I will limit the scope within the same organization.

First, you will need to create a KMS key. This key will be used to encrypt the data stored on S3. S3 needs to be able to use the KMS key add following statements to the key policy:

{
    "Effect": "Allow",
    "Principal": { "AWS": "*" },
    "Action": [ "kms:Decrypt", "kms:Encrypt", "kms:ReEncrypt*", "kms:GenerateDataKey*" ],
    "Resource": "*",
    "Condition": {
        "StringEquals": {
            "kms:ViaService": "s3.eu-central-1.amazonaws.com",
            "aws:PrincipalOrgID": "o-xxxxxxx"
        }
    }
}

Next, we allow kms:Decrypt and kms:GenerateDataKey within the organization. These actions are needed so that the Session Manager can encrypt the session. And it can confirm that it can log to the given S3 bucket.

{
    "Effect": "Allow",
    "Principal": {
        "AWS": "*"
    },
    "Action": [
        "kms:Decrypt",
        "kms:GenerateDataKey"
    ],
    "Resource": "*",
    "Condition": {
        "StringEquals": {
            "aws:PrincipalOrgID": "o-xxxxxxx"
        }
    }
}

We then need to create a S3 Bucket. You need to enable default encryption using the KMS key you created. You need to grant other accounts to write to this bucket. So, you need a bucket policy. Here is an example of a bucket policy that you need:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": "s3:GetEncryptionConfiguration",
            "Resource": "arn:aws:s3:::my-bucket-name",
            "Condition": {
                "StringEquals": {
                    "aws:PrincipalOrgID": "o-xxxxxxx"
                }
            }
        },
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": "arn:aws:s3:::my-bucket-name/*",
            "Condition": {
                "StringEquals": {
                    "aws:PrincipalOrgID": "o-xxxxxxx"
                }
            }
        }
    ]
}

Alright, the logging account has been prepared. It can now receive logs from all accounts within your organization. Now it is time to set up the account that will use the Session Manager.
Create a JSON file locally with the following content:

Replace the name of the bucket and the KMS key! You could use the account id as a prefix.

{
    "schemaVersion": "1.0",
    "description": "Document to hold regional settings for Session Manager",
    "sessionType": "Standard_Stream",
    "inputs": {
        "s3BucketName": "[NAME OF THE BUCKET]",
        "s3KeyPrefix": "",
        "s3EncryptionEnabled": true,
        "cloudWatchLogGroupName": "",
        "cloudWatchEncryptionEnabled": false,
        "cloudWatchStreamingEnabled": false,
        "kmsKeyId": "[FULL ARN OF THE KMS KEY]",
        "runAsEnabled": true,
        "runAsDefaultUser": "",
        "idleSessionTimeout": "20",
        "maxSessionDuration": "",
        "shellProfile": {"windows": "", "linux": ""}
    }
}

Apply this SSM Document to the account using the Session Manager:

aws ssm update-document \
    --name "SSM-SessionManagerRunShell" \
    --content "file://SessionManagerRunShell.json" \
    --document-version "\$LATEST"

At this point everything is inplace to make this work. Except for the IAM permissions on the instance profile. So to get this to work you will need the following permissions:

• Attach the arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore managed policy.
• Allow the "s3:GetEncryptionConfiguration" action on the arn:aws:s3:::[NAME OF THE BUCKET] resource.
• Allow the "s3:PutObject" and "s3:PutObjectAcl" action on the arn:aws:s3:::[NAME OF THE BUCKET]/* resource.
• Allow the "kms:Decrypt" and "kms:GenerateDataKey" action on the [FULL ARN OF THE KMS KEY] resource.

With these permissions in place you are able to start an SSH shell.

Conclusion

When you use Session Manager you not only remove the need to expose port 22. Or don’t need to rotate keys when a team-mate leaves. You get a lot more! Your SSH session is encrypted using a customer encrypted KMS key. That you manage and maintain!
Next to that, once the session is closed the complete session is stored on S3. And again encrypted using a customer encrypted KMS key. That you manage and maintain! This allows you to archive shell sessions for later use. Giving you full auditability and improving your security posture.
And because you don’t need to bother your team-mates with new keys for this week. Easier for your team to use as well.