AWS Systems Manager Session Manager implementation

Session Manager Overview

• Session Manager is a fully managed AWS Systems Manager capability that lets you manage your Amazon EC2 instances through an interactive one-click browser-based shell or through the AWS CLI.

• Session Manager provides secure and auditable instance management without the need to open inbound ports, maintain bastion hosts, or manage SSH keys.

• Session Manager also makes it easy to comply with corporate policies that require controlled access to instances, strict security practices, and fully auditable logs with instance access details, while still providing end users with simple one-click cross-platform access to your Amazon EC2 instances.

Architecture

How Session Manager works

• Admin users are authenticated through IAM roles and policies.
• If authentication is successful, SSM session manager is accessible by the AWS Management Console or AWS CLI (requires session manager CLI plugin).
• An agent running on each EC2 instance connects to then System Manager endpoints and executes the command over the instances.
• Any action performed over session manager is logged on CloudTrail. Optional: Session outputs can be forwarded to CloudWatch logs and/or S3 buckets (Exception port-forwarding action logs will not be Pushed to cloudwatch logs and s3 bucket).

Main benefits

• Systems Manager components are reliable and highly available (AWS Console, AWS CLI, SSM endpoints).
• Session manager can leverage multi-factor authentication (by enforcing IAM policies).
• Centralization of access to EC2 instances and granular control over who can start sessions on specific instances.
• Open inbound SSH connection port for EC2 instances is no longer needed.
• Jump or Bastion host can be removed to improve security and save cost.
• Deploy and manage ssh-keys for EC2 instances is not necessary.
• Sessions are logged based on the IAM user. Logs include the executed command, outputs, time when the command was executed and more.
• Command outputs can be stored in S3.
• Command outputs can be forwarded to CloudWatch logs and generate alerts as response for undesired behavior.
• Full support for logging and auditing features in AWS (CloudTrail, S3, CloudWatch logs). (Exception port-forwarding action logs will not be Pushed to cloudwatch logs and s3 bucket)

Requirements

SSM agent installation

SSM agent should be installed in every Ec2 instances or on-premise machine with Administrative access.

Internet Access

SSM agent needs communication with the AWS API, this communication uses standard HTTPS ports. Because the agent always starts the communication, allow any inbound rules is not necessary.

System Manager IAM setup

SSM requires an instance profile role that should be associated with each EC2 instances. In addition, an IAM service role is necessary for hybrid environment.

Limitations

a) Transferring files is not possible by default with AWS Session Manager.

b) For Windows, RDP is not supported (port forwarding can be used instead) and “Run As” capability is not available.

c) Session manager is compatible with on premise system but requires the advanced on-premises instance tier (payment required).

d) Session manager is not a native ssh service, most of the tools that can work with ssh are not supported.

IAM policy example for Session Manager

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ssm:StartSession"
            ],
            "Resource": "arn:aws:ec2:*:*:instance/*",
            "Condition": {
                "StringEquals": {
                    "ssm:resourceTag/tag_key":[
                        "tag_value"
                    ]    
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "ssm:GetConnectionStatus",
                "ssm:DescribeSessions",
                "ssm:DescribeInstanceProperties",
                "ec2:DescribeInstances"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ssm:TerminateSession"
            ],
            "Resource": [
                 "arn:aws:ssm:*:*:session/${aws:username}-*"
             ]
         }
    ]
}

You can set IAM EndUser and Administrator policies for Session Manager.

[https://docs.aws.amazon.com/systems-manager/latest/userguide/getting-started-restrict-access-quickstart.html]

You can also restrict access to specific instances individually or restrict based on tags, terminate only their specific sessions, allow full access to all sessions.

[https://docs.aws.amazon.com/systems-manager/latest/userguide/getting-started-restrict-access-examples.html]

Recommended settings

Enable “Run As” support for Linux instances

• By default, sessions are launched using the credentials of a system-generated ssm-user account that is created on a managed instance.

• You can instead launch sessions using the credentials of an operating system account. You can do this either by tagging an IAM user or role or by specifying an OS user name in Session Manager preferences.

• This setting allows start sessions using the credentials of a specified operation system user, instead of the default credential generated by the System Manager agent (ssm-user).

• When this option is enabled, Session Manager checks for access as follows:

a. If the user or role who is starting the session has been tagged with the key “SSMSessionRunAS”, Session Manager will check if the value for that key exist as an OS user inside the EC2 instance selected as target. If exists that user will be used, otherwise the connection will be rejected.

b. If the user or role who is starting the session has not been tagged with the key “SSMSesisonRunAS”, Session Manager will use the OS username specified in the AWS account’s Session Manager preferences.
Run As support always prevents sessions from being started using an ssm-user account on an instance.

[https://docs.aws.amazon.com/systems-manager/latest/userguide/session-preferences-run-as.html]

Managing Account permissions

When a version of SSM Agent that supports Session Manager starts on an instance, it creates a user account with root or administrator privileges called ssm-user.

If you want to prevent Session Manager users from running administrative commands on an instance, you can update its ssm-user permissions. You can also restore these permissions after they have been removed.

You can disable or enable the ssm-user account sudo permissions on Linux or Windows instances.

[https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-getting-started-ssm-user-permissions.html]

Configure Session Manager to use AWS KMS key encryption

Provides additional protection to the data transmitted between client machines and managed instances.

Configure Session Manager to create and send session history logs

• Logs can be stored on Simple Storage Service (Amazon S3) bucket and/or an Amazon CloudWatch Log group. The stored log data can be used for logging and auditing purposes or even to create security alerts.

Session Manager Settings

Session manager allows several integrations with security services on AWS, the most relevant ones are listed below:

CloudTrail

Any action performed over the session manager API is logged in CloudTrail. API actions such as StartSession can easily generate security alerts by SNS or trigger another action on AWS as the result of sessions activity (lambda functions or SSM run command are good examples).

Simple storage Service

As optional feature, Session Manager can store sessions outputs on S3 for further auditing. This option also allows run lambda functions that can be triggered every time when a session output is stored on a specific bucket. Lambda can analyses session outputs and perform any required action.

CloudWatch logs

Another service than can store session output logs for Session Manager is CloudWatch. This option is especially useful to create security alerts based on undesired behavior. By taking advantage of filters and metrics, CloudWatch logs can react and generate alerts if any user try to executes forbidden commands or performs any restricted action on the ssh/powershell sessions.

You can create Session Manager preferences for your account in the selected AWS Region via AWS Cli. Session manager preferences can be created using a JSON file and running a SSM document SSM-SessionManagerRunShell. Check out the steps in below.

[https://docs.aws.amazon.com/systems-manager/latest/userguide/getting-started-create-preferences-cli.html]

To do the same via console -

[https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-logging-auditing.html#session-manager-logging-auditing-cloudwatch-logs]

Linux way of setting permission in instance.

[https://www.linux.com/tutorials/how-manage-users-groups-linux/]

Use the Run Command “UpdateSSMAgent” or send-command SSM action to execute the document to update your SSM agents -

aws ssm send-command --instance-ids i-1234a i-1234b — document-name “AWS-UpdateSSMAgent”

How to use Session Manager

Session Manager user requirements

Federation

You need to federate using your SAML role of your application for the required account. This role should have necessary policies attached for starting session using ssm.

SSM Agent

Version 2.3.68.0 or later must be installed on the target instances.

AWS CLI (Optional)

Required to use the AWS CLI to start sessions (instead of using AWS Manager console), version 1.16.12 or later of the CLI must be installed on the local machine (Instructions to install AWS CLI for all supported systems).

In addition, the Session Manager plugin must be installed to start or terminate session over the EC2 instances. The plugin can be installed on supported versions of Microsoft Windows, macOS, Linux, and Ubuntu Server.

Session Manager Plugin

Install Session Manager Plugin on Linux

a. Download the Session Manager plugin RPM package:

64-bit:
curl “https://s3.amazonaws.com/session-manager-downloads/plugin/latest/linux_64bit/
session-manager-plugin.rpm” -o “session-manager-plugin.rpm”

32-bit:
curl “https://s3.amazonaws.com/session-manager-downloads/plugin/latest/linux_32bit/
session-manager-plugin.rpm” -o “session-manager-plugin.rpm”

b. Run the install command:

sudo yum install -y session-manager-plugin.rpm

c. Verify that the installation was successful running session-manager-plugin command. If the installation was successful, the following message is returned:

The Session Manager plugin is installed successfully. Use the AWS CLI to start a session.

Install Session Manager Plugin on macOS

a. Download the bundled installer:

curl "https://s3.amazonaws.com/session-manager-downloads/plugin/latest/mac/
sessionmanager-bundle.zip" -o "sessionmanager-bundle.zip"

b.Unzip the package:

unzip sessionmanager-bundle.zip

c. Run the install command:

sudo ./sessionmanager-bundle/install -i /usr/local/sessionmanagerplugin -b /usr/local/bin/session-manager-plugin

Install Session Manager Plugin on Windows

a. To install the Session Manager plugin using the EXE installer, download the installer using the following URL:

[https://s3.amazonaws.com/session-manager-downloads/plugin/latest/windows/SessionManagerPluginSetup.exe]

b. Run the downloaded installer and follow the on-screen the instructions.

c. Leave the install location box blank to install the plugin to the default directory:

C:\%PROGRAMFILES%\Amazon\SessionManagerPlugin\bin\

d. Verify that the installation was successful.

Starting a Session (Console)

Users with the access right to use System Manager can start session directly in the AWS Management Console:

a. In the navigation pane, choose Session Manager. In the AWS Systems Manager home page choose Session Manager menu.

b. Select the target instance to start the connection, all the instances managed by SSM should be listed.

c. Choose Start session.

d. If the connection is successful, a bash commands (Linux) or PowerShell commands (Windows) will be opened.

Starting a Session (AWS CLI)

To start a session using the AWS CLI, run the following command:

aws ssm start-session — target instance-id

Instance-id represents of the ID of an instance configured for use with AWS Systems Manager and its Session Manager capability. For example: i-02573cafcfEXAMPLE.

NOTE: To use the AWS CLI to run session commands, the Session Manager plugin must also be installed on your local machine.

Starting a Session (Port Forwarding)

a. Port forwarding session can be used to start RDP session over Windows instances. To start a port forwarding session, run the following command from the CLI:

Linux example

aws ssm start-session — target Instance-id — document-name AWS-StartPortForwardingSession — parameters ‘{“portNumber”:[“3389”],”localPortNumber”:[“12345”]}’

Windows example

aws ssm start-session — target Instance-id — document-name AWSS-tartPortForwardingSession — parameters ‘“portNumber”=[“3389”], “localPortNumber”=[“12345”]’

Instance-id represents of the ID of an instance configured for use with AWS Systems Manager. For example: i-08dbf6bb51edqew34.

PortNumber represents the remote port on the instance where traffic should be redirected to. For example: 3389 to use RDP over windows.

LocalPortNumber represents the local port on the client where traffic should be redirected to. For example: 56789. This port can be reached over the local instances by any service/tool.

b. To reach RDP you can run:

mstsc /v localhost:LocalPortNumber

After credentials verifications, the RPD session should be opened.

Starting a Session (SSH)

Note:
To start a session using SSH, your target instance must be configured to support SSH connections.

Run the following command to start a session using SSH:

ssh -i /path/my-key-pair.pem username@instance-id

Tip:
When you start a session using SSH, you can copy local files to the target instance using the following command format.

scp -i /path/my-key-pair.pem /path/SampleFile.txt username@instance-id:~
scp -i /path/my-key-pair.pem /path/SampleFile.txt ec2-user@instance-id:SampleFile.txt

SSH via ProxyCommand

On the local machine from which you want to connect to a managed instance using SSH, do the following:

• Ensure that version 1.1.23.0 or later of the Session Manager plugin is installed.
• Update the SSH configuration file to enable running a proxy command that starts a Session Manager session and transfer all data through the connection.
• For RDS, you need to ensure that port 22 is opened on your EC2 instance as this is connecting via SSH instead of AWS SSM Session Manager.

Linux

• The SSH configuration file is typically located at ~/.ssh/config.
• Add the following to the configuration file on the local machine:

# SSH over Session Manager 
host i-* mi-*     
ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'"

Windows

• The SSH configuration file is typically located at C:\Users\username.ssh\config
• Add the following to the configuration file on the local machine:

# SSH over Session Manager 
host i-* mi-*     
ProxyCommand C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters portNumber=%p"

Note:
If you want to do ssh to your EC2 instances, remember you are trying to reach a private subnet through internet. So if you want to use SSM in your local environment(cli/ssh), you need to open firewall between your local subnet and SSM DNS for your required region.

Troubleshooting
You can check ssm agent logs in below path in your instances.

/var/log/amazon/ssm/

You can check session manager plugin logs in below path in your local.

/usr/local/sessionmanagerplugin/logs/

For debugging purposes, you can enable logging as below.

[https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html#install-plugin-configure-logs]

Useful commands for reference

cli to ec2:
linux:
aws ssm start-session — target i-09b77018836663333
win: 
aws ssm start-session — target i-006d98bcda883e569
cli to ec2 using ssm port forwarding:
linux:
aws ssm start-session — target i-09b77013336667733 — document-name AWS-StartPortForwardingSession — parameters ‘{“portNumber”:[“22”],”localPortNumber”:[“9008”]}’
win:
aws ssm start-session — target i-006d98b23a88c2569 — document-name AWS-StartPortForwardingSession — parameters ‘{“portNumber”:[“3389”],”localPortNumber”:[“9009”]}’
cli to rds using ssm port forwarding:
<Not Supported>
ssh to ec2 without keypair:
ssh ec2-user@i-09b770132336667733
ssh to ec2 with keypair:
ssh -i key ec2-user@i-09b77238836667733
ssh to ec2 using local port forwarding:
ssh -nNT -L 9000:localhost:80 ec2-user@i-09b77032836667733
ssh to rds via ec2 using local port forwarding:
ssh -nNT -L 9011:<db-host>:3306 ec2-user@i-09b770182116667733

User authentication

Technical Workflow

End to End automation

This approach will provide end to end automation to install System Manager agent in Linux EC2 and attach instance profile to both Windows and Linux machines using AWS Lambda.