Access your AWS database using local port-forwarding on your ECS/Fargate container

If your infrastructure is well-designed, your database is accessible only from the app it’s supposed to serve. And, yes, ECS Exec enables to SSH on your containers, but you can’t use it for port forwarding. How good it would be if it was possible to use your local tooling, like DBeaver or SQL Developer, to connect to the database in a secure way? Well it is, and this blog posts show you how.

AWS-documented bastion solutions for ECS and EC2: a summary

AWS offers two distinct bastion solutions for your workloads. Both solutions rely on IAM for authentication (removing the need for static/shared credentials) and allow logging of all shell commands entered by the operator:

1. On ECS, AWS offers ECS Exec for workload debugging. The CLI command looks like this:

   aws ecs execute-command \
   --profile \
   --region \
   --cluster \
   --task \
   --container \
   --interactive --command "/bin/sh"

There a few pre-requisites:

• The task should be part of an ECS Service, not a standalone one

• “Enable execute command” should be set to true when creating the ECS service [Edit] and the appropriate policy in the task role (not to be confused with the task execution role) [/Edit]

• The container process should have write access to a few directories (use mounts if you’re otherwise following the good practice of hardening by not granting the container process write access to the host filesystem), namely /managed-agents, /var/lib/amazon/ssm and /var/log/amazon/ssm.

• The operator should have the Session Manager plugin for AWS CLI installed.

2. On EC2, the Systems Manager (SSM) Start-Session command makes it possible open an interactive shell and, via the use of SSM Documents, namely the “AWS-StartPortForwardingSessionToRemoteHost” document, to enable port forwarding to a remote host.

• SSM start-session requires a target that is an EC2 instance ID.

The trick: how to use SSM on a Fargate-hosted container

The problem on a a Fargate-hosted container is that, though there may be an EC2 instance supporting the container, you don’t know it and don’t manage it.

Here is the trick: ECS tasks have an identifier called runtimeId which you can use to run SSM start-session commands, instead of execute-command command.

1. Retrieve the runtimeId for your container:

   aws ecs describe-tasks \
   --cluster \
   --task \
   --profile \
   --region \
   --query 'tasks[].containers[?name== ].runtimeId | [0] | [0]'

(The --query clause above post-processes the CLI output so it returns only the requested identifier, not the full describe-tasks output. See here for usage.)

[Edit] The Container runtime ID can now also be found in the ECS console:

[/Edit]

1. Start a session using SSM’s start-session:

   aws ssm start-session \
   --target ecs: \
   --document-name AWS-StartPortForwardingSessionToRemoteHost \
   --parameters '{"host":[""],"portNumber":[""], "localPortNumber":[""]}' \
   --profile \
   --region

The trick here lies in the --target parameter: we append ecs: then the name of the cluster, the the ID for the task, then the runtimeID retrieved from the previous step.

(note that the taskID is included in runtimeId so it effectively appears twice in the --target parameter).

A demo

Need help to run your container workloads on AWS? Please get in touch!