Introduction:
Why SSH Configuration Drift Is a Real Security Risk
Security in cloud environments is not a one-time setup. It is a continuous process. One of the most overlooked but dangerous problems in managing a fleet of EC2 instances is configuration drift — the gradual deviation of a system's actual state from its desired, secure baseline.
SSH (Secure Shell) is the primary remote access protocol for Linux-based EC2 instances. The /etc/ssh/sshd_config file controls how the SSH daemon behaves on a server. A single misconfigured directive in this file can open the door to serious security vulnerabilities. For example, PermitRootLogin yes allows direct root access over SSH, which means an attacker who obtains valid credentials — or exploits a brute-force vulnerability — can gain full administrative control of the server without any additional privilege escalation.
This is not a theoretical risk. CIS (Center for Internet Security) Benchmarks, PCI-DSS, HIPAA, and SOC 2 compliance frameworks all explicitly require that root login over SSH be disabled. Failing this check during an audit can result in compliance violations, financial penalties, and reputational damage.
The challenge is that in a dynamic cloud environment, configuration drift happens constantly. A developer might temporarily enable root login to debug an issue and forget to revert it. A poorly written bootstrap script might overwrite the SSH config. An OS update might reset certain directives to their defaults. Whatever the cause, the result is the same your instances are no longer in the secure state you intended.
Manual audits — where someone periodically SSHes into each server and checks the config — simply do not scale. If you have 50, 100, or 500 instances, manual checks are impractical, inconsistent, and slow. What you need is an automated system that continuously monitors every instance, detects drift the moment it occurs, and remediates it without any human intervention.
This blog presents exactly such a system — a fully automated, event-driven SSH configuration drift detection and remediation pipeline built entirely on AWS-native services, with Ansible handling the actual remediation logic, Terraform managing all infrastructure as code, and AWS CodeBuild providing the CI/CD backbone.
Architecture Overview
┌─────────────────────────────────────────────────────────────────────────────┐
│ AWS Account (us-east-1) │
│ │
│ ┌──────────────┐ Weekly Cron ┌─────────────────────────────────┐ │
│ │ EC2 Fleet │◄───────────────────│ SSM State Manager Association │ │
│ │ (All Managed │ (Every Sunday │ (AWS-RunShellScript) │ │
│ │ Instances) │ at 02:00 UTC) │ Checks: PermitRootLogin no │ │
│ └──────┬───────┘ └──────────────┬──────────────────┘ │
│ │ │ │
│ │ exit 1 (Non-Compliant) │ Association Status │
│ │ │ = "Failed" │
│ │ ▼ │
│ │ ┌──────────────────────────┐ │
│ │ │ Amazon EventBridge │ │
│ │ │ Rule │ │
│ │ │ source: aws.ssm │ │
│ │ │ status: Failed │ │
│ │ └──────────┬───────────────┘ │
│ │ │ Triggers │
│ │ ▼ │
│ │ ┌──────────────────────────┐ │
│ │ │ SSM Automation │ │
│ │ │ Document │ │
│ │ │ (Custom: ssh-permit │ │
│ │ │ rootlogin-remediation)│ │
│ │ └──────────┬───────────────┘ │
│ │ │ aws:runCommand │
│ │ │ (AWS-RunShellScript) │
│ │◄─────────────────────────────────────┘ │
│ │ ansible-playbook /tmp/ssh_config.yml │
│ │ ▲ │
│ │ │ Downloads playbook │
│ │ ┌──────────┴───────────────┐ │
│ │ │ Amazon S3 Bucket │ │
│ │ │ ansible-playbook-* │ │
│ │ │ └── ssh_config.yml │ │
│ │ └──────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ CI/CD Pipeline │ │
│ │ CodeCommit/GitHub → CodeBuild (buildspec.yml) → Terraform Apply │ │
│ │ → CodeBuild (destroybuildspec.yml) → TF Destroy │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
The pipeline involves six core AWS services working in concert: SSM State Manager, Amazon EventBridge, SSM Automation, Amazon S3, IAM, and AWS CodeBuild. Ansible provides the remediation logic, and Terraform provisions every resource. Let's walk through each component in detail.
Component 1:
SSM State Manager — The Compliance Scanner
AWS Systems Manager (SSM) State Manager is a configuration management service that ensures your EC2 instances maintain a defined state over time. Unlike a one-time script, a State Manager Association is a persistent, scheduled policy that continuously re-applies or re-checks a configuration.
In this solution, a State Manager Association is created that targets all SSM-managed instances in the account using the wildcard InstanceIds: ["*"]. This is a powerful targeting mechanism — any new instance that gets the SSM Agent installed and registers with Systems Manager will automatically be included in the compliance check without any manual intervention.
The association runs on a weekly cron schedule — every Sunday at 02:00 UTC (cron(0 2 ? * SUN *)). This timing is intentional: it runs during a low-traffic window to minimize any potential performance impact on production workloads.
The compliance check itself is a simple but effective shell script:
#!/bin/bash
if grep -q "^PermitRootLogin no" /etc/ssh/sshd_config; then
echo "Compliant"
exit 0
else
echo "Non-Compliant"
exit 1
fi
The script uses grep with the ^ anchor to check for the exact directive PermitRootLogin no at the beginning of a line in /etc/ssh/sshd_config. The ^ anchor is important — it prevents false positives from commented-out lines like # PermitRootLogin no. If the directive is found, the script exits with code 0, which SSM interprets as a Success. If it is not found — meaning root login is either enabled, commented out, or the directive is missing entirely — the script exits with code 1, which SSM interprets as a Failed status.
The association is configured with compliance_severity = "HIGH". This means that when an instance fails the check, it appears as a HIGH severity finding in the SSM Compliance dashboard, giving security teams immediate visibility into which instances are non-compliant, even before the automated remediation kicks in.
Component 2:
Amazon EventBridge — The Event-Driven Trigger
Amazon EventBridge is a serverless event bus that enables you to build event-driven architectures by routing events from AWS services to target actions. In this solution, EventBridge acts as the bridge between the compliance check result and the remediation action.
When an SSM State Manager Association run completes — whether it succeeds or fails — SSM automatically publishes an event to the AWS default event bus. The event looks like this:
{
"source": "aws.ssm",
"detail-type": "EC2 State Manager Instance Association State Change",
"detail": {
"status": "Failed",
"instance-id": "i-0abc123def456789",
"association-id": "abc12345-...",
"document-name": "AWS-RunShellScript"
}
}
An EventBridge rule is configured to match exactly this pattern — specifically filtering for events where source is aws.ssm, detail-type is EC2 State Manager Instance Association State Change, and detail.status is Failed. This precision filtering ensures that the remediation pipeline is only triggered when a compliance check actually fails — not for every SSM event in the account.
One of the most important features used here is the input transformer. The raw SSM event payload does not match the parameter schema expected by the SSM Automation document. The input transformer solves this by extracting the instance-id from the event payload using a JSONPath expression ($.detail.instance-id) and mapping it to the InstanceId parameter that the Automation document expects. It also injects the PlaybookName parameter statically, pointing to the Ansible playbook stored in S3.
This approach is elegant because it eliminates the need for a Lambda function as a middleware layer. In many similar architectures, a Lambda function is used to transform the event payload and invoke the next step. Here, EventBridge's native input transformer handles that transformation directly, reducing operational overhead, cost, and the number of moving parts in the system.
Component 3:
SSM Automation Document — The Remediation Orchestrator
The SSM Automation document is the central orchestrator of the remediation workflow. It is a YAML-formatted document that defines a sequence of steps to execute when triggered. In this solution, a custom Automation document named ssh-permitrootlogin-remediation is created.
The document is of type Automation and uses schema version 0.3. It defines two input parameters: InstanceId (the ID of the non-compliant instance, injected by EventBridge) and PlaybookName (the name of the Ansible playbook to run, also injected by EventBridge).
The document has a single step named RunPlaybook that uses the aws:runCommand action. This action invokes the AWS-RunShellScript SSM document on the target instance, executing two shell commands in sequence:
aws s3 cp s3://<bucket>/Playbook/<PlaybookName>.yml /tmp/— This downloads the Ansible playbook from the S3 bucket to the/tmp/directory on the target instance. The instance must have an IAM instance profile withs3:GetObjectpermission on the bucket for this to work.ansible-playbook /tmp/<PlaybookName>.yml— This executes the downloaded playbook using the Ansible CLI installed on the instance.
The document assumes an IAM role called CustomSSMAutomationRole (referenced via a Terraform data source). This role must have the necessary permissions to invoke ssm:SendCommand on EC2 instances. Using a data source rather than creating the role in Terraform is a deliberate design choice — it assumes the role is managed separately (perhaps by a central security team), keeping the blast radius of this Terraform module small and focused.
The parameterized design of the Automation document is a key architectural strength. Because PlaybookName is a runtime parameter rather than a hardcoded value, the same document can be reused to run any Ansible playbook stored in the S3 bucket. This means you can extend the remediation system to cover other security configurations — password policies, audit logging, file permissions — simply by uploading new playbooks and creating new EventBridge rules, without modifying the Automation document at all.
Component 4:
Amazon S3 — The Playbook Repository
Amazon S3 serves as the centralized, durable storage for Ansible playbooks. The S3 bucket (ansible-playbook-837439744) is created by Terraform, and the playbook (ssh_config.yml) is uploaded as an S3 object.
A particularly smart detail in the Terraform configuration is the use of etag = filemd5("../Ansible/ssh_config.yml") on the S3 object resource. The etag attribute stores an MD5 hash of the local file. Every time terraform apply is run, Terraform computes the current MD5 hash of the local playbook file and compares it to the stored etag. If the file has changed, Terraform detects the difference and re-uploads the file to S3. If the file is unchanged, Terraform skips the upload.
This means that updating the remediation logic is as simple as editing the Ansible playbook locally and running terraform apply — or pushing to the repository and letting CodeBuild handle it. There is no manual S3 upload step, no separate deployment process for the playbook. The infrastructure and the remediation logic are deployed together as a single atomic unit.
Component 5:
The Ansible Playbook — Idempotent Remediation
The Ansible playbook (ssh_config.yml) is the actual remediation logic. It is intentionally simple and focused:
---
- name: Remediate SSH Configuration Drift
hosts: localhost
become: yes
tasks:
- name: Disable root login
lineinfile:
path: /etc/ssh/sshd_config
regexp: '^PermitRootLogin'
line: 'PermitRootLogin no'
The playbook targets localhost because SSM Run Command executes the script directly on the target instance — there is no Ansible control node involved. The become: yes directive ensures the task runs with root privileges, which is required to modify /etc/ssh/sshd_config.
The lineinfile module is the right tool for this job for several reasons. First, it is idempotent — running it multiple times produces the same result as running it once. If PermitRootLogin no is already present, the module does nothing. If PermitRootLogin yes is present, it replaces it. If no PermitRootLogin directive exists at all, it inserts PermitRootLogin no. This idempotency is critical in an automated remediation system — you never want a remediation action to cause unintended side effects if it runs more than once.
Second, lineinfile is surgical — it modifies only the specific line matching the regexp pattern, leaving the rest of the SSH configuration file untouched. This is far safer than replacing the entire sshd_config file with a template, which could inadvertently overwrite instance-specific customizations.
Third, the regexp: '^PermitRootLogin' pattern matches any existing PermitRootLogin directive regardless of its current value, ensuring the line is always updated to the desired state.
Component 6:
IAM — Least-Privilege Security
Two IAM resources are created in this solution, each following the principle of least privilege.
The first is the EventBridge IAM role (eventbridge-ssm-role). This role is assumed by the events.amazonaws.com service principal, meaning only EventBridge can assume it. The role's inline policy grants only ssm:SendCommand — the minimum permission needed for EventBridge to invoke the SSM Automation document. In a production hardening step, the Resource in this policy should be scoped to the specific SSM document ARN rather than "*", further limiting the blast radius if the role were ever misused.
The second is the CustomSSMAutomationRole, which is referenced via a Terraform data source rather than created by this module. This role is assumed by the SSM Automation document and needs permissions to run commands on EC2 instances via ssm:SendCommand. Managing this role outside the module is a good practice — it allows a central security team to control what the automation can do, while the application team manages the rest of the infrastructure.
Component 7:
Terraform Infrastructure as Code
Every AWS resource in this solution is defined in Terraform, organized across focused, single-responsibility files:
-
provider.tf— Configures the AWS provider (pinned to version6.54.0for reproducibility) and the S3 remote backend for state management. Remote state in S3 enables team collaboration and prevents state file conflicts. -
variable.tf— Declares all input variables (bucket_name,object_name,ssm_doc,event_rule_name), making the module configurable and reusable across environments. -
terraform.tfvars— Provides concrete values for all variables, keeping configuration separate from logic. -
s3.tf— Creates the S3 bucket and uploads the Ansible playbook with MD5-based change detection. -
stateassociations.tf— Creates the SSM State Manager Association that runs the weekly compliance check. -
ssmdocument.tf— Defines the custom SSM Automation document that orchestrates remediation. -
eventbridge.tf— Creates the EventBridge rule and target with the input transformer. -
iam.tf— Creates the EventBridge IAM role and its inline policy. -
data.tf— Fetches the pre-existingCustomSSMAutomationRoleby name.
This file-per-resource-type organization makes the codebase easy to navigate, review, and extend. Each file has a clear, single responsibility, and dependencies between resources are expressed through Terraform's depends_on and implicit reference mechanisms.
Component 8:
CI/CD with AWS CodeBuild
The project includes two CodeBuild buildspec files that implement a complete infrastructure lifecycle pipeline.
buildspec.yml handles the deploy workflow. It installs Terraform 1.8.5 from HashiCorp's official release, initializes the backend, validates the configuration syntax, generates a plan saved as tfplan, and then applies that exact plan. The separation of plan and apply into pre_build and build phases is intentional — it ensures that what was reviewed and approved in the plan phase is exactly what gets deployed, with no possibility of drift between the two steps.
destroybuildspec.yml handles the teardown workflow. It follows the same initialization steps but generates a destroy plan (terraform plan -destroy) and applies it. Having a dedicated, version-controlled destroy pipeline is a best practice that is often overlooked. It enables clean teardown of ephemeral environments, prevents orphaned resources from accumulating costs, and ensures the destroy process is as auditable and repeatable as the deploy process.
Both buildspec files cache the .terraform/ directory, which contains the downloaded provider plugins. This significantly speeds up subsequent builds by avoiding repeated downloads of the AWS provider binary.
End-to-End Flow:
Putting It All Together
To understand how all these components work together, let's trace a complete remediation event from start to finish.
Every Sunday at 02:00 UTC, the SSM State Manager Association triggers and runs the compliance check shell script on every managed EC2 instance in the account simultaneously.
On a particular instance (
i-0abc123def456789), the script finds thatPermitRootLoginis set toyes(or is missing entirely). The script exits with code1. SSM records the association run as Failed for that instance and updates the SSM Compliance dashboard with a HIGH severity finding.SSM publishes an
EC2 State Manager Instance Association State Changeevent to the EventBridge default event bus, withstatus: Failedandinstance-id: i-0abc123def456789in the event detail.The EventBridge rule matches this event. The input transformer extracts
i-0abc123def456789asinstanceIdand constructs the parameter payload{ "InstanceId": ["i-0abc123def456789"], "PlaybookName": ["ssh_config.yml"] }.EventBridge invokes the SSM Automation document
ssh-permitrootlogin-remediationwith this parameter payload, using theeventbridge-ssm-roleIAM role.The Automation document's
RunPlaybookstep executesaws:runCommandon instancei-0abc123def456789, running two shell commands: first downloadingssh_config.ymlfrom S3 to/tmp/, then runningansible-playbook /tmp/ssh_config.yml.Ansible's
lineinfilemodule finds thePermitRootLogin yesline in/etc/ssh/sshd_configand replaces it withPermitRootLogin no. The SSH daemon configuration is now compliant.The following Sunday, the compliance check runs again on the same instance. This time,
grepfindsPermitRootLogin noand the script exits with code0. The association run is recorded as Success. The HIGH severity finding is cleared from the SSM Compliance dashboard.
The entire remediation cycle — from detection to fix — happens automatically, without any human intervention, typically within minutes of the compliance check completing.
Key Design Decisions
No Lambda function. Many event-driven architectures on AWS use Lambda as a glue layer between services. Here, EventBridge's native input transformer handles the payload transformation, and SSM Automation handles the orchestration. Eliminating Lambda reduces cost, operational overhead, cold start latency, and the number of components to monitor and maintain.
Ansible over raw shell scripts. The remediation could have been implemented as a raw shell script embedded in the SSM document. Ansible was chosen because it provides idempotency guarantees, a declarative syntax that is easier to read and audit, and a rich module library that makes it easy to extend the playbook to cover additional SSH directives or other configuration files.
Playbook in S3, not embedded in the SSM document. Embedding the playbook content directly in the SSM Automation document would tightly couple the remediation logic to the infrastructure definition. Storing it in S3 decouples them — you can update the playbook without modifying or redeploying the Terraform infrastructure. The etag = filemd5(...) trick in Terraform ensures the S3 object stays in sync with the local file automatically.
Parameterized Automation document. The PlaybookName parameter makes the Automation document a generic Ansible runner, not a single-purpose SSH remediation tool. This is a forward-looking design — as your compliance requirements grow, you can add new playbooks to S3 and new EventBridge rules without touching the core automation infrastructure.
Prerequisites and Deployment
Before deploying, ensure the following are in place on all target EC2 instances:
- SSM Agent installed and running (included by default on Amazon Linux 2, Amazon Linux 2023, and Ubuntu 20.04+).
- Ansible installed (
sudo pip install ansibleor via the OS package manager). - An IAM instance profile attached to each instance with
s3:GetObjectpermission on the playbook bucket andssm:*permissions for SSM Agent communication. - A pre-existing IAM role named
CustomSSMAutomationRolewithssm:SendCommandpermissions. - An S3 bucket for Terraform remote state.
Deployment is straightforward:
git clone <your-repo-url>
cd sshconfig_remediation_systemmanager/Terraform
terraform init
terraform plan
terraform apply
Or push to your connected repository and let the CodeBuild pipeline handle it automatically.
Conclusion
This solution demonstrates that you do not need complex tooling or a large number of services to build a robust, production-grade security remediation pipeline on AWS. By thoughtfully combining SSM State Manager, EventBridge, SSM Automation, S3, Ansible, and Terraform, you get a system that is automated, scalable, auditable, and extensible.
The compliance check runs on a schedule across your entire fleet. Failures are detected immediately and trigger automated remediation without human intervention. Every component is defined as code, version-controlled, and deployable through a CI/CD pipeline. The remediation logic is decoupled from the infrastructure, making it easy to update. And the architecture is generic enough to extend to any OS-level configuration that needs to stay compliant — not just SSH.
In a world where security misconfigurations are one of the leading causes of cloud breaches, having an automated system that continuously enforces your security baseline is not a luxury. It is a necessity.



Top comments (0)