Summary: Locked myself out via my own iptables policy. Other possible ways to connect are laid out but the one that worked is explained in the What really solved it section.
What I'm using
- EC2 t2.xlarge instance with Elastic IP
- I haven't installed any AWS-related agents before the incident
- Security groups are set to allow only connections from either my IP or the security group itself
- I didn't modify the default parameters set in the iptables-config file
I was playing around with DROP and REJECT on iptables and I'm using the "--protocol" as criteria to apply policy on. Since I've used udp and icmp already, I tried to drop tcp connections - bad idea.
$ sudo iptables -A INPUT --protocol tcp --in-intersection
After some searching, I learned that I'm basically fcked up. From a Stackoverflow discussion How to fix iptables if i have blocked all incoming and outgoing connections?
From comments, we've established this is on an Amazon AWS EC2 instance, and that you've locked yourself out from SSH access remotely. By using Amazon EC2, you're going to have a bit of a headache here. There's no real serial/console mode for access, nor anyone who can just 'fix' it, and by disabling all connections as you did, you've locked yourself out completely.
You don't really have much of a solution here but to destroy the EC2 instance and start over.
While this was helpful, note that this answer was posted in the Stackoverflow site last 2017 and there have been other ways created to connect to the instance. To see the options on how to connect to your EC2 instance, select your instance and click Connect.
Possible solutions
SSH Client
I've already used up SSH client when I blocked myself through iptables, so that leaves me with 3 more options.
EC2 Instance Connect
This didn't worked out as well since I needed to have EC2 Instance Connect installed on the machine before anything else. I tried to go the link provided in the error message Task 1: Configure network access to an instance. but it tackled more about security groups which isn't the one that's restricting me.
Session Manager
This would have been really useful because this uses SSH to connect to the instance. However, like EC2 Instance Connect, the Session Manager needs to be installed on the machine before anything else.
NOTE: Have this installed upon launching your instance or better yet define it in the startup script. Steps on how to install Session Manager in Linux can be read here.
EC2 Serial Console
This was a new one which I haven't seen last year during my review for the Associates certification exam. It required me to allow the account to use the EC2 Serial Console so I simply clicked Managed access and then ticked the Allow box and Update on the next page.
Going back to the previous EC2 instances landing page, selected my instance and hit Connect to see the EC2 Serial Console tab again. This time it showed another message - "This instance type is not supported for the EC2 serial console."
I went over the provided links and breezed through them. After allowing EC2 serial console access to the account, the instance type I'm using should also one of the supported instance type - Nitro. These are references you may check out:
Detaching the root volume and attaching to a second instance
This is actually an interesting method though I decided not to opt for this because I already found a solution - a really simpel one actually!
Keep scrolling down to read about it.
Also, I might create a separate article about this method.
EC2 Support
When all else fail, reach out to support. My free tier account has already expired years ago and I'm currently using AWS free credits I've obtained by joining the AWS Community Builders program. I am pretty sure that I'll have someone from their end answered some of my inquiries in the next 6 hours but I decided to not take the rouble of raising any support requests to them.
Since my focus at the moment is doing labs for the RHCSA topics and not on AWS, going through the linked documentations would be a rabbithole and would tend to be more complicated that just by simply deleting this instance and spinning up a fresh one.
Having said, I am considering to destroy it and start over. But this time, I'm having SSM Session Manager installed!
What really solved it!
I was almost on the verge of completely deleting the instance but then I remembered that iptables has a default config file called iptables-config, which contains two important parameters:
# Save current firewall rules on stop.
# Value: yes|no, default: no
# Saves all firewall rules to /etc/sysconfig/iptables if firewall gets stopped
# (e.g. on system shutdown).
IPTABLES_SAVE_ON_STOP="no"
# Save current firewall rules on restart.
# Value: yes|no, default: no
# Saves all firewall rules to /etc/sysconfig/iptables if firewall gets
# restarted.
IPTABLES_SAVE_ON_RESTART="no"
Basically what this two parameters meant is, if these two are set to "no", then rules will not persists when the machine is stopped or restarted. This means rules will be flushed out - yes!
So, after some restarting, I was able to log in again! All rules are also flushed out!
Didn't thought that a simple restart would save the day once again. Whew!
$ sudo iptables -L -v
Chain INPUT (policy ACCEPT 379 packets, 47895 bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 486 packets, 50206 bytes)
pkts bytes target prot opt in out source destination
Lesson Learned
Going back to the original Stackoverflow discussion I've read earlier, it provided some important pointers which I thoguht to include here. I'm putting the original thread in the references section at the bottom.
Use the EC2 security group firewall instead. This is a little easier to configure, and it's already there without any additional configuration - it's part of the EC2 infrastructure where you have to permit ports to actually get to the EC2 instance. You also aren't going to lock yourself out as easily (though you can get locked out, it's trivial to fix it then because you just allow port 22 again in the rule set from the Amazon EC2 settings panel, provided you don't mess with iptables as well).
Use a decent iptables ruleset and don't log out from PuTTY on your EC2 until you are absolutely sure the rules you've put in place don't completely torpedo your access to the system.
Recommended to not mess with the default policies on a server, because it has some... evils... if not done correctly, and I usually only filter ingress traffic and FORWARD traffic, and permit Outgoing traffic because of time sync servers,
Decent iptables - A Working Rule Set for iptables per your requirements
You don't need to type lines that have a # at the beginning, those're just my comments explaining what each command does. Also, replace YOUR.IP.ADDRESS.HERE with your actual IP address, where it shows up below.
Inbound filtering
# Permit localhost to communicate with itself.
iptables -A INPUT -i lo -j ACCEPT
# Permit already established connection traffic and related traffic
iptables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
# Permit new SSH connections into the system from trusted IP address
iptables -A INPUT -p tcp --dport 22 -s YOUR.IP.ADDRESS.HERE -m conntrack --ctstate NEW -j ACCEPT
# Permit all other traffic from trusted IP Address
iptables -A INPUT -s YOUR.IP.ADDRESS.HERE -j ACCEPT
# Drop all other traffic
iptables -A INPUT -j DROP
Outbound filtering
# Allow Localhost to itself
iptables -A OUTPUT -i lo -j ACCEPT
# Allow RELATED,ESTABLISHED state traffic (related to Inbound for example)
iptables -A OUTPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
# Allow all other traffic to trusted IP address
iptables -A OUTPUT -d YOUR.IP.ADDRESS.HERE -j ACCEPT
# Drop all other unpermitted outbound traffic.
iptables -A OUTPUT -j DROP
Warning: This will block access to the update servers, time sync servers, etc. so ONLY filter on Outbound if you absolutely need to, otherwise don't do this section at all
Forward filtering:
# Drop FORWARD target traffic, we don't need it
iptables -A FORWARD -j DROP
NOTE: Unless you really need to restrict things like forwarding traffic to the Internet via a tunnel or VPN to your server as a 'proxy' to the 'net, you really don't need to mess with the FORWARD rulesets, so I would suggest not doing this because nothing else is really going to use this function or ever land in this rule set table
References
- How to fix iptables if i have blocked all incoming and outgoing connections?
- Should you use iptables with EC2 instances?
Top comments (0)