Photo by sebastiaan stam on Unsplash
Life is unpredictable they say, for software developers there can't be a more true statement. We don’t know what awaits us each day when we open our laptops.
Today was such a day in my life.
If you have been reading my articles, you would know that I was building a tweet scheduler for fun and learning. Nothing serious, just wanted to figure out how deployment to AWS works and what are the basic steps to be followed to develop and deploy a product from scratch.
Every day I had a new challenge to face and I was learning about AWS faster than I could have by enrolling in a course. So far so good.
Yesterday I found two issues with the app. One was that the app deployed on elastic beanstalk had its health status shown in RED. The issue was labeled as “Severe” with the following warning message,
ELB health is failing or not available for all instances
I was not sure how to troubleshoot the issue and I was also getting a proper response from the health check API of the application.
As the app was deployed just for demo purposes, I had only one EC2 instance up for this without a load balancer. So the problem turned out to be that, health check on AWS is by default performed on “/” path. That is, an application would need to create an API that can be accessed at the root path and responds with a 200 OK. If we need a custom health check API to be invoked instead of a default one then that needs to be configured.
So I created a new API that responds with a 200 OK.
Committed the code and the deployment started using the CI/CD pipeline and deployment FAILED.
Turns out my MongoDB didn't have any of the collections that are needed for my application to start. I wondered how that happened and didn't give it a second thought. I already had the data backup so I just created those collections again in the MongoDB that’s deployed on another EC2 instance.
All good again, the app is up and tweets again.
The same thing happened the next day as well, I couldn’t find the collections in my Mongo DB, my first hunch was something is autodeleting the collections if not in use (Developer’s wild thoughts). This time though I really wanted to understand the issue.
After googling the issue for a while, I got a hint that an exposed DB(one without access restrictions) has security concerns and is vulnerable to attack. I didn’t have any sensitive data stored in DB, just a few links to my articles and a few tokens created for the app for testing purposes, and that made me a little insensitive towards the security aspect of an application and data that can be accessed from any IP address.
As any developer, I started with the logs to find the clue. Went through the mongo logs and found this,
{"t":{"$date":"2022-02-09T01:59:59.878+00:00"},"s":"I", "c":"STORAGE", "id":22206, "ctx":"conn86","msg":"Deferring table drop for index","attr":{"index":"_id_","namespace":"config.system.sessions","uuid":{"uuid":{"$uuid":"a6892f16-e340-4197-9a94-a5d8b247e1ad"}},"ident":"index-5-2402867815965632471","commitTimestamp":{"$timestamp":{"t":0,"i":0}}}}
{"t":{"$date":"2022-02-09T01:59:59.878+00:00"},"s":"I", "c":"STORAGE", "id":22206, "ctx":"conn86","msg":"Deferring table drop for index","attr":{"index":"lsidTTLIndex","namespace":"config.system.sessions","uuid":{"uuid":{"$uuid":"a6892f16-e340-4197-9a94-a5d8b247e1ad"}},"ident":"index-6-2402867815965632471","commitTimestamp":{"$timestamp":{"t":0,"i":0}}}}
{"t":{"$date":"2022-02-09T01:59:59.878+00:00"},"s":"I", "c":"STORAGE", "id":22214, "ctx":"conn86","msg":"Deferring table drop for collection","attr":{"namespace":"config.system.sessions","ident":"collection-4-2402867815965632471","commitTimestamp":{"$timestamp":{"t":0,"i":0}}}}
{"t":{"$date":"2022-02-09T01:59:59.878+00:00"},"s":"I", "c":"COMMAND", "id":20336, "ctx":"conn86","msg":"dropDatabase","attr":{"db":"config","numCollectionsDropped":1}}
{"t":{"$date":"2022-02-09T01:59:59.886+00:00"},"s":"I", "c":"COMMAND", "id":20337, "ctx":"conn86","msg":"dropDatabase - starting","attr":{"db":"local"}}
{"t":{"$date":"2022-02-09T01:59:59.886+00:00"},"s":"I", "c":"COMMAND", "id":20338, "ctx":"conn86","msg":"dropDatabase - dropping collection","attr":{"db":"local","namespace":"local.startup_log"}}
{"t":{"$date":"2022-02-09T01:59:59.886+00:00"},"s":"I", "c":"STORAGE", "id":22206, "ctx":"conn86","msg":"Deferring table drop for index","attr":{"index":"_id_","namespace":"local.startup_log","uuid":{"uuid":{"$uuid":"3a353cc7-7fd3-44cc-8959-24aa8b02de8c"}},"ident":"index-3-2402867815965632471","commitTimestamp":{"$timestamp":{"t":0,"i":0}}}}
{"t":{"$date":"2022-02-09T01:59:59.886+00:00"},"s":"I", "c":"STORAGE", "id":22214, "ctx":"conn86","msg":"Deferring table drop for collection","attr":{"namespace":"local.startup_log","ident":"collection-2-2402867815965632471","commitTimestamp":{"$timestamp":{"t":0,"i":0}}}}
{"t":{"$date":"2022-02-09T01:59:59.886+00:00"},"s":"I", "c":"COMMAND", "id":20336, "ctx":"conn86","msg":"dropDatabase","attr":{"db":"local","numCollectionsDropped":1}}
{"t":{"$date":"2022-02-09T01:59:59.893+00:00"},"s":"I", "c":"COMMAND", "id":20337, "ctx":"conn86","msg":"dropDatabase - starting","attr":{"db":"test"}}
{"t":{"$date":"2022-02-09T01:59:59.893+00:00"},"s":"I", "c":"COMMAND", "id":20338, "ctx":"conn86","msg":"dropDatabase - dropping collection","attr":{"db":"test","namespace":"test.identity"}}
{"t":{"$date":"2022-02-09T01:59:59.893+00:00"},"s":"I", "c":"COMMAND", "id":20338, "ctx":"conn86","msg":"dropDatabase - dropping collection","attr":{"db":"test","namespace":"test.cred"}}
{"t":{"$date":"2022-02-09T01:59:59.893+00:00"},"s":"I", "c":"COMMAND", "id":20338, "ctx":"conn86","msg":"dropDatabase - dropping collection","attr":{"db":"test","namespace":"test.tweet"}}
{"t":{"$date":"2022-02-09T01:59:59.901+00:00"},"s":"I", "c":"STORAGE", "id":20320, "ctx":"conn86","msg":"createCollection","attr":{"namespace":"READ__ME_TO_RECOVER_YOUR_DATA.README","uuidDisposition":"generated","uuid":{"uuid":{"$uuid":"9c5b4513-66b6-499d-924b-3ecf1a9d9140"}},"options":{}}}
{"t":{"$date":"2022-02-09T01:59:59.912+00:00"},"s":"I", "c":"INDEX", "id":20345, "ctx":"conn86","msg":"Index build: done building","attr":{"buildUUID":null,"namespace":"READ__ME_TO_RECOVER_YOUR_DATA.README","index":"_id_","commitTimestamp":null}}
Well, clearly someone out there had been dropping my collections!
Towards the end of the logs, I found that the hacker was creating a new collection and the name said it all “READ__ME_TO_RECOVER_YOUR_DATA”.
Clearly, the script that hacked my database had no idea if the app was a real-world application or a dummy one.
Just out of curiosity, I read the message that was left on the newly created collection. The collection had one document in it and this was the content,
{"_id":{"$oid":"62049c7cc4b8ad66550b7383"},"content":"All your data is a backed up. You must pay 0.021 BTC to 17jHiu7FGUX8xcotaxBnxnNZRTqU86kr8b 48 hours for recover it. After 48 hours expiration we will leaked and exposed all your data. In case of refusal to pay, we will contact the General Data Protection Regulation, GDPR and notify them that you store user data in an open form and is not safe. Under the rules of the law, you face a heavy fine or arrest and your base dump will be dropped from our server! You can buy bitcoin here, does not take much time to buy https://localbitcoins.com or https://buy.moonpay.io/ After paying write to me in the mail with your DB IP: recmydb+17o8d@onionmail.org and you will receive a link to download your database dump."}
Hacker wants 0.021 BTC in exchange for my data, in USD that’s close to 958$ as of Feb 10th, 2022.
To those, who like me, haven’t heard of onionmail before, one can create an emailId that would be private and anonymous at onionmail.org. (Sounds like hackers' favorite space).
So I have stopped the EC2 instance, regenerated the tokens, and stopped my application as well. I will bring it up as and when I’m trying something new on it.
Lessons Learnt
- Any instance of the app that’s deployed for demo needs to be shut down right after the demo purposes.
- Basic security must be taken care of even for fun projects.
- With more apps being built using blockchain technology, cyber-security has become an extremely crucial aspect of any application.
Top comments (0)