and ever since then we never look at a kitty the same way again. meow!
TheĀ beauty of optimismĀ isĀ āĀ when you know of things that can go wrong,Ā and you also know of ways to prevent them from going wrong,Ā yet ā you choose to gamble it all off andĀ hope that nothing will go wrong!
Confusing? Edgy? Silly? ā yes all of it and more.
Somewhere in Aug 2020, I & my team were working in frenzy on expanding our RedHat Ansible Playbooks repository. (if you are uninitiated into the world of Ansible āĀ start here)
We had built & tested neat playbooks (human-readable code which allows us to define the desired states of our components) to install & configure MySQL, PostgreSQL & MongoDB onto (multi-flavor) Linux VMs. These could also create databases & users.
In order to test these ā Two Classic Infrastructure Virtual Server ā Public Instances on IBM Cloud were setupā one with CentOS Minimal install and the other with Ubuntu. Opted for default automatic private VLAN network interface. Felt too optimistic and skipped adding SSH keys to the server during creation (vulnerability-1 introduced!).
Next come security groups āĀ An IBMā¢ Cloud Security Group is a set of IP filter rules that define how to handle incoming (ingress) and outgoing (egress) traffic to both the public and private interfaces of a virtual server instance.
While selecting from the default provided IBM security groups for the public & private interface of the provisioned devices ā dabbled around as shown below and ended up withĀ vulnerability-2. Terminology across different cloud providers always intrigues me :) , āallā does not imply āingress+egressā rather it means all-ingress-only.Ā Quite a wordy affair!
Our requirement comprised:
- Assigning these VMs as target hosts in Ansible Tower Inventory onto which multiple users from within company can execute the DB setup (& other) playbooks andĀ connect to the created databases over internet via any IDEĀ on their local machine. (Internet facing Database instances!)
- Use these machines to test the playbooks remotely during the authoring phase. And opted for the typical super user privileges and open access, wherever we were tempted to!Ā Do not do that! :)
Letās call it an experimental niche Lab setupā different from usual Dev or Prod environments.
Life was hunky & dory, till one fine morning ā a cybersecurity red alert popped in, yikes!
We had left insecure internet facing MongoDB databases like sitting ducks.
TheĀ infamous āmeowā attackĀ had happened! Though shocking, it was all a little amusing as none of our databases had any relevant or confidential data. Bazinga! (that moment when you find your wallet stolen, but it was your empty wallet).
Meow is a scripted bot attack that scrambles the data & indexes of victim Mongo databases by overriding it with numbers and appending the string āmeowā.Ā Attackers have an ironic sense of humor!Ā Find out more about meow here.
Awakened by the jolt, it was time to secure the setup for our internet facing Mongo databases. Though, there are multiple options & levels of security, readĀ MongoDBās official security checklist here; but based on our unique need, cost considerations & smart recovery options ā following were the steps to security.
1. Recreated the IBM Cloud Classic Virtual Server, this timeĀ added a pair of SSH keys and disabled root login.Ā Wondering if choosing to recreate was a tough call?Ā Well, not really. We were working on Ansible, remember?
So, we already have playbooks that automated all our subsequent configurations and setup. One-click redo. Recreation was preferable to avoid running into malware traces later on.
2. Back toĀ Security GroupsĀ of IBM Cloud Classic Virtual Server. removed allow_all ingressĀ on all ports.
Custom security groups with incoming ip-whitelisting though more secure & limits network exposure; was not preferable as our users would be accessing from local machines across globe. It would be unmanageable to whitelist user IPs on demand.
So, how would our in-house end users experimenting ā access database instances from their local machine ā IDE ?Ā The core requirement!
A hassle free shortcut wasĀ SSH Tunnel with port forwarding.
Users could connect via encrypted ssh tunnel to the database hosting Virtual Server and forward traffic from local machine db port to Server side DB port.
For example:Ā traffic from 27017 port of local machine would be forwarded to Server side mongo dB 27017 port, within a secure & encrypted SSH tunnelā thus allow IDEs to connect.
SSH tunnel is not a fool proof solution,Ā it does have potential backdoor threats. But this approach in combination with prior tightly managed SSH keys ā did increase our setupās security.
3.Ā Enable Access Control & Enforce Authentication on the Mongo DB instances from the Ansible Playbook.
Mongo DB does not enable access control/authorization by default (dated Oct 2020, current latest version ā MongoDB 4.4).
So, in the mongo DB installation ā Ansible playbook, the step toĀ enable security.authorizationĀ was added.
This wouldĀ ensure a user can access only the database resources and actions for which they have been granted privileges.Ā Prior to it āa user could access any database and perform any action, yikes!
4.Ā Configure TLS/SSL Transport layer encryption to Mongo DBĀ ā as defined here.Ā However, after enabling this ā connections made from Robo 3T (or RoboMongo), an open source MongoDB GUI were successful but MongoDB Compass GUI started behaving erratically.
And I did not want our users to be forced to pick one GUI over other for a post-playbook verification step.Ā So, decided to skip this configuration from final assembly.
- - Other options like āĀ āencryption at restā was not needed ā as the databases were purely for use-and-tear-down. It was not necessary to secure the data at rest, however those with different business needs ā may certainly do.
- - I did findĀ Auditing featureĀ interesting and it would have helped us secure our mostly transient Mongo databases better. But, this feature was only available in Enterprise Edition and Atlas (MongoDB-as-a-Service); and we were using free Community Edition.
Alrighty then ā this was the real threat received :) from Meow attackers. Never paid any bitcoin, phew!
Lesson learnt: Stay safe even whilst experimenting!
Also published at https://medium.com/wabi-sabi-techexpress/when-the-meow-attack-on-our-mongodb-shook-us-86230d79c509
- Soul staring Cat & sitting cute ducks - Image source: openclipart.org ;
- ssh tunneling - Image from ssh.com;
- rest images - first hand captures & self created explainers from the experience.