What is this about?
We all have had those times where we take on a task and at first glance it seems like this should be straight forward. Suddenly you are left with a million questions and many internet searches you've entertained have led you down a wormhole with no solutions in sight. In my case this came about from a MongoDB migration in which I had to lead.
Prior to this we hosted MongoDB on a few instances with collections for all our maintained environments. One of the most important projects for the year meant we would be expecting a large influx of users which we were not ready for. The need for higher availability, scaling and speed were critical.
Enter MongoDB Atlas. Before proceeding any further let me start off by saying that MongoDB has some pretty good documentation; So I will be skipping over most of the "Getting Started" which can be found in MongoDB docs. While I won't discuss every single obstacle. Hopefully I can save others some precious time where I had little to spare.
TLDR version:
A Few Gotchas to watch out for when doing a migration or starting out with MongoDB Atlas
Using a VPN? - Always check the Whitelist!
I figured let's start off with my homer Simpson moment. VPN is used to access any of our instances and I am pretty much always running one. At first it took me a while as I worked through basic connections steps. However I soon realized this was my classic "D'OH" moment and I needed to whitelist my IP and make a mental note that when I see a connection error. Update it!
Fortunately MongoDB makes it really easy to do this. Just a few clicks and you can add your current IP Address.
Check and match all your versions. It's a Must!
(If you're already using containers such as Docker you can skip this part about Matrix Hell. We were not!)
Remember those match card memory games? In the game you needed to match up every pair and complete the whole set to win. I found the same to be true in my case of dealing with MongoDB. Through a ton of trial, documentation and yes even contacting support(which basically they just reiterated what I already had learned and attempted through documentation). It came down to me tracking down versions. Ruby, mongodb, mongoid, linux distro, mongo shell.... Just basically check everything. As I was also in the process of upgrading our mongo version; this meant format changes had to be applied to the mongoid yaml configuration. An example template of this can be found at this link.
Mongoid.yml
Connection issues? - Try working your way through a manual connection.
While MongoDb has some pretty good documentation there is still room for improvement. The suggestions for how to connect your app just were not working for me. Even support pretty much just repeated what I already had attempted thus far but they also never asked about what version I am using to help connect the dots.
Finding the difference between when to use "mongodb+srv://" or using "mongodb://" is a pretty significant piece of info. It was only when comparing my working manual connection with the sites suggestions for setting up a config that I noticed this key difference. Replica sets, required info and options are all dealt with differently depending on which one you use. I feel this has some key differences that maybe could have been highlighted in a stronger way. As it took the efforts of using a fine tooth comb to help figure out the discrepancies.
Just take for example a barebones connection string.
Older version:
mongodb://mongodb0.example.com:27017,[mongodb1.example.com:27017](http://mongodb1.example.com:27017/),[mongodb2.example.com:27017/admin?replicaSet=myRepl](http://mongodb2.example.com:27017/admin?replicaSet=myRepl)
Newer version:
mongodb+srv://mongodb0.example.com/[admin](http://mongodb2.example.com:27017/admin?replicaSet=myRepl)
With the new version you only really need the address but the older one has you explicitly define each replica node.
The best thing to actually do was just break out my manual connection command in its entirety and put it in the config. Had I never looked at their suggestion I probably would have saved myself a bit of a headache.
MongoDB has Connection Instructions which lets you choose your desired method.
Remember your old services
Don't forget during a migration your sometimes launching the new while still working with the old. When you run into some funky behavior. Check to see if you still have the older version up. For example if you use Redis with MongoDB. Which ever way you chose to implement could possibly mean sharing the service between old and new instances. This will have your app and/or data performing chaotic at times.
Use Drop to Update
If you are doing several imports/restores over the course of the migration. You may not want to delete your collection every single time. I have a preference for full delete & restore when finalizing my migrations. However if I just want to test/verify the most recent data; an update is all that is desired. Problem is MongoDB does not have an update in the traditional sense. During restore MongoDB does not allow overwriting of records. Try to run a restore and you will encounter a number of duplicate key errors.
An Example error:
- E11000 duplicate key error collection: mydb.mycollection index: id dup key: { : "MyKey" }
To avoid the annoying errors you just need to add the —drop flag. This removes the collection from your target right before it restores from your source dump.
Ex.
mongorestore --host <replSetName>/<hostname1><:port>,<hostname2><:port>, authSource=admin" —drop <dbname> <dbpath>
Bonus
This is not really a gotcha at all. It just feels weird to show the restore command without the dump especially since you have to dump before you can restore. By default mongodb provides a dump directory with the contents for you if a path is not provided.
mongodump --db <dbname>
Good Luck!
Well time to wrap it up! Hopefully this helps someone out there even if its just to confirm nope you are not crazy. Please let me know if you have any questions or if something is not clear in the post.
Top comments (0)