If you died in a car accident on your way to work tomorrow would your replacement be able to access your systems and work products? Or would they be locked out of everything and struggle for weeks or months to gain access and move forward? Business continuity planning is something most people associate with CEOs and big companies. However, as the sole software developer in my company for many years I was acutely aware of how screwed my employer would be if I died suddenly.
In this post in going to show you the steps I took to ensure my company wouldn't be crippled if I died.
Document whatever you can in a private wiki and share it with your boss
The first rule of business continuity planning is documentation. I created a private wiki (hosted by a third party) and documented everything I deemed important. Then I shared my boss on the wiki with full admin privileges so he could share it with my replacement in the event of my death.
I put all kinds of useful information in our wiki like:
- an overview of all the pieces of our production environment and how they fit together
- SaaS products we use like BitBucket, Jira, Campaign Monitor, etc.
- web server configurations
- firewall configurations
- project specific configuration files
- router configuration settings with screenshots
- network layout including static and dynamic ranges along with switch and printer IP addresses
- DNS settings
- SSL certificate details
- how our database backups work and where the backups are stored
I had three criteria for creating this kind of documentation. It must:
- not damage our security if the wrong person sees it (that means no usernames, passwords, or private encryption keys)
- help me recreate our systems if the need ever arises
- help my replacement take over my duties if I die
When we grew from one programmer to two, the wiki also helped me orient our new programmer and delegate work to him, which was a nice side benefit.
Some tips
It is easiest to create this documentation when you are setting up a new system. That's what I did most of the time. But sometimes I would stumble upon something I forgot to document and I found it a bit more of a pain to document after the fact.
I documented to the expected skill level of my replacement. That means I assumed my audience was a programmer with a reasonable understanding of Windows and Linux administration, web servers, networking, etc.
I freely documented text commands and made screenshots of settings where I didn't write a script to execute all the commands for a task. My goal was to take as much guesswork out of the process as possible.
Finally, I updated the documentation whenever I used it to:
- reduce ambiguities
- record new settings
- document updates to GUIs or commands
Business continuity planning for passwords and private encryption keys
Because I was an admin on everything, I felt I needed to take extreme precautions with my passwords and private encryption keys. I was very aware that someone could literally destroy our business if this information fell into the wrong hands.
On the other hand, I also needed to be able to access this information on a regular basis, ensure I never lost it, and somehow make it available to my boss if I died suddenly.
My approach
I took a two pronged approach. First I setup separate accounts for my boss and made him an owner on as many accounts as appropriate. This is stuff like AdWords, Google Analytics, shared Google Docs, our domain name provider, DNS, etc.. In the event of my death, he could simply remove my access and add my replacement on those accounts.
But, if you are following the principle of least privilege--which you should be--then there are certain accounts you shouldn't share with anyone such as server logins, ssh passwords, database logins, router passwords, private encryption keys, and the like. Most non-technical people aren't in a position to protect highly sensitive credentials. And if they don't need access to these accounts to do their job, they shouldn't have access to them.
So, for the second prong of my approach I:
- created strong, long, random passwords (the more important account, the longer the password)
- stored them in an encrypted container with a super-strong master password that I memorized (and never wrote anywhere)
- stored the encrypted container in the cloud so I can access it from anywhere and to take advantage of the automated versioning and backups of my preferred storage provider
- created a printout of my passwords and periodically and securely shared it with my boss (see details below)
How I shared my passwords with my boss without exposing them
I printed a copy of my passwords and sealed them in an envelope, which I signed and dated on the seal. Then I made arrangements to store the envelope in the company safe. And I actually watched the safe keeper put it into the safe because I'm that kind of paranoid.
My boss knew to request the envelope in the event of my death. And the safe keeper knew to only give the envelope to me if I was alive or to my boss if I was dead.
Every couple of months--or more often if something important changed--I'd replace the envelope in the safe with an updated version. If the old envelope was missing or had been opened, I'd know my passwords were compromised. I'd also open the old envelope to ensure it contained the password file I was expecting and then I'd cross-cut shred it and the envelope (so nobody would see how I signed and dated it).
That's it. My boss was happy because he knew he could access all our systems if I died. And I was happy because I knew that our most sensitive passwords weren't written on a Post-It note stuck to the bottom of someone's keyboard. Nor were they vulnerable to social engineering attacks like Kevin Mitnick was famous for perpetrating.
Just a little warning
Pay particular attention to the stuff you've encrypted with symmetric or public key cryptography. Your replacement isn't going to be access that stuff with a simple password reset or a call to your vendor. So make sure those passwords (and private keys) are part of your plan and that they are always up to date.
Wrapping up
Business continuity planning is important for every key employee in your company, not just for owners and senior managers.
I believe everybody should be documenting their systems. That's just common sense. It really wasn't that much work and it has paid for itself many times over in the time it's saved me over the years.
As the sole programmer in my company I had to get a little creative with my business continuity planning for passwords and encryption keys. I believe I struck a good balance between security and convenience, given my circumstances. But what's right for you may be different. I encourage you to think about your circumstances and engage in some business continuity planning of your own so your company won't be crippled if you die in a car accident on your way to work one day.
Agree or disagree? I'd love to hear your thoughts.
Top comments (16)
If I did my work as good as I intended to, my colleagues could pick up things while hardly missing a beat. However, The development throughput of my team would be reduced until a replacement developer would be sufficiently onboarded (~1 month at least).
Reduced throughput is okay and should be expected. You need to worry if your team will be locked out of servers and encrypted data.
That shouldn't ever happen if the IT department did their job.
Agreed.
You are in a specific situation as working in a very small company. I guess this kind of stuff apply while the company has a quite small IT department; I mean for the specific of password etc. At least in a big company few people would have the problem and they would share the info.
Still the documentation aspect and in general knowledge sharing is important. as working in a bigger company (11K people roughly) with more than half on IT, our situation is different.
In a sense it occurs all the time. People leave for another company (you might too) or simply they change team, get promoted and so on. So documentation is important, but honestly not so many people even think to read it so we try to compliment that with knowledge sharing, to ensure any specific type of task, code, methodology, whatever is known by 2,3 people at least, ideally much more.
But actually, even if nobody is unique and everybody can be replaced easily; the most value you or me have for our company is not what we did and what we know, but what we will do tomorrow and for the years to come. A skilled employee, that think well, know the internal politics, the business etc is extremely valuable. He will take the right decisions, do the things efficiently and work well with others, teammates, boss, clients, whoever. That what give the most value to your company.
What ever you do, you are dependent on the new employee replacing you to be at least decent at his job... And in you case, as you seems to be at least devops and manage everything by yourself, you need a similar profile... Maybe not so easy to find.
Thanks for your thoughtful comments, Nicolas.
You're right, big companies deal with these issues differently than small companies. My experience in small companies has been that people rarely commit resources to continuity planning because there is always something seemingly more urgent going on. But if someone critical to the company dies or leaves suddenly, its very existence can be threatened.
Nobody thinks to read your documentation? That strikes me a strange. Is your documentation useful/helpful? Or is it verbose, out of date, misleading, etc., etc.?
Finding talented software devs/IT people with the range of skills required to succeed in a small business is always a challenge. And small businesses don't have the resources to invest in huge amounts of training so hiring is a perennial problem.
For the documentation reading, even when it is available and quite decent, I noticed that some people still don't read it.
This include doc of official recognized libraries with honestly very good doc. And of course it is the same for internal documentations.
Some people just don't have the "hook" to look for documentation. Maybe a part of that is they worked on projects or company/whatever where the doc was not that helpful but I think also they may take the habit to ask colleagues instant, and that fine, you gain time, but only to a point. If you want to grow yourself, you also need to know way to do things yourself.
Sometimes these things are more complicated than they first appear. It could be a habit, like you say, or there could be hidden benefits to asking for help instead of just reading the documentation. Being on the outside, I could only speculate.
Cheers.
Thanks for this article! I just have a question: how do you fight against this?:
“But sometimes I would stumble upon something I forgot to document and I found it a bit more of a pain to document after the fact.”
The solution I found is Bus Factor Reviews, but do you have any other strategies to keep your documentation up to date and to have feedbacks on this?
Missing documentation is a big problem.
I'm aware of bus factor reviews but I have to admit I've never done one. I work in a small shop and our systems are about as simple as it gets, by design.
I know everyone talks about the value of automation but if you never do 'x' because you automated it with a tool or a script, you'll soon forget how to do 'x' and then if the automation breaks, you are left struggling.
I had an aha moment when I was watching 'Dirty Jobs' on the Discovery Channel. Mike was supposed to change the tire on a big military vehicle. The tire probably weighed 300 pounds and the military tech was doing it all by hand. No air powered torque wrench or pneumatic jack. Just hand tools. At first I thought it was stupid and then it occurred to me that the tech might need to change this giant tire in a war zone, without power or advanced tools. So they practice with simple tools. There's some wisdom there.
But I've gotten off topic. Here's what I do:
If you can't follow my steps, bus factor reviews are probably more important than ever.
Good tips, thanks for sharing!
To add to that, use a company email address for anything work related. Most of the time, someone with access to your email address can get/reset everything else as well. So if something happens to you, they have a way to approach the situation.
Yes. A corporate email account is an excellent idea.
When an employee leaves the company, I've also found it easier to make sure they can't access confidential information if we've used a corporate email address for their dropbox, google docs, google sites, etc.. By changing the password on that one email, we solve a whole class of problems.
Unlike smaller businesses, in many, if not most, larger non-tech corporations, software developers are seen by mid to upper management as cogs that can quickly and easily be replaced by a random contractor. This attitude makes it more likely that developers will not be that conscientious about the kinds of arrangements you've made for business continuity. When you know that you can be sent packing out the door immediately on the whim of a director or VP, it doesn't breed much loyalty to the organization.
However, this "programmer are disposable" can have significant costs and bridges can be burned both ways.
For example, one mid-sized public corporation I worked at decided to lay off an entire team of developers (myself included) complete with the "security escort them out the door" treatment. No thought was given to continuity. The result was a billing automation system failed to run, resulting in $35M in lost revenue before someone discovered the problem and their "hired gun" contractor figured it out and fixed it.
Wow, that's a great story, Frank. No doubt, the manager that fired your team kept his job despite his/her $35M mistake.
Thanks for this. I'm the only developer where I work, so I share a lot of these concerns. My strategy:
Thankfully, my employer has a strong culture of documentation, and has had it since long before I started programming. Everyone's job processes are documented, not just mine.
You're welcome, Brian.
That's for sharing your list. It's nice to give people more than one way to cover their bases.