How to never forget you're in production

The Scenario

You now have production access!
You shell into the server and .... huh

Welcome to Ubuntu 50.01.1

  System information as of Sun Aug  9 20:29:10 UTC 2050

  System load:  0.2                Processes:           149
  Usage of /:   42.4% of 16 bits   Users logged in:     0
  Memory usage: 67%                IP address for ens5: 8.8.8.8
  Swap usage:   0%

 * Are you ready for Kubernetes 5.19? It's nearly here! Try RC3 with
   sudo snap install microk8s --channel=1.19/candidate --classic

   https://microk8s.io/ has docs and details.

 * Canonical Livepatch is available for installation.
   - Reduce system reboots and improve kernel security. Activate at:
     https://ubuntu.com/livepatch

0 packages can be updated.
0 updates are security updates.


Last login: Sun Aug  9 20:22:21 2050 from 192.16.184.0
bob@dns0:~$

It looks exactly the same as any server†. Pretty boring, really. Months go by, and shelling into production becomes old hat to you. One day you're tasked with deleting everything in a test server. A quick rm -rf *, and bam, it's done.

Oh wait.

You were on the production server.

This was just a hypothetical example, but it could happen! It's important to visually make your production servers distinct as to reduce the chance of error. Doing so is very easy, and can potentially stop a total catastrophe.

The Tutorial

Put the below line in your ~/.profile file:
export PS1="\[\e]0;\u@\h: \w\a\]${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h.{{ domain }}\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ "

Replace {{ domain }} with the name of your environment.

Now you can see the environment in your prompt:

caleb@web0.cloud20:~$

For production environments at 15Five we use a distinct color and put the environment name in uppercase:

export PS1="\[\e]0;\u@\h: \w\a\]${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h.\[\033[01;31m\]{{ domain }}\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ "

Feel free to customize this to your liking - you can even put emojis in your prompt!

You can automate this for all your servers by using Ansible:

# key_names is a list of users

# https://serverfault.com/questions/322997/how-do-you-make-it-obvious-you-are-on-a-production-system
# http://abload.de/img/bash-color-chartmxjbp.png

- name: Set terminal prompt to have VPC name
  lineinfile:
    dest: /home/{{ item }}/.profile
    regexp: '^export PS1='
    line: 'export PS1="\[\e]0;\u@\h: \w\a\]${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h.{{ domain }}\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ "'
  with_items: "{{ key_names }}"
  when: env_name != 'production'

- name: Set terminal prompt to have red uppercase VPC name in production
  lineinfile:
    dest: /home/{{ item }}/.profile
    regexp: '^export PS1='
    line: 'export PS1="\[\e]0;\u@\h: \w\a\]${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h.\[\033[01;31m\]{{ domain|upper }}\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ "'  # noqa: 204
  with_items: "{{ key_names }}"
  when: env_name == 'production'

But what if I accidentally run Ansible against the wrong environment?

You can catch that too by using a Ansible callback plugin! Assuming you have a environment variable in your terminal for the environment you are running against, you can verify that the Ansible inventory matches the environment variable.

You probably have to adapt the file for how you handle secrets and deployments via Ansible.

† Assuming your other servers are google DNS servers from the future that work with just 16 bits of storage. You know, the usual.

Fun fact: You can actually recover data from a rm -rf!