I am happy to say that I have learned how to run my most resource-intensive computations on the cloud (that is, on commercial external servers) rather than on my computer. This is huge because of 2 reasons:
- Computational tasks can be kept running in the cloud even after your computer is turned off.
- If more cloud resources are needed, you can upgrade your resources while keeping your same virtual environment at all times (i.e. programs / applications installed on the cloud space)
Without more ado, I will share my bookmarked links which helped me do this in every subtitle below:
- Sign-up to DigitalOcean
- Installing Anaconda python on droplet
- Installing Git on droplet
- Transferring files between droplet and computer
- Use screen to run process on droplet even after turning off your computer or exiting ssh
- How to take a snapshot of created droplet
- Machine Learning - How to Install TensorFlow on droplet
I have used DigitalOcean virtual machines known as droplets for this cloud computing activity. I consider the DigitalOcean platform quite user-friendly and their pricing of droplets is also very reasonable and flexible; the cheapest droplet, for instance, will only cost about $5 a month and if you decide to only use it for 2 days, the price would be around $(2/30) x 5 = 33 cents TOTAL and not $5 (as long as the droplet is destroyed after the 2 days, see more on this below) You may click on the link above to sign up
Once you have the droplet, I recommend installing the Anaconda suite which comes with several python modules pre-packaged. The tutorial above allows you to do so with command lines alone.
Git may already be installed in your DigitalOcean droplet, but above is the tutorial nonetheless. Git can be used to communicate with your Github repository, so that Python programs developed in your computer can be pulled to your droplet, and viceversa. (P.S. You may also need a Github account)
This approach uses Filezilla to do this. I use this in the case my programs generate files and I need to transfer them from droplet to computer.
Screen is a program which allows you to do just that. You may need to install it INSIDE your droplet (i.e. ssh protocol) and then call "screen". After calling screen you can run your program (e.g. "python myprogram.py") and then press "Ctrl+A", followed by letting go off the keys, followed by pressing "D" to detach the screen.
Then you may exit the droplet or even turn off the computer and the python process will keep on running! To access again, access the droplet, then type "screen -r" to re-attach the screen and see how far your process has been running on the server.
In case you have been using an expensive droplet and you do not want to be charged for it for a period of time, you may need to destroy it. No worries! If you want to "revive" your droplet later on preserving all its installed programs and environment*, you may do so by taking a snapshot. Click on the link above for this option.
*I have not explored this option yet, but I've done some reading on it, so this is how it should go.
If you're ready to take your droplet to the next level, machine learning, then this tutorial may be just for you. The current recommendations are to install TensorFlow in a droplet with at least 4GB RAM.
Let's say you created your whole development environment on a smaller droplet. That's ok. You may take a snapshot of the smaller droplet and then re-execute it on a larger memory droplet. This strategy may also be useful if you would want to take time setting up your development platform on a smaller droplet to spend less money before moving to the larger one.
DigitalOcean droplets are designed to be suitable for developers and small applications. Their application to cloud computing goes beyond python, as it can be extended to as many programming languages as can be run on a computer.