Python + Flask + Selenium + Chromedriver + Lightsail + SSL

1. Secure the server

# edit the configuration file
sudo nano /etc/ssh/sshd_config

# set these lines, if not already set
PermitRootLogin no
PasswordAuthentication no
ChallengeResponseAuthentication no
PubkeyAuthentication yes

# restart the service
sudo service ssh restart

2. Install dependencies

sudo apt-get -y update
sudo apt-get -y install python3 python3-venv python3-dev
sudo apt-get -y install supervisor nginx git

3. Install the application

Follow repo instructions. Remember to install everything inside the virtual environment.

4. Setup Gunicorn and Supervisor

sudo nano /etc/supervisor/conf.d/transit.conf

# example file
[program:transit-cu]
command=/home/ubuntu/transit-cu/venv/bin/gunicorn -b localhost:8000 -w 3 transit:app
directory=/home/ubuntu/transit-cu
user=ubuntu
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true

5. Enable Lightsail HTTPS rule and create a private address

6. Update A record on DNS

My domain is registered with Godaddy and also created a sub-domain to setup as my server name. Go to your DNS manager page and update or create an A record with your host pointing to the Static IP address provided by Lightsail.

7. Setup NGINX

sudo rm /etc/nginx/sites-enabled/default
sudo nano /etc/nginx/sites-enabled/transit

Replace "transit" with your app name.

8. Setup Let's Encrypt

I decided to install the certificate manually since certbot --nginx failed to reach my domain. However it worked later after I was able to solve the certbot challenge.

sudo certbot -d mydomain.com --manual --preferred-challenges dns certonly

Confirm any required step and then create a TXT record on your domain DNS manager. In my case, It went like this:

The name was _acme-challenge.rsscraper which was different from what cerbot displayed. Actually just the first part of the entire string.

9. Install Chrome (headless) and Chromedriver on Ubuntu Server 18.04

Let's go back into the project dependencies since this is a tricky part, use the following commands to install the latest Chrome version available:

curl https://intoli.com/install-google-chrome.sh | bash

sudo mv /usr/bin/google-chrome-stable /usr/bin/google-chrome

google-chrome --version && which google-chrome

Then you'll have to install the respective chrome driver, go to this site and download it using wget then unzip the package and move to the binaries folder.

cd/tmp/
wget https://chromedriver.storage.googleapis.com/2.37/chromedriver_linux64.zip
unzip chromedriver_linux64.zip
sudo mv chromedriver /usr/bin/chromedriver
chromedriver --version

The above code is just an example.

Make sure the way you use Selenium with the Chrome driver goes as follows:

from selenium.webdriver.chrome.options import Options
options = Options()
options.headless = True
browser = webdriver.Chrome("./chromedriver", options=options)
browser.set_window_size(1920, 1080)

Not sure if the set_window_size is required with this driver but that fixes issues when the browser screen is not able to see other elements because of the responsiveness of the site and your scrapping code might fail when trying to find elements.

The Important part here is that you have to either specify the binary location of the chrome driver or leave it on the root location of the project (like in my case).

You may experience issues due to driver incompatibility with the Chrome version so that's why it was better for me to have it inside the project.

Another possible issue that you could have is related to some lib libraries because we are using a server version that is not connected to a monitor. The issues are related with display and can be easily fixed with a google search.

10. Initializing services

If everything was done correctly with all your project dependencies, you should be able to access to your endpoints using your custom domain name after running the following commands:

// Reload supervisor

sudo supervisorctl reload
sudo supervisorctl status

// Check Nginx syntax and reload

sudo nginx -t
sudo service nginx reload

References

• https://github.com/dandalpiaz/lightsail-flask
• https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-xvii-deployment-on-linux
• http://chromedriver.chromium.org/downloads
• https://medium.com/@praneeth.jm/running-chromedriver-and-selenium-in-python-on-an-aws-ec2-instance-2fb4ad633bb5
• https://stackoverflow.com/questions/43382548/can-i-still-specify-a-path-to-chromedriver-using-chromeoptions-in-python
• https://lightsail.aws.amazon.com/ls/docs/en_us/articles/amazon-lightsail-using-lets-encrypt-certificates-with-nginx#request-a-lets-encrypt-certificate-nginx
• https://www.godaddy.com/community/Managing-Domains/Problem-adding-txt-record-for-letsencrypt/td-p/115376#