Installing Sendy as a replacement for Mailchimp — 2019 version 4 update, the complete guide, with automated backups to S3, and https configuration
This information is compiled from many sources and is the complete record of my successful install. Start here to save yourself a lot of time.
This install guide you are now reading is for Sendy version 4. It was tested on Sendy version 4.0.2 with the latest server config available on Digital Ocean
The version 3 install guide is outlined in my previous article. It is nearly identical to this install guide, except it was tested on version 3.1.2.3
Intro and motivation for the change
I had a new-music-release email list hosted on Mailchimp to send infrequent updates. I learned the hard way that if a basic account is unused for two years, it’s deleted and the old username is permanently blocked. Just vaporized. No notification sent before the deletion, and no easy way to get the unsubscribe data back either. According to Mailchimp’s compliance department, this is a “security measure”. I still had a backup of the original subscriber emails, but not the important records of who previously unsubscribed from my list over the years. So while I could easily reimport my list again with Mailchimp or another host, there is a “spammer blacklist” risk related to sending email to someone who had unsubscribed in the past.
After jumping through hoops with customer service, I was able to get an export of all my previous Mailchimp data, but I was convinced that it makes more sense to find an alternative to the industry heavies such as: Mailchimp, Sparkpost, ActiveCampaign, AWeber, or Emma.
Fortunately, I had recently scanned the archives of Wes Bos’ web-dev blog posts, and remembered the catchy title “From MailChimp to Sendy — how I saved $2,400/year on my email list”. With my relatively small email list, the change wasn’t motivated by saving money, but by hosting my own list to avoid another surprise account deletion or other issue related to not controlling my own data. Wes’ article convinced me to check out Sendy. It’s a self-hosted mailing list software solution. But his article didn’t describe how to install it.
After purchasing it and combing through the product support forums, it looked like DigitalOcean was a great choice for hosting based on both price and features. While reading many, many pages of documentation and stack overflow posts,
I took detailed notes on my installation process so I could present a complete and detailed record of the steps you can take to get a running instance of Sendy.
I will also cover how to secure your Sendy server traffic with HTTPS with a free Certbot certificate, and set up automated mysql database backups to an AWS S3 bucket at the frequency of your choosing.
How to — start here
- Read about and purchase Sendy from the developer here.
- If you don’t already have an account with DigitalOcean, signing up through this link will get you $100 in free credit to use over 60 days.
- You will need an AWS account.
Since the developer of Sendy says “if it doesn’t work out, we’ll refund you”, you can try this whole experiment without financial risk. I set up my Sendy installation on a $5/month standard droplet. The only other costs are the Amazon SES e-mailing, which is $0.10 per thousand e-mails, and your Sendy software purchase which is a one time fee with no additional costs until major version updates.
Caveats
Nowhere again in this post will I use the word ‘simple’. But I do believe that following this guide will make your installation process as easy as possible.
I recommend browsing it first, and then following the steps in the order they are presented. Make sure you have extra hairs available in case you need to pull some out.
I reinstalled Sendy by following and validating all of the instructions that I’ve documented here.
The only constant is change, and you may find yourself needing to jump through some unexpected hoops, just as I did during my initial install process. Hopefully everything that I have documented will be current and useful for you. If you found something that has changed, please describe it in the comments and I will update accordingly.
I have made extra sure to implement a quality guide that is up to modern standards by making sure to use some instances of dry humor, at least one sci-fi movie reference, and occasional emoji insertions. As an added bonus, I used the example domain www.contoso.com throughout the guide. I also documented the actual keys/secrets, and ip addresses I used while setting this up so you won’t have the extra mental baggage of trying to parse the syntax of things like .
I’ve put in the extra effort to describe various scenarios you may encounter, and also to describe why something is done so you’ll be able to make informed choices if some step has changed since I documented my process.
This is totally doable , so read this guide, warm up your google chops, and I wish you the best on your installation process!
It may be helpful to breathe deeply, drink water, and walk around every once in a while.
Documentation conventions
Text inside a block like this represents something you should type or something visible on the browser page or terminal display.
Text inside a box like this represents the terminal display
$ This represents your local terminal (after $)
# This represents the remote shell (after #)
Sail the DigitalOcean: create your droplet (cloud based virtual machine)
Log into your new account, and verify that you have the free $100 credit. If you are a new customer but you did not sign up through a referral link, send an email to customer service with the referral link. In my experience, they will credit you. I signed up right after finding somebody else’s referral link, and when I e-mailed it to them, they gave me the credit. If you need a referral link, here is mine: https://m.do.co/c/c29fea8989aa
Create a new ‘project’
The next window will ask you: Move resources into the new project? click Skip for now
Next, create your droplet and I will detail the settings below:
Click the blue button, or the green drop down
The next set of options is Choose an image . Select Marketplace
At the time of writing, the LAMP Marketplace app was version 18.04. The number is the version of Ubuntu (flavor of linux distribution). Newer versions (higher numbers), if available, may present a similar installation experience. If you have some experience working with the command line and aren’t afraid to google things, go ahead and install with the newer version.
Choose a size. For my currently puny mailing list, I’m confident with the Standard plan and the smallest size virtual machine and I chose $5/mo. In the future, you can scale cpu/ram up and down as you wish. You can scale disk size up, but not down again.
Add backups? I chose not to, since you can make and store manual snapshots. However, automating backups on a $5/month container will only cost you $1/month, so you may opt for this in order to save having to remember to make snapshots.
Add block storage? Not required for this tutorial
Choose a data center region. I chose New York. You may have legal obligations to host your data in a particular region depending on the type of data you are storing, so choose accordingly.
Select additional options? I left all of these unchecked
Add your SSH keys. I did not do so, and therefore use a password when I sign in via SSH. The initial root password is sent via email. This tutorial discusses using the password, so skip that step for now. SSH keys allow you to log into the remote server without needing to authenticate with a password.
Finalize and create. Make sure that you are only deploying 1 droplet, and rename it something sane (such as sendy-droplet in my example below), and make sure it is in your newly created ‘project’ (which it will be, unless you already have multiple ‘projects’)
Click the “Create” button. You will then be redirected to your ‘project’ page, and see a status bar of your droplet creation progress. It takes a couple minutes.
Followed by its successful creation, your login details and password have been e-mailed to you.
Congratulations, your newly spawned creation is running in the cloud. Now perform a sanity check by entering the ip address into your browser and you should see something like the following:
The “Quick”start Guide button points to this document on the Digital Ocean website, which details what software your One-Click installation contains, as well as some configuration information.
Let’s log into the new droplet and set it up!
You’re going to need the email you just received from DigitalOcean with the root password for your droplet. We will login with ssh (secure shell).
$ ssh root@157.230.133.78
Which will be greeted by the following appropriately paranoid message
The authenticity of host '157.230.133.78 (157.230.133.78)' can't be established.
ECDSA key fingerprint is SHA256:0U9apoBN+7h1ajk2+KZoYQiL35X4Zjp+9MQLC71mX7y.
Are you sure you want to continue connecting (yes/no)?
You do want to continue connecting, so type yes and you should then see the following:
Warning: Permanently added '157.230.133.78' (ECDSA) to the list of known hosts.
root@157.230.133.78's password:
The terminal will not show the characters you enter when you type or paste the password. Enter the password and then press return and you should be logged in and see some boilerplate text describing some of the default configurations of your server.
You will be prompted immediately to reset the root password. First enter the current password that you just used. Then enter a new secure password that you immediately record somewhere safe. You will need to enter your new password again right away to confirm, and then you will be logged into your droplet, a Linux Ubuntu virtual machine. The normal pwd ls cd mkdir etc terminal commands work as you would expect.
Update the list of available software
# apt-get update
will update the list of available packages and their versions, but not install or upgrade any packages. It will create output similar to what is shown next:
Get:1 [http://security.ubuntu.com/ubuntu](http://security.ubuntu.com/ubuntu) bionic-security InRelease [83.2 kB]
Hit:2 [http://ppa.launchpad.net/certbot/certbot/ubuntu](http://ppa.launchpad.net/certbot/certbot/ubuntu) bionic InRelease
\*\*\*[Lots more stuff …]\*\*\*
Get:37 http://mirrors.digitalocean.com/ubuntu bionic-backports/main Translation-en [448 B]
Get:38 http://mirrors.digitalocean.com/ubuntu bionic-backports/universe amd64 Packages [3468 B]
Fetched 14.0 MB in 5s (3035 kB/s)
Reading package lists... Done
Upgrade available packages
apt-get upgrade
Accept the confirmation message and you will be updated to the latest version of all installed packages.
Install some new software needed by Sendy
Next we need to install a couple software packages that are not part of the default 1-click installation, but will be needed by Sendy. The first one is php-curl
figure out which major.minor version of curl you need
php --version
will return a bunch of output including the (major.minor.patch) version that is installed on your server. In my case it’s 7.2.10 which we can use in the next command, but we only need the [major.minor] numbers.
# apt-get install php7.2-curl
If in doubt what version to install, type # apt-get install php7. and then press tab a couple times to see an auto completion of the various software packages that are available for the major version of php that you are using. Mine auto-completed with # apt-get install php7.2-curl and then I clicked enter to install and also accepted the confirmation dialogue that may pop up.
Here’s the next one to install:
# apt-get install php-xml
accept the confirmation dialogue that may pop up and then restart the droplet’s web server with:
# systemctl restart apache2
Next, Create New Virtual Host Files
Virtual host files are the files that specify the actual configuration of our virtual hosts and dictate how the Apache web server will respond to various domain requests.
In English: we need to set up the server to respond to the domain name that you are going to assign to it when you create a DNS record with your domain registrar. Example: if you run the website www.contoso.com, and you plan to install Sendy at sendy.contoso.com, you need to configure the apache web server on the droplet to serve the appropriate files when you visit the sendy domain. These files are hosted in /var/www/
Create The Directory Structure for the files to be served
# mkdir -p /var/www/contoso.com/public\_html
Create New Virtual Host Files
These are the files that tell the Apache web server how to respond to web requests. We are going to copy and configure the default host file (named 000-default.conf). The following shell command is one line
# cp /etc/apache2/sites-available/000-default.conf /etc/apache2/sites-available/contoso.com.conf
Now are going to edit this file using nano or vim. If you’ve never used vim, I would not start now. Nano is fairly user-friendly and responds to arrow key movements to move the cursor around the screen just like a familiar text editor. After editing, press control and x together to save the file and return to the terminal.
# nano /etc/apache2/sites-available/contoso.com.conf
The file that appears should begin with
<VirtualHost \*:80>
and end with
</VirtualHost>
Don’t worry too much about what’s in the middle. It may vary a bit from what you see below.
Leave everything else as it is. We are going to add one directive (line of configuration code) after the tag:
ServerName sendy.contoso.com
and modify two directives (lines of configuration code). First, change theDocument Root directive to:
DocumentRoot /var/www/contoso.com/public\_html
and then remove the word “Indexes” from the line that looks like this:
Options Indexes FollowSymLinks
so that after you edit it looks like this:
Options FollowSymLinks
And then also modify the text inside the tag to:
<Directory /var/www/contoso.com/public\_html/>
The final result should be something similar to the following. Don’t worry about all the other lines matching exactly:
<VirtualHost \*:80>
ServerName sendy.contoso.com
ServerAdmin webmaster@localhost
DocumentRoot /var/www/contoso.com/public\_html
<Directory /var/www/contoso.com/public\_html/>
Options FollowSymLinks
AllowOverride All
Require all granted
</Directory>
ErrorLog ${APACHE\_LOG\_DIR}/error.log
CustomLog ${APACHE\_LOG\_DIR}/access.log combined
<IfModule mod\_dir.c>
DirectoryIndex index.php index.pl index.cgi index.html index.xhtml index.htm
</IfModule>
</VirtualHost>
This is what mine looks like, but I’m installing Sendy as a subdomain of my personal website domain. If you plan to run Sendy as a standalone server (something like www.contoso.com),,) then your final result should probably be: (notice the extra directive of ServerAlias)
################# WHEN NOT INSTALLING AS A SUBDOMAIN
<VirtualHost \*:80>
ServerName contoso.com
ServerAlias www.contoso.com
ServerAdmin webmaster@localhost
DocumentRoot /var/www/contoso.com/public\_html
<Directory /var/www/contoso.com/public\_html/>
Options FollowSymLinks
AllowOverride All
Require all granted
</Directory>
ErrorLog ${APACHE\_LOG\_DIR}/error.log
CustomLog ${APACHE\_LOG\_DIR}/access.log combined
<IfModule mod\_dir.c>
DirectoryIndex index.php index.pl index.cgi index.html index.xhtml index.htm
</IfModule>
</VirtualHost>
If you edited using nano, when you use control + x to write and exit, you will confirm Save modified buffer? with Y and then enter again to accept the File Name to Write.
Activate The Virtual Host File (and deactivate the default host file)
# a2ensite contoso.com.conf
this will show some confirmation message including a notification about restarting apache. We will do that in a moment.
First you need to deactivate the default virtual host file with:
# a2dissite 000-default.conf
this will also show some confirmation message.
And as the confirmation messages stated, it is now time to reload the Apache server to enable the new Apache configurations.
Reload the Apache server
# systemctl reload apache2
This reloads the Apache web server only , and you should remain logged into the shell.
If you visit the ip address now, you should see something like the following:
Which is what we want. This is the result of removing Indexes from the Options directive inside of the tag.
**If you see something like this:**…
…then the Options directive wasn’t properly updated. Check the syntax above. After you correct it, restart Apache with # systemctl reload apache2
When you reload the browser, it should show the correct Forbidden message. This is going to keep the public from peering into your /uploads folder after you install Sendy.
Regarding HTTPS traffic to the droplet
Since the 1-click droplet is preconfigured with Certbot, securing your servers web traffic with SSL can be done with just a few commands. However, before we generate a certificate for the server, we need to set up DNS routing to the server so that we make sure we’re generating the certificate for the right domain. Detailed HTTPS instructions will follow, at first we will cover how to get the proper DNS records configured for your droplet.
DNS Routing
Log into your domain name registrar so that we can create some records that will point to your new server.
In my case, for this example server, I would be adding a new record for the DNS records of my domain contoso.com. I would create an “A” record. The A name record would be: sendy, and the value: 157.230.133.78
I just tested this out with my registrar, and within a couple minutes my subdomain resolved to the server and I saw the same image as when accessing the ip address directly.
If you plan to install a standalone server without a subdomain, then the name of your “A” record would be www, and you would create a second “A” record for the bare root domain (contoso.com in this example). The syntax for this differs by registrar, and may be “@”, or sometimes just a blank field. The value of both of these “A” records would be the ip address of your droplet.
Ideally it will only take a few minutes for your DNS changes to propagate. Feeling impatient or wondering if it worked? Check this out (and update it with your own domain name):
[https://www.whatsmydns.net/#A/sendy.contoso.com](https://www.whatsmydns.net/#A/sendy.contoso.com)
This is a pretty cool resource that shows you the status of the DNS propagation. My changes made it around the world in just a few minutes.
Enable HTTPS with SSP certificate installation with Certbot
As part of your 1-click installation, there is an automated SSL certificate generator already installed called Certbot. Now that we have the virtual host files set up correctly, and DNS records correctly configured we can run it.
For my example server, I only need to generate a certificate for the sendy subdomain, and this is what I would run:
# certbot --apache -d sendy.contoso.com
If you are running it for a standalone Sendy installation, you will run it as follows (for the www subdomain, and root domain):
# certbot --apache -d contoso.com -d [www.contoso.com](http://www.contoso.com)
After you run the command, you will be asked
Enter email address (used for urgent renewal and security notices) (Enter 'c' to cancel):
I personally consider entering an email address that you actually check a very good idea so that you can receive notifications in case the automated SSL certificate renewal process doesn’t work. If your certificate expires, browsing to your formerly secure site will not show your site, but instead a “Your Connection Is Not Private” warning page.
If you choose to generate your certificate without the email address you can do so with the command that’s given after you press c at that prompt.
After this step, you will be given the terms of service link and asked to agree. If you agree, you will be asked whether or not you want to share your email address with EFF.
After you answer that, you will see a bunch of output related to the installer trying to access your domain. If your DNS propagation has not completed, or there was an error with how you entered your DNS records, this step will fail.
The output so far should be something like:
root@sendy-droplet:~# certbot --apache -d sendy.contoso.com
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for sendy.contoso.com
Enabled Apache rewrite module
Waiting for verification...
Cleaning up challenges
Created an SSL vhost at /etc/apache2/sites-available/contoso.com-le-ssl.conf
Enabled Apache socache\_shmcb module
Enabled Apache ssl module
Deploying Certificate to VirtualHost /etc/apache2/sites-available/contoso.com-le-ssl.conf
Enabling available site: /etc/apache2/sites-available/contoso.com-le-ssl.conf
Please choose whether or not to redirect HTTP traffic to HTTPS, removing HTTP access.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: No redirect - Make no further changes to the webserver configuration.
2: Redirect - Make all requests redirect to secure HTTPS access. Choose this for
new sites, or if you're confident your site works on HTTPS. You can undo this
change by editing your web server's configuration.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate number [1-2] then [enter] (press 'c' to cancel):
At this point you are asked whether or not you also want to allow HTTP traffic to your server, or redirect it to HTTPS. I selected 2 and had the installer create a redirect from HTTP to HTTPS. When it comes time to configure Sendy, you can specify whether or not you prefer HTTP or HTTPS. Your Sendy configuration file must specify one or the other.
Here is the remaining output:
Enabled Apache rewrite module
Redirecting vhost in /etc/apache2/sites-enabled/contoso.com.conf to ssl vhost in /etc/apache2/sites-available/contoso.com-le-ssl.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Congratulations! You have successfully enabled
https://sendy.contoso.com
You should test your configuration at:
https://www.ssllabs.com/ssltest/analyze.html?d=sendy.contoso.com
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
IMPORTANT NOTES:
- Congratulations! Your certificate and chain have been saved at:
/etc/letsencrypt/live/sendy.contoso.com/fullchain.pem
Your key file has been saved at:
/etc/letsencrypt/live/sendy.contoso.com/privkey.pem
Your cert will expire on 2019-04-12. To obtain a new or tweaked
version of this certificate in the future, simply run certbot again
with the "certonly" option. To non-interactively renew \*all\* of
your certificates, run "certbot renew"
- If you like Certbot, please consider supporting our work by:
Donating to ISRG / Let's Encrypt: [https://letsencrypt.org/donate](https://letsencrypt.org/donate)
Donating to EFF: [https://eff.org/donate-le](https://eff.org/donate-le)
If you want to feel extra secure that this worked, go ahead and follow the link to the test site that was just output after the You should test your configuration at: phrase. For this site it would be: https://www.ssllabs.com/ssltest/analyze.html?d=sendy.contoso.com
Now refresh or navigate to your server url, with your browser (go to your site, such as https://www.contoso.com, NOT the ip address), and you should see the:
Your secure server is up and rolling!
Let’s take a look at the ssl site configuration .conf file and make sure it properly copied over the directives from the original non-ssl version.
# nano /etc/apache2/sites-available/contoso.com-le-ssl.conf
Should bring up a page that looks pretty much like this. Look to see that the ServerName / DocumentRoot and Directory directives look similar. And also that the Options directory doesn’t have the Indexes keyword. If this .conf file wasn’t copied properly from the http version, you are currently in the nano editor, and can make the necessary changes now. Exit out of the editor with ctrl + x and you will return to the ssh terminal.
<IfModule mod\_ssl.c>
<VirtualHost \*:443>
ServerName sendy.contoso.com
ServerAdmin webmaster@localhost
DocumentRoot /var/www/contoso.com/public\_html
<Directory /var/www/contoso.com/public\_html/>
Options FollowSymLinks
AllowOverride All
Require all granted
</Directory>
ErrorLog ${APACHE\_LOG\_DIR}/error.log
CustomLog ${APACHE\_LOG\_DIR}/access.log combined
<IfModule mod\_dir.c>
DirectoryIndex index.php index.pl index.cgi index.html index.xhtml index.htm
</IfModule>
SSLCertificateFile /etc/letsencrypt/live/sendy.contoso.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/sendy.contoso.com/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>
</IfModule>
Next is the initialization of the mysql database and then the upload of Sendy to your server.
Installing Your Mysql Database
Surprise…it’s already installed. Welcome again to the joy of the 1-click droplet installation. We just need to create the Sendy database, a database user/password, and grant the new database user privileges to use the Sendy database.
Login into mysql with # mysql and you should see something like:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.7.26-0ubuntu0.18.04.1 (Ubuntu)
Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql>
and create your Sendy database (don’t forget the ; at the end)
mysql> create database sendy;
and you should see:
Query OK, 1 row affected (0.00 sec)
Now we need to create a database user with a password, and grant them permission to use the Sendy database (remember the ;):
Come up with a username and password and record it somewhere secure. You’re going to need it again soon.
mysql> CREATE USER 'sendy\_db\_admin'@'localhost' IDENTIFIED BY '2zrQYe5NC8z2sw87TpC4SWDK';
which should respond with:
Query OK, 0 rows affected (0.00 sec)
Now we grant them privileges (remember the ; at the end):
mysql> GRANT ALL PRIVILEGES ON \*.\* TO 'sendy\_db\_admin'@'localhost';
again which should respond with:
Query OK, 0 rows affected (0.00 sec)
Our work is done here. Exit the mysql command line interface with
mysql> exit;
and the terminal will say Bye. Now you are back at the prompt for the root user inside the droplet.
You now have your database name, authorized user and password, and you can set up your Sendy configuration file.
Setting Up Your Sendy Configuration File
Assuming you have purchased and downloaded Sendy, unzip the file and edit /includes/config.php in your code editor.
This is the part that we’re going to modify:
/\* Set the URL to your Sendy installation (without the trailing slash) \*/
define('APP\_PATH', 'http://your\_sendy\_installation\_url');
/\* MySQL database connection credentials (please place values between the apostrophes) \*/
$dbHost = ''; //MySQL Hostname
$dbUser = ''; //MySQL Username
$dbPass = ''; //MySQL Password
$dbName = ''; //MySQL Database Name
The relevant values for the install in this tutorial would be:
/\* Set the URL to your Sendy installation (without the trailing slash) \*/
define('APP\_PATH', 'https://sendy.contoso.com');
/\* MySQL database connection credentials (please place values between the apostrophes) \*/
$dbHost = 'localhost'; //MySQL Hostname
$dbUser = 'sendy\_db\_admin'; //MySQL Username
$dbPass = '2zrQYe5NC8z2sw87TpC4SWDK'; //MySQL Password
$dbName = 'sendy'; //MySQL Database Name
'localhost'will be the Hostname regardless of the other values. Enter the username/password for the database user we just created in the last step. And enter the database name you entered in the previous step, which in the demo was'sendy'.
Notice: since we installed the SSL certificate, the ‘APP_PATH’ value has been set to https, and there is no trailing /after the .com
Did you enter a different domain when you purchased Sendy than what you are going to end up using? No problem, just update your domain at the “ Changing your licensed domain” link sent to you in your purchase confirmation email. I’ve done this a couple times and the update was immediate.
It’s now time to upload Sendy to your server.
Uploading Sendy To Your Server
You can use sftp or rsync. Sftp is probably more intuitive with relatively simple documentation, so I will show the command for rsync. The following command will be issued from a new terminal window, not the ssh shell we’ve been using. The command should be entered as one line. The -ahrvz flags mean “archive/human-readable/recursive/verbose/compressed”. The first folder path is for your local Sendy folder (remember to include the trailing /) and the second location is the DocumentRoot web content folder inside your droplet, which we specified in the virtual host file we edited previously. You will be prompted to supply your droplet root password after issuing this command:
IMPORTANT: the following command must be entered in a new terminal window, not in the SSH shell that we have been issuing commands to the server with, (as specified by the $ at the beginning of the command)
$ rsync -ahrvz -e ssh /path/to/your/sendy/folder/ root@157.230.133.78:/var/www/contoso.com/public\_html/
This will be followed by a confirmation such as:
root@157.230.133.78's password:
After you successfully enter the root password, you will see a massive file list of everything that will be transferred to the server. The listed order of the files may be different, but it should start with something like:
root@157.230.133.78's password:
building file list ... done
./
.htaccess
\_compatibility.php
\_install.php
app.php
autoresponders-create.php
autoresponders-edit.php
autoresponders-emails.php
and end with something like:
locale/en\_US/
locale/en\_US/LC\_MESSAGES/
locale/en\_US/LC\_MESSAGES/default.mo
locale/en\_US/LC\_MESSAGES/default.po
uploads/
sent 5.62M bytes received 18.78K bytes 1.00M bytes/sec
total size is 13.41M speedup is 2.38
$
IMPORTANT: now we’re going to issue commands to the SSH shell to the server again
After uploading, you need to make sure that the permissions of the uploads folder is correctly set to 0777 . This can be achieved with
# chmod 0777 /var/www/contoso.com/public\_html/uploads/
. And you can check if this worked with
# ls -al /var/www/contoso.com/public\_html/uploads/
You should see something like
total 8
drwxrwxrwx 2 501 staff 4096 Jan 12 00:25 .
drwxr-xr-x 10 501 staff 4096 Jan 12 04:31 ..
The important permission bits are the drwxrwxrwx on the same line as the .
The other line relates to the permissions of the parent folder.
If everything has gone according to plan so far, when you reload your browser that is pointing to your new server, you should be redirected to /_install.php.
If you see the following, you’re walking the happy path:
If you don’t, don’t worry. It’s also possible that you might see something like the following:
Not to worry, no matter what is missing, it’s easy to correct. Following the instructions, we navigate to: /_compatibility.php?i=1 and see what needs to be installed. Your results may vary slightly, but likely not too much if you have followed along with this this guide from the beginning.
Whatever is missing, search the support forum at: sendy.co/forum/ by entering the error into the search box to figure out how to remedy it.
After hunting down any possible remaining packages, your server compatibility checklist should all be green, score 10/10 and navigating to your Sendy url should show you the login page…
…or possibly this:
If this is what you see, you will need to make sure that the domain that you entered as your APP_PATH variable in /includes/config.php matches the domain that you are currently pointed to with your browser, as well as being the domain that you entered when you purchased the product. Domain changes can be accomplished easily. The link for that form is in the email you receive when purchasing the software. In the form, you enter your license key and in theChange domain to field, enter the updated domain. You do not need to preface it with http(s) when you enter the domain (see below). However you do need the http(s):// preface when defining the APP_PATHvariable in the Sendy config file.
What I would need to enter in that form to enable Sendy for the examples here would be:
For the rest of the setup of Sendy, I leave you to the capable hands of the official getting started documentation. You can pick up at step 5. There is very important information regarding getting your Amazon Web Services IAM credentials set up properly, as well as information on increasing your SES Sending Limits. This usually takes at least 2 days for a new account, so if you are browsing through this documentation and planning on installing this software in the near future:
I recommend you actually handle this part now so that when you are ready to install you don’t have to deal with the speed bump of waiting for AWS to approve your increased sending limits.
One related tip that I will share is that the first time you log into your Sendy dashboard, you might see this, even if you already had your SES limit raised :
If you know that you are no longer in “Sandbox mode”, the fix is to go into /settings and make sure that this is set properly:
So what’s left? Let’s set up automatic backups of your database, and automatic transfers of the backups to an AWS S3 container!
Things Go Wrong. Protect Your Valuable Mailing List Data By Setting Up Automatic Backups And Automating Cloud Storage Of Those Backups.
Setting Up Automatic Mysql Dumps
Before we set up the scheduler, let’s create a storage directory for the backups and backup script. This command:
# mkdir /srv/db\_bak/
will create the empty folder where we will have our database backups dumped into.
Let’s create our actual backup script with the nano editor again:
# nano /srv/dump\_script.sh
Paste the following into the editor and press Esc and then $ to get the lines to word-wrap so you can see everything on the screen at once. For the first chunk, you will substitute the database username and password you created previously. The second line copies the uploads folder to your backup directory. It’s good to have a backup of that, as any attachments or images that you upload via Sendy when creating campaigns will be stored in this folder, and the e-mails you send out will be served from your droplet. In case you are sending out massive email campaigns, you will probably want to link to images and uploads served via a CDN, or S3 bucket, and not put undue stress on your server.
/usr/bin/mysqldump -u'sendy\_db\_admin' -p'2zrQYe5NC8z2sw87TpC4SWDK' sendy > /srv/db\_bak/$(
date +%F)\_sendy\_backup.sql
cp -r /var/www/contoso.com/public\_html/uploads/ /srv/db\_bak/
now control + x to exit, and confirm you want to save the changes, and confirm the file name. Now to configure the scheduler.
There is a fantastic utility that lives in the Unix-like world, including your droplet, called Cron. We’re going to use it to set up two jobs. One will take periodic snapshots of your database. The other will periodically transfer those snapshots to a secure cloud storage bucket. I’m going to show you how to transfer those backups to an AWS S3 bucket.
Initially there is no user configuration file, known as a crontab, for the cron scheduling utility on your computer. Here’s how to create one:
# crontab -e
which will probably be followed by:
no crontab for root - using an empty one
Select an editor. To change later, run 'select-editor'.
1. /bin/nano <---- easiest
2. /usr/bin/vim.basic
3. /usr/bin/vim.tiny
4. /bin/ed
Choose 1-4 [1]:
Unless you already know basic vim commands, go ahead and select the default nano editor by pressing 1 and then enter. You will be greeted with this screen which shows the default file which is entirely commented out with “helpful” boilerplate
# Edit this file to introduce tasks to be run by cron.
#
# Each task to run has to be defined through a single line
# indicating with different fields when the task will be run
# and what command to run for the task
#
# To define the time you can provide concrete values for
# minute (m), hour (h), day of month (dom), month (mon),
# and day of week (dow) or use '\*' in these fields (for 'any').#
# Notice that tasks will be started based on the cron's system
# daemon's notion of time and timezones.
#
# Output of the crontab jobs (including errors) is sent through
# email to the user the crontab file belongs to (unless redirected).
#
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 \* \* 1 tar -zcf /var/backups/home.tgz /home/
#
# For more information see the manual pages of crontab(5) and cron(8)
#
# m h dom mon dow command
Let’s erase that boilerplate and replace it with this:
# backing up to /srv/db\_bak/
# every day at 8:12 utc, dump database
12 8 \* \* \* . /srv/dump\_script.sh > /dev/null 2>&1
Crontable scheduling syntax gets no awards for readability. Check out this site if you want to see more details about what those obscure symbols mean. https://crontab.guru/#12_8_*_*_*
This will run our database backup script every day at 08:12 UTC, which is currently just past midnight here on the West Coast of the US. I picked that time because it’s unlikely that there will be many people interacting with my site in a way that will access the database at that time, and I wanted it to not be exactly on a 5-minute interval since that’s when other scripts will run. Perhaps not necessary to schedule like that, but that’s how I was thinking about it.
You can feel free to set that interval to be more frequent than every day, since the way it’s written, it will overwrite the previous file as long as the current day of the system date is the same. It is going to be saving the filenames as…
db\_bak/2019–05–03\_sendy\_backup.sql
…auto-updating the file prefix as the date changes.
I’m guessing you might want to see this actually work and not have to wait a day. If you want to update the cron scheduling interval temporarily to see if it works, change the first part of the third line to */2 * * * * and it will run every two minutes. Also don’t forget the . after the last *. Without that, the dump script won’t run. When you save the file, it will ask you to confirm the save, and then confirm the file name. Confirm the temporary file name that it shows you, and after you accept, it will install a new crontab that you can access again for edits and updates with crontab -e. You can also inspect your crontab at any time with crontab -l. You can also change your preferred crontab editor with select-editor.
If you remove the > /dev/null 2>&1 part from the end of the cron script, the output of the job will generate a system mail message each time it runs. You can check it with cat /var/mail/root and delete the mail messages with rm /var/mail/root.
Since you’ll be back in your Crontab file soon after you set up Sendy, I will save you some time and give you the code for the cron jobs that you are going to be setting up in order to schedule campaigns, activate auto responders, and import subscription oriented csv files. Feel free to enter these now, or wait until you get into configuring your Sendy dashboard.
# Five minute cron job to send scheduled campaigns
\*/5 \* \* \* \* php /var/www/contoso.com/public\_html/scheduled.php > /dev/null 2>&1
# One minute interval job for importing csv
\*/1 \* \* \* \* php /var/www/contoso.com/public\_html/import-csv.php > /dev/null 2>&1
# One minute interval job for activating auto responders
\*/1 \* \* \* \* php /var/www/contoso.com/public\_html/autoresponders.php > /dev/null 2>&1
Part 1 of the automation tasks complete! Next is getting this all transferred automatically to your S3 bucket.
Setting Up Automatic Transfers Of The Mysql Backups To The S3 Bucket
First we need to create the AWS S3 bucket that we’re going to send the transfers into. From the S3 console, create a new bucket. The Region is arbitrary, unless you have a specific need to host your data in a particular region.
Then click next
I didn’t need to change anything in the “configure options” panel.
But you need to “Set permissions” appropriately so that you will be able to upload to the bucket
Review your options and press:
Then select your bucket, select Permissions , and set the following Access Control List:
Then select the “Overview” tab and create a folder
Keep the bucket and folder name handy, you will need them soon.
And that’s it for setting up the bucket. Now to create a user and give them appropriate permissions to store things in your bucket.
Setting Up An AWS User In IAM And Granting Them Permission To Access Your Buckets
Head over to the AWS IAM https://console.aws.amazon.com/iam/ console and click on the Users tab over on the left. Click on Add user and give them a sensible name and Programmatic access. And then click Next : Permissions.
Select Attached existing policies directly
and in the search box enter: s3ReadOnly and give the user AmazonS3ReadOnlyAccess
And then click on Next: Tags. I did not enter anything here. So next click on Next: Review and if everything looks appropriate click on Create user
Make note of the Access Key Id and Secret access key values. You will need those soon, and this is the only time that you can access the Secret access key so if you need that value later, you will have to create a new key for this user.
Heading into the home stretch!
Installing AWS S3 Utilities On Your Droplet To Handle The Transfer Of Your Database Backups To The S3 Bucket
Back in your root SSH terminal connection, install the S3 utilities with:
# apt-get install s3cmd
Accept the installation confirmation, and then you will have these tools installed. Now you need to enter your credentials.
# s3cmd --configure
will bring up a dialogue that looks something like:
Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.
Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key:
You will enter your Access Key, and Secret Key and for the rest of the values, just click enter to accept the default values unless there is something you want to explicitly set. At the end of the process, make sure that when it tests your credentials that it passes, and then agree to Save settings.
Things will probably have looked something like this:
Default Region [US]:
Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint [s3.amazonaws.com]:
Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]:
Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password:
Path to GPG program [/usr/bin/gpg]:
When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [Yes]:
On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name:
New settings:
Access Key: AKIAJ3O4AL4VMLGBEPAQ
Secret Key: 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
Default Region: US
S3 Endpoint: s3.amazonaws.com
DNS-style bucket+hostname:port template for accessing a bucket: %(bucket)s.s3.amazonaws.com
Encryption password:
Path to GPG program: /usr/bin/gpg
Use HTTPS protocol: True
HTTP Proxy server name:
HTTP Proxy server port: 0
Test access with supplied credentials? [Y/n]
Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine :-)
Now verifying that encryption works...
Not configured. Never mind.
Save settings? [y/N]
Save the settings and you are finished with this.
You can test it out now with the following command, assuming you have a db backup in that folder. If you don’t already have a backup file, do this first:
# echo 'just a test file' > /srv/db_bak/testFile.txt and then:
s3cmd sync /srv/db\_bak/ s3://contoso-sendy-backups/sql-dumps/
which should result in some output like this:
root@sendy-droplet:/srv# s3cmd sync /srv/db\_bak/ s3://contoso-sendy-backups/sql-dumps/
WARNING: Empty object name on S3 found, ignoring.
upload: '/srv/db\_bak/2019-01-12\_sendy\_backup.sql' -> 's3://contoso-sendy-backups/sql-dumps/2019-01-12\_sendy\_backup.sql' [1 of 1]
7945 of 7945 100% in 0s 85.26 kB/s done
Done. Uploaded 7945 bytes in 1.0 seconds, 7.76 kB/s.
root@sendy-droplet:/srv#
Now checking my S3 bucket, I see that the contents of that folder transferred successfully
Okay…let’s make this a job for Cron!
Add this to your crontab with crontab -e
#every day at 9:12 utc, synchronize .sql dumps with s3 bucket
12 9 \* \* \* s3cmd sync /srv/db\_bak/ s3://contoso-sendy-backups/sql-dumps/ > /dev/null 2>&1
This will send the db dump to S3 daily at 9:12 UTC. Again you might want to test this by setting the interval to something much lower (detailed in the first Cron section, above), don’t forget to set it back. I think a daily external cloud backup is a sane default. Your needs may differ.
Restoring Database From a Saved Backup
So what do you do if a database disaster happens? If you set up the S3 automated backups as described above, you have a backup!
Restoring your database can be achieved as follows:
- sign into your S3 console
- select the backup that you want to restore
- When you download the file, it will likely be downloaded as a .txt file. Just rename the file extension to .sql
- Send the backup to your droplet from the command line with (this is all one line):
$ rsync -ahrvz -e ssh /path/to/your/sql/backup/2019-01-14\_full\_sendy.sql root@157.230.133.78:/srv/db\_bak/backup\_to\_restore.sql
Now login to your droplet and login to mysql to drop the existing corrupt database. **THIS WILL IRREVERSIBLY DESTROY THE CURRENT DATABASE** So make sure you are only doing this if the db is already corrupt and needs to be restored from one of your backups.
# mysql
then ( this is the destructive part ):
mysql> drop database sendy;
then re-create the sendy database
mysql> create database sendy;
then exit out of my sql with:
mysql> exit;
and perform the database restoration from the remote shell with:
# mysql sendy < /srv/db\_bak/backup\_to\_restore.sql
Then refresh Sendy in your browser and you should be back where you started prior to the database corruption.
Let’s Create A Snapshot Of The Droplet
It’s nice to have an “undo” point. DigitalOcean gives you the ability to create snapshots of your virtual servers at any point in time. This would be a good time to create a snapshot, in case you ever need to restore the virtual server. If you did, you would only need to reimport your database to the latest saved version in your S3 bucket and you’d be back in business.
To do this, first we need to shut down the droplet from the shell with:
# shutdown
this should give you some output similar to:
Shutdown scheduled for Tue 2019-01-15 04:49:54 UTC, use 'shutdown -c' to cancel.
Your server is now going through the shutdown scripts, which is better than a hard power off. Now navigate to https://cloud.digitalocean.com/droplets and select your Sendy droplet. You should see that it is currently off.
Click on the Snapshots tab and give your snapshot a sane name. Then click on Take snapshot and it will take a couple minutes.
For the costs quoted at the time of this article, you’re looking at about $0.10 per month to store a snapshot.
After the snapshot is complete, head back to the Power menu and turn your droplet back on. Give it a couple minutes to fully reboot before you try to reload, or you may see errors related to connecting to the server or database.
You can take snapshots at any time like this, but of course be mindful of whether or not there are running jobs when you shut it down.
Congratulations, That’s it!
Please comment on how it went! I would love to hear how your process went, and if this guide helped you.
I hope this guide saves you a lot of the headaches that I went through while setting this up. I’m happy knowing that it may have helped you, and if you want to say thank you, please leave a comment, and / or feel free to buy me a beer! 🍻 Cheers!
Top comments (0)