A Quick Guide to Move Database Backups to s3 Cloud via Cron

This blog walks you through a step by step guide to move your database backups to your Amazon AWS s3 cloud. Provided your website is hosted on an AWS instance, you can follow the undermentioned steps to keep your backups safe just-in-case your instance crashes, gets hacked or cleaned by mistake.

Don’t forget to use Netsuite amazon integration connector for a smooth transition of NetSuite and amazon store.

This guide is the continuation of my last blog, where you learned steps on scheduling MySql database backup using cron. Keeping backups on the same server is not always the best choice, it can only work if your database alone crashes and not the whole instance. Additionally, these backups occupy space on your live server, adding hard disk to a live server for keeping backups alone leads to high costs.

Amazon’s AWS cloud services, which span 54 availability zones to date within 18 geographic regions and 1 local region around the world, provide you this amazing feature called “s3 bucket” that helps in keeping your backup’s load off the live server. To avail this feature, you need to set up a s3 bucket first. Let’s have a look at how you can do it:

Step 1: Set up a s3 Bucket

You can find the updated method of creating a s3 bucket on AWS’ own website, click here to see their recent guide. The other steps linked to this guide also gives you information on how to add, download and delete files in the s3 bucket other than the hosting server.

Let’s assume your s3 bucket’s name to be “my-db-backups”

Step 2: Configure s3 Bucket in your Unix Instance (ubuntu in our case)

Now to enable your server to upload files to this bucket, execute the following commands on your AWS instance server:

        sudo s3cmd –configure

You will be asked to provide details like, access_key and secret_key. Fill in the required information, you can always modify it by using the aforementioned command or by editing the following file:

/home/ubuntu/.s3cfg

To verify whether or not you have configured s3 cloud correctly on your server, execute the following command and check the file in the s3 bucket online:

sudo s3cmd -c /home/ubuntu/.s3cfg put <any_test_file> s3://<your_bucket_name>

Step 3: Add Command to Move your Backup to s3 Cloud via Cron Job

Now, modify your database backup script and enable it to upload these backups to the bucket instead of storing them on the server:

# (1) set up all the mysqldump variables
DATE=`date +"%d_%b_%Y_%H%M"`
SQLFILE=/var/backup/db/db_backup_${DATE}.sql
DATABASE=<database_name>
USER=<db_user>
PASSWORD=<db_user_password>

# (2) in case you run this more than once a day,
# remove the previous version of the file
unalias rm     2> /dev/null
rm ${SQLFILE}     2> /dev/null
rm ${SQLFILE}.gz  2> /dev/null

# (3) do the mysql database backup (dump)
sudo mysqldump -u ${USER} -p${PASSWORD} ${DATABASE}|gzip > ${SQLFILE}.gz
sudo s3cmd -c /home/ubuntu/.s3cfg put ${SQLFILE}.gz s3://my-db-backups
sudo find /var/backup/db/. -mtime +7 -exec rm {} \;

The last two lines are the additional commands in the script, the first one is same as the test command you executed at the end of step 2, whereas the other command in the script removes backup files that have been present for more than 7 days, assuming that they were moved to the bucket at the time of creation and are not required to be kept on the server. You can modify the +7 parameter as per your requirement.

Additional Information: s3 cloud can be used to store all types of content and not just database backups. You can even configure them on servers other than AWS but that requires installation and configuration of s3cmd and its pre-requisites, which shall be covered in my next blog.

That’s it! I am certain that this blog will help you in moving database backup to s3 cloud via cron efficiently.

Please feel free to comment or reach out if you have any questions. In case you need any help with server management, cron setup, installation, upgradation and customization of your Magento2 or Magento 1.9 web store, please get in touch with us.

Blog Post