This time I would like to explain how I backup my server with the help of some build in tools and a script written by myself. As I’m running my blogs and my mail server, which is Scalix, on a Linux server I need to make sure, that no data is lost during an outage and that I will be back online in a short period of time. Therefore I have two strategies to backup my stuff.
- Using proxmox, it is very easy to do an online backup of my virtual machine, as described here:
How To: Proxmox Live Backup
The drawback with this method is, that it takes some time to do the backup, it consumes a lot of system resources (I/O) and the backup is very large in size.
- Using a file based backup from within the virtual machine, backing up only the real important stuff.
The drawback here, you cannot restore the system from this backup. But in combination with method one, it works perfectly.
I will not describe the first method, as this is already described in the mentioned post. I do this kind of full backup once a week and use method two to get the increment data form every day between two full backups. As method two will not work out of the box, you have to make sure the data is consistent, I will cover this method in this post.
Requirements for the Backup
First of all, you need to have an external backup server to make sure the data is protected against loss. I will use an external rsync server which is included in my server subscription. This is still in the same data center, but finger crossed, it is in a different fire compartment. The full backup, produced with proxmox is copied to my NAS system at home. This will make sure, that at least one of the backups will survive.
In addition to an external storage, you should also have LVM (Logical Volume Manager) installed on your system. This is the only option to make online backups of your system and keep data consistent using LVM snapshots. Instead of LVM, some hardware raid systems support a similar feature.
LVM (Logical Volume Manager)
I will not describe how to configure LVM, this is outside of the scope of this post, but I will explain shortly, what LVM is. LVM is included in the Linux kernel and provides a layer between the file system and the block device. LVM has features like encryption, easy expanding the volume and many more, but we would like to use only the snapshot function in the first place.
To use the snapshot function of LVM you need to make sure, that you still have some free unallocated space on your drive. This space will be used during the snapshot phase to write new changes to the disk. Make sure to leave enough free space on the disk, as you can expand you volume any time. There is no reason to oversize your volume in the first place. Keep in mind, if the space is not enough to hold all the data changes during the snapshot phase, you can be in big trouble.
Consistent LVM Snapshot
Creating a LVM snapshot is very fast but not instantaneous. There is a very little chance that some applications can be in an inconsistent state. MySQL such as could be in the middle of a transaction. Doing a snapshot in this moment will lead to an inconsistent MySQL backup. The same issue can happen with scalix. As those are the only applications which needs spacial handling in my environment, I will only cover them. But keep this in mind and check if you have applications on your system which needs special handling as well.
To get a consistent backup of the Scalix message store, you need to suspend Scalix before creating the backup. This could be accomplished using omsuspend like this:
omsuspend -s 120 & sleep 2 sync; sync; sync omsuspend -r
This first line will suspend Scalix and no user interaction with the Scalix system is possible. The last line will resume the Scalix system again and Scalix will run normal.
To avoid the situation, the you create the snapshot, when MySQL is in the middle of a transaction, you can lock MySQL. This will somehow pause MySQL. You need to issue this command to the MySQL server:
FLUSH TABLES WITH READ LOCK;
This will write the database to disk and lock the tables. Do this with a permanent session, as the lock is removed as soon as the session is terminated.
To unlock the tables again, after the snapshot is created, use this command:
You should now have a consistent snapshot and you can start the backup process… wait, I didn’t told you, how to create the snapshot.
Create the Snapshot
The snapshot is created very easy and needs only one command, two if you actually would like to get access to the snapshot data. It takes some more effort to get all the information needed, before creating the snapshot.
First of all, you need to know, the name of the volume for which you would like to create the snapshot of. In a very simple environment, you will have only one volume for all data. If you have split your data across multiple volumes, you can check the specific path using this command:
df /var/opt/scalix/sx/ Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/scalix-root 96816868 6806528 85092276 8% /
This will tell me, that the scalix message store is on the volume group “scalix” on the volume “root”. To get the correct path to the volume use this command:
lvscan ACTIVE '/dev/scalix/swap_1' [4.09 GiB] inherit ACTIVE '/dev/scalix/root' [93.80 GiB] inherit
As you can see, there are two volumes, but as the command from above told us to use the volume “root” we have now the complete path to the volume, which is
You also need a name for the snapshot volume. I chose “backup-snapshot”. Using this two pieces of information we can create the snapshot using this command:
lvcreate -l100%FREE -s -n backup-snapshot /dev/scalix/root
This will use 100% of the free space on the disk and create the snapshot. If you issue lvscan again it should look like this:
lvscan ACTIVE '/dev/scalix/swap_1' [4.09 GiB] inherit ACTIVE Original '/dev/scalix/root' [93.80 GiB] inherit ACTIVE Snapshot '/dev/scalix/backup-snapshot' [1.87 GiB] inherit
The last step would be to mount the snapshot to get access to the files. We can use a simple mount command here:
mount /dev/scalix/backup-snapshot /mnt/backup
Make sure, to create the mount folder before using this command. After backing up everything you need, you can simply unmount and drop the snapshot:
unmount /mnt/backup lvremove -f /dev/scalix/backup-snapshot
As we now know, how to create the snapshot and remove it, I would like to share my thoughts about the backup itself. I run this kind of backup every day, except the day, when I run the full VM backup. I use rsync and create a backup folder for every day. This works fast and reliable and I can benefit from just backing up the changes.
My backup includes all configuration files within “/etc” and all logs in “/var/log”. To include the scalix messages store I also backup “/var/opt” and to get the MySQL files I include “/var/lib/mysql”. “/var/www” is also included, as I have a photoblog running, which stores the pictures there.
From my point of view, this includes all files needed to recover from a crash, using the full VM backup plus the file backup.
The Backup Script
To do all the above steps automatically I wrote a script for my usage. Feel free to use it and adapt the script to your own needs.
You need to modify some of the variables at the top of the script, to make it work for you environment.
You can run the script as a cron job to run it automatically. I run my script every night at 04:30am.
If you have any questions, regarding this post or if you would like to provide feedback, please use the comment function below.