Virtual machine backup without downtime (update)
March 2016
Using QCOW2 file based VMs in Linux has lots of neat features. One of my favourites is the virsh blockcopy operation (assuming you are using libvirt, or are familiar with it - please read the the blockcopy section in virsh’s man before proceeding).
With libvirt, it’s possible to make use of a powerful snapshot toolkit. For now, I only want to copy an image for backup purposes without having to shut the virtualized guest down. This is where the blockcopy command comes into play. It’s simple enough, the only requirement is to temporarily undefine the guest during the blockcopy operation.
You can test it with a few commands - but be careful. Clone your VM and it’s configuration manually (shutdown & cp to somewhere else) beforehand, as files are easily overwritten by accident. Both the name of the guest and of the image are identical in my example (guest123). The target device is sda - yours might be vda or hda, take a look the guest configuration, namely the disk section. Depending on your hardware and the size of the guest, the blockcopy process might take some time. I’m using htop / iotop to monitor activity during the operation.
virsh dumpxml --security-info guest123 > guest123.xml
virsh undefine guest123
virsh -q blockcopy guest123.qcow2 sda guest123-backup.qcow2 --wait --finish
virsh define guest123.xml
That’s it. You now have a backup image of your running guest, without any downtime. Libvirt does not sparse the copied image, meaning it’s as large as the original image at the moment the operation finishes.
I’m using a cron & a simple script to periodically pull backups of my VMs. It’s assuming the name of the guest and image file are the same, as in the example above. It can be used as follows:
$ ./libvirt-backup guest123
With several VMs, each one gets it’s own cronjob. For the moment, my crontab looks similar to this:
# m h dom mon dow command
05 00 * * 1 /vm/backup/libvirt-backup.sh guest1
05 01 * * 1 /vm/backup/libvirt-backup.sh guest2
05 02 * * 1 /vm/backup/libvirt-backup.sh guest3
I am having the blockcopy-backup.sh in the same directory as the images (/vm/backup). It works for now, but I might change that setup in the future. Don’t forget to set the executable flag.
$ chmod +x libvirt-backup.sh
And the script itself. It checks if the target file is truly *.qcow2, and if the guest is running, logs the time & size of the VM, dumps the XML, undefines the VM, blockcopies the guest to /vm/backup and adds the current date to the file name, defines the VM again, transfers the copy to a “target-host” using rsync and deletes the local copy. Important note - using the “-S”flag with rsync transfers sparsed QCOW2 files, saving space & bandwidth.
#!/bin/bash
GUEST=$1
BACKUP_LOCATION=/vm/backup
XML_DUMP="${BACKUP_LOCATION}/xml"
GUEST_LOCATION=`virsh domstats $GUEST | grep block.0.path | cut -d = -f2-`
BLOCKDEVICE=`virsh domstats $GUEST | grep block.0.name | cut -d = -f2-`
DATE=`date +%F_%H-%M`
GUEST_SIZE=`du -sh $GUEST_LOCATION | awk '{ print $1 }'`
if [ `qemu-img info $GUEST_LOCATION | grep --count "file format: qcow2"` -eq 0 ]; then
echo "Image file for $GUEST not in qcow2 format."
exit 0;
fi
if [ `virsh list | grep running | awk '{print $2}' | grep --count $GUEST` -eq 0 ]; then
echo "$GUEST not active, skipping.."
exit 0;
fi
logger "Guest backup for $GUEST starting - current image size at $GUEST_SIZE"
virsh dumpxml --security-info $GUEST > $XML_DUMP/$GUEST-$DATE.xml
virsh undefine $GUEST > /dev/null 2>&1
virsh -q blockcopy $GUEST $BLOCKDEVICE $BACKUP_LOCATION/$GUEST-$DATE.qcow2 --wait --finish
virsh define $XML_DUMP/$GUEST-$DATE.xml > /dev/null 2>&1
rsync -S $BACKUP_LOCATION/$GUEST-$DATE.qcow2 target-host:/libvirt_daily_backups/$GUEST/
rsync -S $XML_DUMP/$GUEST-$DATE.xml target-host:/libvirt_daily_backups/$GUEST/
rm $BACKUP_LOCATION/$GUEST-$DATE.qcow2
logger "Guest backup for $GUEST done"
exit 0;
The blockcopy and rsync operations are rather I/O heavy. If you are scheduling VM backups, it’s always good practise to leave enough time between cronjobs and to avoid other processes on your system which might be triggered at similar times, such as a smartd scans for example.
Also, as mentioned before - development on KVM/QEMU and libvirt is ongoing & very active. For Debian based systems, it might be worth considering to upgrade to unstable APT sources for at least these packages.
Update November 2020
I’ve since revisited the backup script a few times, the biggest change being that it automatically backs up all running VMs, without taking the names as arguments. There’s also an rsync subroutine to move backups, including the XMLs, to a remote host.
#!/bin/bash
BACKUP_LOCATION=/virt/backup
XML_DUMP="${BACKUP_LOCATION}/xml"
if [ ! -d "$XML_DUMP" ]; then
mkdir -p $XML_DUMP
fi
mapfile -t active_guests < <( virsh list | grep running | awk '{print $2}' )
for GUEST in ${active_guests[*]};
do
GUEST_LOCATION=`virsh domstats $GUEST | grep block.0.path | cut -d = -f2-`
BLOCKDEVICE=`virsh domstats $GUEST | grep block.0.name | cut -d = -f2-`
DATE=`date +%F_%H-%M`
GUEST_SIZE=`du -sh $GUEST_LOCATION | awk '{ print $1 }'`
if [ `qemu-img info $GUEST_LOCATION --force-share | grep --count "file format: qcow2"` -eq 0 ]; then
echo "Image file for $GUEST not in qcow2 format."
continue 2
fi
if [ `virsh list | grep running | awk '{print $2}' | grep --count $GUEST` -eq 0 ]; then
echo "$GUEST not active, skipping.."
continue 2
fi
logger "Guest backup for $GUEST starting - current image size at $GUEST_SIZE"
virsh dumpxml --security-info $GUEST > $XML_DUMP/$GUEST-$DATE.xml
virsh undefine $GUEST
virsh -q blockcopy $GUEST $BLOCKDEVICE $BACKUP_LOCATION/$GUEST-$DATE-temp.qcow2 --wait --finish
virsh define $XML_DUMP/$GUEST-$DATE.xml
/usr/bin/logger "$GUEST defined, sparsing backup image .."
/usr/bin/qemu-img convert -O qcow2 $BACKUP_LOCATION/$GUEST-$DATE-temp.qcow2 $BACKUP_LOCATION/$GUEST-$DATE-sparsed.qcow2
rm $BACKUP_LOCATION/$GUEST-$DATE-temp.qcow2
rsync -avSe ssh $BACKUP_LOCATION/$GUEST-$DATE-sparsed.qcow2 backuphost:$GUEST-$DATE-sparsed.qcow2
rsync -ave ssh $XML_DUMP/$GUEST-$DATE.xml backuphost:xml/$GUEST-$DATE.xml
rm $BACKUP_LOCATION/$GUEST-$DATE-sparsed.qcow2
logger "Guest backup for $GUEST done"
done