Veeam Backup & Replication V8 is out! (As part of the new Availability Suite V8, so yes also Veeam One V8 is already available)
After months of previewing the new upcoming features, several beta versions, posts, webinars, demos, everyone can now finally download it and upgrade their installations, or try it out before deciding to move to this new versions. I’m not going to repeat the list of major and minor feature that this latest versions has, we talked already extensively about them, the official Veeam blog has many articles about the major ones and the best thing one can do now is to try it by himself. The release notes have an insane list of technical enhancements, and there will be for sure time from now on to focus more on some of them.
What I’d like to talk about today, is a last minute fix that was included directly into the GA (General Availability) version of the software.
If you follow the VMware community, you probably heard about one of the latest discovered bugs, described in VMware KB 2090639. CBT is a great solution available for identifying changed blocks in virtual disks. All data protection solutions, and so also Veeam, rely on these informations to determine which areas of a virtual disk have changed since the last backup, and this information is used to decide which blocks to include in an incremental backup. Because of this bug, when you expand a virtual disk which has Change Block Tracking (CBT) enabled past any 128G boundary, the change tracking data becomes unreliable. Due to the faulty changed block information, some changed blocks might not be captured by an incremental backup, so that a restoring from an incomplete backup could cause a data loss.
Pretty scary indeed!
The only possible solution, since there is no patch for this, is to reset CBT informations whenever a virtual disk that was originally smaller than the boundary is expanded. The KB states that every multiple of 128 is affected (so also 256, 384, 512 and so on…), even if in some internal tests we didn’t saw the issue for example going above 256. Anyway, the biggest problem is not to reset the CBT, but the fact that some virtual disks has been expanded at some point in the past, and noone maybe remembers when it was done. Say you expanded the disk above the boundary more than one year ago, unless you have executed a restore (or you are using automatic verification like Veeam SureBackup), probably your entire chain of backups following the disk increase are useless since its content is corrupted. Even if you fix the problem by resetting the CBT, your retention is gone!
Again, scary!
As said, a manual solution that everyone can apply is to identify any disk that is bigger than 128 Gb, and reset CBT informations. And from now on, to track any size change especially when the size is increased above a boundary, and then to reset CBT informations again. Veeam forums has a dedicated thread on this topic that has grown insanely fast in few days, and in some posts users have posted some nice scripts to automate the process. If you want to apply them, remember that the following backup after the CBT reset will have to use disk scan to identify changed blocks, so backup times will be severely impacted by this, especially in large environments with a large number VMs.
We at Veeam decided that the bug was too critical to be addressed in a later patch, and we worked to add a solution directly into the GA version. I have to praise our developers that were able to finish the patch and add it to the GA release without having to delay its release!
So, when you are going to install (or upgrade to) Veeam Backup & Replication V8, there will be directly a solution to protect you from the CBT bug. It’s not a fix, that one will have to come from VMware, but meanwhile you’ll be protected and your backups will be not corrupted.
What does it do? Veeam will automatically reset CBT informations on a processed VM upon detecting a virtual disk configuration change, regardless the change has not crossed any known boundary. We think is a safe approach since the details of the issue are not clear yet. A disk size change is not a common activity, so there should be no big issue on your environment, but with this you will not have to worry about the bug anymore, and you will not have to setup complex tracking solutions to identify when a disk size has been changed. Veeam will do it for you.
Enjoy Veeam Backup & Replication V8 and the entire Availability Suite!