I got recently some jobs suddenly failing in one of our Veeam repository servers. the error message was:
Error: Client error: Cannot allocate memory for an array. Array size: [4198400].
After some checks, a strange situation arised: the sum of all the processes visible in the task manager was far below the available RAM (6 Gb in this case), but nonetheless the summary tab of Task Manager was effectively showing a complete usage of the available RAM.
The problem is, you cannot see all the memory consumption using Task Manager. You need a better tool, and when it comes to Windows DeepDive, there is nothing better than Sysinternal tools! In my situation, the tool of the trade is RamMap:
In the first tab, I was able to see how the memory was almost completely consumed, and what kind of Usage was consuming it. As I already saw in Task Manager, there were many Veeam related processes, but none using a huge amount of ram:
In the first tab however there was already a hint about what was happening: 1.5 Gb were consumed by Mapped Files. Opening the File Summary tab showed me were the problem was:
A single backup job, configured to run in reverse incremental mode, was eating 1.4 Gb of ram alone. The problem itself was not the backup job, but Windows 2008 R2 allocating too much memory to the Disk Cache service in order to manage this file.
Even if Microsoft stated that Dynamic Cache service was no longer needed in 2008 R2 (you can read their explanation here), the symptoms were the same as those related to this “no more needed” service.
To solve the problem, another Sysinternals tool came to the rescue: CacheSet. This tool allows to check the current cache value, clear the cache itself and set lower limits so Windows 2008 cannot eat all the memory again:
As you can see, the peak size of the cache has been as high as 4 Gb. I set a new maximum of 1.5 Gb, and I also hit the Clear button to flush the cache itself. As soon as CacheSet cleaned the cache, memory consumption lowered to 4.5 Gb, leaving the server 1.5 Gb of free RAM:
If you are going to size a Veeam repository, take also this behaviour into account, and size the server memory accordingly. Especially if you are using Reverse Incremental, the repository server needs to open both the VBK file and the VRB files in order to inject changed block into the first and write the differences into the second. This activity is I/O intensive, but also RAM consuming. The bigger the backup set, the bigger the cache you will need.
Plan accordingly.