for people managing data protection, VSS is probably one of the best features Microsoft have released inthe last 10 years. Since its launch this feature of the operating system have helped a lot in doing applications’ consistent backups, but most of all helped to dismiss all those dedicated agents for every application: with VSS what’s needed is only the VSS support from the application you need to protect, so a backup program only needs to connect to the OS’s VSS libraries, that will in turn manage applications’ VSS.

Nowadays there are several softwares supporting VSS, obviously all the Microsoft ones (Exchange, SQL, and even Active Directory for example), but also third party programs like for example Oracle DB (starting from 11g version).

Unfortunately, there also many softwares that still do not support VSS after all these 10 years. In my personal case, I was requested to create an application backup of Lotus Dominio, that does not support VSS. So this post has a double goal: to show you how to create a “application-aware” backup for Lotus Domino, and more in general to show how you can use VMware Tools to workaround the lack of VSS support.

First of all, as you can imagine from this post’s title, the virtual machine needs to have VMware tools installed and running. Tools allows a backup program to run two scripts, usually called pre-freeze and post-thaw, where you can configure and run activities to be executed before or after the vSphere snapshot. In this way, you can “quiesce” an application that has no VSS support, thus guaranteeing data integrity.

Once VMware tools are installed, you need to create the directory C:Program FilesVMwareVMware Toolsbackupscripts.d. If it’s already there, chances are some backup agent used in the past created it.

In this directory, you can create for instance a single script with all the needed commands in it. However, I use a different approach: I create a generic script, called master.bat, that helps me run the final script, pre-freeze and post-thaw This is the script code:

@echo off
if "%1" == "freeze" goto dofreeze
goto dothaw

:dofreeze
call c:scriptspre-freeze.bat
goto EOF

:dothaw
call c:scriptspost-thaw.bat

:EOF

In this way, script configuration is already completed, , so you can test commands by editing those two scripts until you find a good solution. This has always seemed to me a better method to manage the quiescence made with VMware Tools, but it’s not an absolute rule.

One the first section is configured, we can edit the two scripts. In many situations, sadly the only way to quiesce a program is to stop it completely and start it again after the snapshot, and Lotus Domino is no different. You will find around many examples where the two scripts only had inside commands like “net stop service” and “net start service”. In Lotus Dominio, the pre-freeze script is a little bit more complicated since we first stop the service, and then it checks for still active processes and kill them. This is the code:

pre-freeze.bat

Net Time \%computername% >> C:scripts\logs\freeze.log

rem ***************************************
rem creates and inventory of all running Domino processes
rem ***************************************

pslist | findstr /I /C:"nadminp" >>C:scriptslogspid.lst
pslist | findstr /I /C:"naldaemn" >>C:scriptslogspid.lst
pslist | findstr /I /C:"namgr" >>C:scriptslogspid.lst
pslist | findstr /I /C:"ncalconn" >>C:scriptslogspid.lst
pslist | findstr /I /C:"ncatalog" >>C:scriptslogspid.lst
pslist | findstr /I /C:"nchronos" >>C:scriptslogspid.lst
pslist | findstr /I /C:"ncollect" >>C:scriptslogspid.lst
pslist | findstr /I /C:"ncompact" >>C:scriptslogspid.lst
pslist | findstr /I /C:"nconvert" >>C:scriptslogspid.lst
pslist | findstr /I /C:"ndesign" >>C:scriptslogspid.lst
pslist | findstr /I /C:"ndircat" >>C:scriptslogspid.lst
pslist | findstr /I /C:"ndrt" >>C:scriptslogspid.lst
pslist | findstr /I /C:"ndsmgr" >>C:scriptslogspid.lst
pslist | findstr /I /C:"nevent" >>C:scriptslogspid.lst
pslist | findstr /I /C:"nfixup" >>C:scriptslogspid.lst
pslist | findstr /I /C:"nhttp" >>C:scriptslogspid.lst
pslist | findstr /I /C:"nhttpcgi" >>C:scriptslogspid.lst
pslist | findstr /I /C:"nimap" >>C:scriptslogspid.lst
pslist | findstr /I /C:"nimsgcnv" >>C:scriptslogspid.lst
pslist | findstr /I /C:"nisesctl" >>C:scriptslogspid.lst
pslist | findstr /I /C:"niseshlr" >>C:scriptslogspid.lst
pslist | findstr /I /C:"nldap" >>C:scriptslogspid.lst
pslist | findstr /I /C:"nlivecs" >>C:scriptslogspid.lst
pslist | findstr /I /C:"nlnotes" >>C:scriptslogspid.lst
pslist | findstr /I /C:"nlogin" >>C:scriptslogspid.lst
pslist | findstr /I /C:"nmtc" >>C:scriptslogspid.lst
pslist | findstr /I /C:"nnntp" >>C:scriptslogspid.lst
pslist | findstr /I /C:"nnsadmin" >>C:scriptslogspid.lst
pslist | findstr /I /C:"nnotesmm" >>C:scriptslogspid.lst
pslist | findstr /I /C:"nobject" >>C:scriptslogspid.lst
pslist | findstr /I /C:"nomsgcnv" >>C:scriptslogspid.lst
pslist | findstr /I /C:"nosesctl" >>C:scriptslogspid.lst
pslist | findstr /I /C:"noseshlr" >>C:scriptslogspid.lst
pslist | findstr /I /C:"notes" >>C:scriptslogspid.lst
pslist | findstr /I /C:"npop3c" >>C:scriptslogspid.lst
pslist | findstr /I /C:"npop3" >>C:scriptslogspid.lst
pslist | findstr /I /C:"nreport" >>C:scriptslogspid.lst
pslist | findstr /I /C:"nrouter" >>C:scriptslogspid.lst
pslist | findstr /I /C:"nreplica" >>C:scriptslogspid.lst
pslist | findstr /I /C:"nsapdmn" >>C:scriptslogspid.lst
pslist | findstr /I /C:"nsmtpmta" >>C:scriptslogspid.lst
pslist | findstr /I /C:"nsmtp" >>C:scriptslogspid.lst
pslist | findstr /I /C:"nstatlog" >>C:scriptslogspid.lst
pslist | findstr /I /C:"nstaddin" >>C:scriptslogspid.lst
pslist | findstr /I /C:"nstats" >>C:scriptslogspid.lst
pslist | findstr /I /C:"nsched" >>C:scriptslogspid.lst
pslist | findstr /I /C:"nservice" >>C:scriptslogspid.lst
pslist | findstr /I /C:"nserver" >>C:scriptslogspid.lst
pslist | findstr /I /C:"ntaskldr" >>C:scriptslogspid.lst
pslist | findstr /I /C:"ntsvinst" >>C:scriptslogspid.lst
pslist | findstr /I /C:"nupdate" >>C:scriptslogspid.lst
pslist | findstr /I /C:"nupdall" >>C:scriptslogspid.lst
pslist | findstr /I /C:"nwrdaemn" >>C:scriptslogspid.lst
pslist | findstr /I /C:"nweb" >>C:scriptslogspid.lst
pslist | findstr /I /C:"nxpcdmn" >>C:scriptslogspid.lst
pslist | findstr /I /C:"nccmta" >>C:scriptslogspid.lst
pslist | findstr /I /C:"ncctctl" >>C:scriptslogspid.lst
pslist | findstr /I /C:"nccmctl" >>C:scriptslogspid.lst
pslist | findstr /I /C:"nccttcp" >>C:scriptslogspid.lst
pslist | findstr /I /C:"nccbctl" >>C:scriptslogspid.lst
pslist | findstr /I /C:"nccmin" >>C:scriptslogspid.lst
pslist | findstr /I /C:"nccmout" >>C:scriptslogspid.lst
pslist | findstr /I /C:"nccdctl" >>C:scriptslogspid.lst
pslist | findstr /I /C:"nccdin" >>C:scriptslogspid.lst
pslist | findstr /I /C:"nccdout" >>C:scriptslogspid.lst
pslist | findstr /I /C:"ngdsscan" >>C:scriptslogspid.lst
pslist | findstr /I /C:"ngsscan" >>C:scriptslogspid.lst
pslist | findstr /I /C:"ngstmgr" >>C:scriptslogspid.lst

rem ***************************************
rem Stops Dominio daemon in a controller fashion
rem ***************************************

net stop "Lotus Domino Server (LotusDominoData)"

rem ***************************************
rem Wait a fair amount of time for processes to stop
rem ***************************************

Sleep 300

rem ***************************************
rem If some Domino processes are hanged, it kills all of them
rem ***************************************

for /f "tokens=2" %%I in (C:scriptslogspid.lst ) do pskill %%I

Net Time \%computername% >> C:scriptslogsfreeze.log

the check the result of the script, we write the execution date of the script itself into the freeze.log file. The only parameter you can change is Sleep: in my example is 5 minutes, but you could have to increase it if you have huge programs or slow systems.

The start script is obviously much more simple:

post-thaw.bat

net start "Lotus Domino Server (LotusDominodata)"

These two scripts are executed by VMware tools when the creation of the virtual machine snapshot happens. In this way the application is stopped when the virtual machine is freezed, so its data are consistent. The stop can last few seconds or several minutes, depending on the activities recorded into the script and the speed of the underlying infrastructure. So you need to plan really carefully when to run this kind of backup, because you are going to create a disservice to users.

To test the script, you can simply run a snapshot in vCenter and ask for the VMware Tools quiescence. You will see how the snapshot creation process (usually lasting few seconds) now takes much more time, right because it needs to execute the script. This is how this behaviour is seen inside a backup software like Veeam.

Tweet

In a previous article, I wrote about the acquisition of NexGen made by Fusion-IO, what were the technological and business reasons behind this acquisition, and how Fusion-IO wants to change and increase the NexGen business. In this article instead I’m going to describe you in more details how a NexGen storage works.

A quick view on the hardware

First of all, the NexGen storage is a midrange system designed to manage several workloads, mainly virtualized. The team who developed it (John Spiers as CEO and Kelly Long as CTO) is the same who created LeftHand many years ago, and this heritage is clear for example in the choice of iSCSI as the only available frontend protocol.

The chassis is 3U high, and holds two active-active controllers. Each controller has on-board non-shared Fusion-IO cards, and their content is syncronized between controllers in real time thanks to their operating system, named ioControl and internally developed by NexGen itself.

One of the distinctive aspects of NexGen in regards to other storage vendors is for sure the usage of Fusion-IO cards. Based on NexGen explanation, flash cards usage as the first tier make possible to use even bigger and slower disks without negatively affecting the overall performances, since flash memory is much faster than ssd. SSDs are usually connected to the same controller, who turns out to be the bottleneck itself, since ALL data from the SSD or HDD go through it. Fusion-IO cards are instead connected directly to the PCI Express bus, having so higher performance limits.

there are several models, with different onboard storage, Fusion-IO models in use (from 730 Gb up to 4.8 Tb) and also the number of shared SATA disks between the two storage processors (starting from 16 TB up to 192 TB with additional shelves). The models offers speed ranging from 50k to 150k IOPS (4k, 50/50 read-write). A fascinating element of this design is the double expandability: you can add both Fusion-IO or SATA disks, depending on your needs for high or low storage.

As said before, frontend connectivity is only iSCSI, made with 4 * 10G cards, or with 1G cards (used as backup).

Guaranteed performances

As said before, primary market target for NexGen is midrange. Based on NexGen itself researches, and approved by Fusion-IO too, these customers want a unique storage for all their different workloads, but I/O demanding or space demanding. This is the foundation for the design of a hybrid solution, coupling a fast tier made with Fusion-IO cards (for performances) and a second tier made with mechanical disks (for cheap disk space).

One of the problems often emerging in hybrid storage, and one of the problem Nexgen worked to solve, is the unredictability of performances. Often blocks of the same VM ar not in the right place at the right time (usually they are in the mechanical disks when I/O starts to rise) and autotiering has a hard time to fullfil applications requests. The risk, for many customers, is to be forced to use dedicated storage for a demanding workload in order to isolate it and to guarantee its performances. VDI is a clear example, since it’s usually an invasive workload, always creating problems to its neighbors.

Silos however are an additional problem, since they bring increased additional complexity and so additional problems. NexGen wants to offer to those customers a storage for all their workloads, guaranteing performances to all of them.

NexGen says the problem lies in performances ruled by caching and tiering algorythms, instead of business needs and workload policies.

Right now, even if the ioControl (NexGen operating system) allows for fine and detailed controls, end users only see 5 policies so they have much ease of management. These policies are applied to workloads in real time, dinamically.

Another way to improve performances is deduplication. Right now it’s in an initial phase, but it already allows to save on disk writes, so it accelerates disk usage and increases the endurance of the Fusion-IO cells.

Finally, it’s interesting to know they developed a highly customized operating system, not based on a classic linux distribution (I said before about LeftHand heritage, well they were using CentOS there…) but directly on linux source code.Always with performances in mind, NexGen’s OS is completely executed in RAM; this allows to save Fusion-IO from operating system’s activities, in order to further increase their endurance.

The interface

Even interface has been designed with performances in mind. This means two things: first, interface needs to be light and fast and compatible with all the web browsers; second, the interface is going to be the gateway for data analysis.

Qith al this focues on performances and their QoS, the system needs to show all the available performance metrics, and to allow the customization of graphical rendering. To do so, they choosed a radical approach: the whole interface is based on JSON and REST APIs, and all the commands are available in GUI and CLI with no difference. This means that in reality, both GUI and CLI are only two ways to access the APIs, that are the real interface.

The graphical interface impressed me for its simplicity (in a positive way): few buttons, main feature quickly available like the access to metrics, and a general feel of neatness and speed.

VMware integration

About the integration with VMware, first of all remember this storage is an active-active one, or better said ALUA. There is not LUN ownership, but there is indeed a preferred path.

As of today, integration is mainly based on their vCenter plugin. There is an interesting work to simplify its usage to vCenter admins that are not also storage admins, so that a vCenter admin for example is going to create a “datastore” more than a “volume” o “LUN”, even if the underlying storage is going to create right these elements.

Aparte fro the plugin, there nothing more right now. The storage does not supports VAAI libraries, and even if intensive activities are relieved by data replication based on redirect-on-write, support for those libraries would be desirable. Will see in the next software releases. Same applies to SRM support: when the storage will offer replication features, for sure the SRM plugin will be needed.

Summary

The development of NexGen storage has been focused on a storage offering high performances with a low price per IO. Looking at NexGen proposition, this goal seems it has been reached. In the future, we should expect further improvements in these features, but most of all a higher integration with other systems, first of all VMware, but maybe also Hyper-V or others. The time frame for these improvements is not available, but I do think the acquisition from Fusion-IO will bring higher budget to accelerate the development.

We will see in the next software releases which new features will be introduced.

Tweet

This is a story of a disadventure I had that made me cold sweat for several days, but finally ended well…

Usually, I shut down my MacBook only when I need to install or update a software that specifically requires a reboot. Standby mode never gave me any problem, and allows me to start using my computer few seconds after I pull it out of my bag, so is common for me to have more than a month of uptime. Last week, one of those frequent Flash Player update required me a reboot, so I did it, right after checking as usual the uptime (28 days and some hours at that time).

Few minutes after the reboot, I placed my MacBook in standby and I drove back home. Once arrived, I opened it up and, to my surprise, the system was completely freezed. Black screen, mouse visible and able to move around, but no way at all to log into it. There is no perfect system, not even Mac OS X, so I did a hard reset, and after a cold reboot I opened again my usual programs and I started using them.

Suddenly, after some time, it appeared, the infamous “spinning beachball of death“!!

For those mot using Mac OS X, usually it means problems. Not responding applications, disk queues filling up, the Mac equivalent of the Windows hourgrass. Since usually the reason is a crashing application, I tried to open the “Force Quit” menu to kill it, but I was not even able to go there, nor to start any other program or to cycle among those already opened, not even finder or dock.

Another hard reset, and from here my calvary began, for almost three days. Long story short, I did many hardware checks for cpu, fans, checked task manager for broken applications or daemons, removed old apps I was not using anymore, and so on. I even feared my beloved SSD was reaching the point where too much cells were ruined, but a check with DriveDX showed me it was still running well.

While I was running all these tests, crashes were more and more frequent, forcing me to reboot the system every time. One of the lines of this report however acted like an alarm bell in my head: “buggy/unknown firmware config”. A problem in my SSD drive maybe? Mi disk was 19 months old, and a quick search in Google about “Crucial M4 firmware bug” brought me to this article, saying this:

This bug would cause systems that reached a total of 5184 hours on their m4 SSD to become unstable. Once restarted systems would be stable until another hour had passed. The 0309 firmware update released today fixes this problem and allows users to update directly to the latest version regardless if the current firmware revision is 0001, 0002, or 0009.

My firmware was indeed 009, so among the buggy ones, even if my uptime counter was not as described in the article. My disk had 5788 hours of uptime, far more than 5184 stated in the bug report. But thinking about it for a while, seemed like it was my case anyway: if the uptime value was going to be updated at every boot cycle, my previous value was lower of those 28 days of uptime. So I needed to subtract about 672 hours, giving me my previous uptime value of about 5116 hours…

Mistery solved: at the first reboot, the uptime was increased with other 672 hours, moving the uptime value above the  5184 threshold. A couple of tests using the shell and the “watch uptime” command gave me the confirmation, the system freezed every time exactly after 1 hour of uptime, as explained in the bug report. After upgrading the SSD firmware (with a complete Time Machine backup first!), I was able to go beyond the 1 hour limit:

During these days lived one hour at time, I learned so much about SSD, and how they are far from beeing simple disks, and how they are instead really similar to micro computers. The firmware can be upgraded, it has bugs, and inside an SSD many activities are executed just like OS daemons: Wear Leveling, Garbage Collection and many others. By reading the release notes of those firmware versions I see clearly the improvements described, in terms of services, performances, right as in a proper operating system. Nothing comparable to old spinning disks, whose firmware is only an interface to interact with their controllers.

Tweet

Right the day before the meeting with NexGen Storage, it came the news the were just bought by Fusion-io. So, it became impossible to talk about NexGen without thinking about their technology as a part of the overall Fusion-io strategy; so I decided to wrote two different articles, this one about Fusion-IO and the future of Nexgen, and a second one deep-diving the NexGen Storage solution.

The acqusition brought even two CEO at the meeting with me and other Tech Field Day delegates. It’s not so common to be able to listen from these two main people of the two companies their thoughts and reasons behind the acquisition itself. I really liked the speech from David Flynn, Fusion-io CEO, who explained us why the company he leads has decided to buy NexGen (you can see the complete video at the end of this article).

Why a vendor known for its PCI Flash cards, that is always fighting “against” traditional storage vendors, claiming the superiority of “share-nothing” and “scale-out” designs (where each server has its own local flash storage, and replicas are managed only through software), now jumps into this market?

David Flynn gave us his reasons.

First reason, so obvsious it was not even told, is NexGen is using Fusion-IO cards inside their sotrage systems, so it was for sure among the possible targets of Fusion-IO acquisitions plans.

With NexGen, Fusion-IO can now have better access to the MidRange market, that many analysts evaluate around 16 billion dollars per year, that is twice the enterprise market. This is no doubt an attractive amount of money, so there is no wonder FIO bought a storage system in order to enter this market segment. FIO in fact knows its solution based on server-side flash cards is the most efficient in terms of performances and price at high scale, but it requires applications specifically designed to “scale-out”. that’s something that big enterprise companies like Facebook or Apple can do (in fact those are their two biggest customers), but is really unhandy for other companies using software wrote by someone else and without “scale-out” features. Even with hybrid solutions like ioTurbine or Ion available, Fusion-io has decided they need something more specific for the midrange market.

Midrange companies usually run the same storage system for a mix of workloads, so it makes more sense (by Fusion-io) to have a “multi-purpose” storage, able to take advantage of flash cards performances AND huge SATA disks at the same time, in order to have a storage that can be fast and cheap at the same time.

NexGen, unlike other primary hybrid storage vendors, uses the flash layer to accelerate not only read operations, but also writes. In this way the storage uses flash as a real tier, and not as a simple cache. This high level of performances is a key factor for migrate workloads to new storage systems, right because is now possible to consolidate many workloads in a single storage (saving money) without compromising performances.

Right because a midrand storage is not dedicated to a single workload, but it holds at the same time many workloads, to be able to guarantee the requested performances to each workload is a primary goal, in order to give to each of them a certain amount of I/O, without hindering other workloads.

NexGen has all this described features, so these are additional reasons that brought Fusion-IO to buy it.

It was really interesting the scenario Fusion-IO described about how the would like to transform the flash platforms (and also the NexGen solution) from simple components into a platform to be used by resellers to build their own solutions for final customers. According to Fusion-IO, the software stack developed by NexGen for their storage system is really sophisticated, and it allows for the creation of OEM storage systems that can be fast and reliable as the one directly created by NexGen itself.

This could lead in a medium time frame (I’m not sure is an easy thing) to software-only versions of NexGen, to be used by several system integrators in conjuction with Fusion-IO cards to realize their own storage systems. This is for sure a fascinating idea, already embraced by other vendors like Nexenta for example. Watch out for new “Fusion Powered IO” storage systems in the future, a brand that Fusion-IO wants to become a new “Intel Inside”.

Finally, Fusion-IO wants to send this message mainly to the final users, because according to David Flynn “Fusion-IO has really few friends among other IT vendors, because our technology helps customers to decrease the hardware expenses, so reducing the revenues of those vendors”.

For sure this is an interesting challenge, that could become more clear in the next few months. Fusion-IO has prepared the way by acquiring NexGen, ando also by hiring some key sales experts from 3Par, LeftHand and EqualLogic.

Will see in the next months if they will be able to succeed.

Tweet

I recently met Hitachi Data Systems (HDS) for a quick introduction to their object storage solution. Historically known for their midrange and hi-end storage solutions (for example VSP), Hitachi has instead a catalog full of products and applications.

The topic of our meeting has been as I said an overview on their object storage solution. Object storage is one of the “hot topic” right now, and all major vendors like Hitachi and others have worked to offer these kind of technologies, usually integrated with their own block or file storage devices.

Their solution is called Hitachi Content Platform (HCP, Hitachi loves three letters names…) and has all the features you can expect from an object storage, like a single contiguous filespace (it’s limit is at the moment 40 PB per namespace), support for different external filesystems that can be accessed and indexed (NFS, SMB, HTTP/REST, WebDAV…), policy based access, life-cycle management pf the objects, and others.

I’m not going to deep-dive in technical aspects, if you are interested the link I posted at the beginning of this article has many informations. Instead, I would like to understand the values of this solution compared to its competitors available in the market. The listed features are almost a “base line” of an object storage.

So, what are the distinctive advantages of the HCP solution?

Well, first of all it’s from Hitachi. I know it may sounds like a sales pitch, but just like the solutions offered by their competitors (EMC, IBM, HP to name few of them), this product has a starting target in the Hitachi existing customers, that can take advantage of a good integration with the other products of their portfolio.

But, there are for sure other interesting features.

Even if the Hitachi website only shows the hardware models of HCP, there is also a software-only version. In an IT world moving rapidly towards the “software defined” model, the possibility to use existing hardware rather than the one forced by the vendor is a common request coming from customers, especially if there is a virtua machine available and ready to be used; exactly the format offered by Hitachi for its software only solution.

Also, licensing for the virtual HCP is easy and smart: a simple license per managed TB, with a starting point at 4 TB regardless the number of nodes you deploy. It’s easy to make your own maths, and for Service Provider is easy to charge customers. The licensing for hardware versions has also and additional fee for the hardware itself, not so smart but nothing different fro similar solutions…

What’s most interesting in my opinion is the solution for branch offices (HDI, another three letters…): you can deploy a distributed solution, with a central system acting as repository for all the data stored at branch offices. A small storage is then deployed at the branch (and even here it could be an hardware one or the virtual appliance), and it replicates all its data towards the central storage following the rules you configured about frequency, retentions and so on. This solution guarantees local and fast access to data for branch offices (data centralization sometimes is more a problem than a solution when bandwidth between the offices is not so fast) and at the same time additional secirity, both in terms of confidentiality (you can cypher data) and protection (you only need to do backups in the central site).

Finally, the system does not have a proper backup tool, but you can rely on the internal versioning system, and the possibility to replicate data towards another HCP system, or even towards Amazon S3.

To summarize, Hitachi offers an object storage solution with many interesting features, maybe not so innovative like other ones (but this is usually something coming from a startup more than a big vendor) but for sure rock-solid and well-designed, and with many many available features allowing for the deploy of a complete solution.

Hitachi has a strong reputation made of solid and reliable solutions for enterprise markets, and for sure this is a well-designed solution. If you need a complete solution, HCP is for sure on of the possibility you should look at.

Tweet

Like every year, it’s time for voting for the sessions of the next VMWorld 2013. VMware just published the list of all the submitted sessions, and each of you can vote for them and help them to be approved. Voting will be open until 6th May, and even if public vote is not the only criteria for the final selection, it will surely help.

Even if this year there has been a drop in the number of session submission, the number of sessions is anyway almost a thousand, and so it’s really hard to choose which one you prefer, scrolling the list and have a look at all the presentations is a cumbersome task.

This year I decided to submit a session, and since it’s not among the first ones, I would like to help you to find it out :

And this is the complete abstract. If it sounds interesting, vote it; if it will not pass the selection, I will publish it anyway in the future via vBrownbags or other ways.

Tweet

Storage Field Day 3 event in Denver has started at full speed, and we were happy and lucky at the same time to have Exablox as our first presenting company: they launch publicly right at SFD3 coming out of stealth mode!

Exablox

If I should have to describe them in a sentence: “Object storage for the masses”.

After two years of development before launch, Exablox guys presented their object storage solution, designed from the groud-up to be scalable, reliable, with enterprise level features, but also easy to use and cheap. The solution has two building blocks, the Storage Appliance (OneBlox) and the management system (OneSystem).

The principles inspiring the design are basically four: – Easy to Manage – Easy to Grow – Enterprise Features – Low Cost

The Exablox solution is completely scale-out, and uses share-nothing principles to distribute data and metadata between the appliances forming the cluster (or ring, as they call it).

The first nice solution I saw was the chassis. While almost any new storage vendor uses Dell or SuperMicro chassis and simply re-brands them, Exablox designed their own chassis for better usability of internal space and components placement. Among those, there are the CPU and a really smart way to manage disks: to lower prices, Exablox allows customers to use off-the-shelf disks, and you do not need to use a screwdriver and sliding rails to mount them, but instead it has a simple trigger to release the disk, easy and quick. Even those small details makes a difference!

Inside the chassis, there is an uncommon processor: not an usual X86, but a Cavium Octeon II. It’s a MIPS 64 bit processor, guaranteeing high performances for cypher operations and during SHA-1 hash creations. Each data is in fact saved in an encrypted state; this offers a higher security to stored data, and also eliminates the problem of disk disposal.

About disk management, there is no RAID in the Exablox solution. The minimum configuration of an Exablox design is a single unit with 3 disks. That’s because redundancy can be obtained by using at least three store areas, that can be 3 different units, or 3 different drivers inside the same unit when a new cluster is deployed. This kind of replica has a penalty on the usable disk space; thus Exablox introduced data deduplication, inline and realtime, so they can regain disk space (obviously at variable ratios depending on the type of data).

The several nodes (up to a limit of 6 actually) are interconnected thanks to 4 1G ethernet connections, where the system transmists both data both management commands. A system can be expanded by simply adding or swapping both disks or nodes in a cluster, up to 192 TB of raw space (6 nodes with 8 * 4TB drivers each), that convert into 64 TB of usable space, plus the benefit coming from deduplication.

Also, a “ring” is made of up to 6 nodes, but it can be replicated towards a second remote ring, buying the remote replication licensing. This allows to create a quick and easy Disaster Recovery solution for an Exablox ring.  Data are moved in deduplicated state, thus offering high savings on bandwidth consumption.

From a front-end perspective, actually Exablox only exposes storage to users via network over SMB protocol. This is a design choice, good but limited (NFS or other protocols are not supported right now). Please however remember we are talking about a 1.0 release launched today.

OneSystem

The management activities of an Exablox cluster is possible thanks to OneSystem.

This solution has a unique way to work: it’s not a software you can install on-premise, but a web-based service offered by Exablox itself. The solution is well designed, and allows to create shares, connect to an Active Directory, and other activities.

One of the design goal in the creation of Exablox was from day  multi-tenancy: each part of the solution is delegable, assigned to different users and groups, even coming from different customers. This is a perfect design for an MSP willing to offer this solution to his customers, while managing all of them via a single console.

I still have a doubt, about the need to connect a system to internet in order to be managed, and about the fact some activities (like new shares provisioning or authorization changes) are not usable if the internet connection is down. Again, in the future those feature would probably be added if enough customers would ask for them.

Use cases

For sure, this solution is designed and targeted to SMB markets. Ease of use, simple interface, auto-configuration activities, scale-out, cost reductions thanks to off-the-shelf disks are all aimed to SMB customers. I still have doubt about a “internet-dependant” management solution, but remember it’s a 1.0 version.

I see some clear use cases like for example backups or archival, where you often need a small amount of space, with the possibility to grow (in small amounts is even better) as your saved data grow. Also, the possibility to replicate a ring to a second one offer an immediate Disaster Recovery solution.

We will see in the upcoming months how this new solution will be embraced by customers, and how it will grows in terms of features.

Last, if you want to learn more about the underlying technology, Tad Hunt, CTO at Exablox, offered us an impressive deep-dive about Exablox. You can see the complete presentation here:

Tweet

I’m a Mac user since many years, and I always used MacBook Pro notebooks only. I never had any problems to find good alternatives to each Windows software I was using before, even the less common ones. And just as a last resort, there is always a Virtual Machine (XP before, now 7) running inside VMware Fusion.

For the hardware part however, problems has always been a little bit worse. Even if I always checked Apple compatibility before buying any new piece of hardware (especially printers) there has been situations where I was not able to solve my problems. Console cables for connecting to network appliances like switches or routers has been my main problem.

Those console cables usually have an RJ45 jack towards the network appliance, and an RS232 jack towards the computer. But in the last 7 years, since I had my first Mac, I’ve never seen an Apple notebook with a serial cable. There are many Serial-to-USB adapters, but there are few drivers for OS X. Or at leat, until my last small discovery few days ago.

One of my colleagues, working primarily as a Network Administrator, uses a Ubuntu Linux notebook, and he owns this adapter:

Has many adapters of its kind, they have no visible brand or information to read. My colleague told me he got this one many years ago in a electric fornitures shop. It is automatically recognized and mounted by Linux kernel, and this is enough for him. I tried to connect it to my Mac OS X but it was not recognized, as you can see:

ls -l /dev/cu.* /dev/tty.*
crw-rw-rw- 1 root wheel 33, 3 Mar 17 18:07 /dev/cu.Bluetooth-Modem
crw-rw-rw- 1 root wheel 33, 1 Mar 17 18:07 /dev/cu.Bluetooth-PDA-Sync
crw-rw-rw- 1 root wheel 33, 2 Mar 17 18:07 /dev/tty.Bluetooth-Modem
crw-rw-rw- 1 root wheel 33, 0 Mar 17 18:07 /dev/tty.Bluetooth-PDA-Sync

The only serial ports available are those created by the Bluetooth antenna, even if the adapter is correctly identified by the USB internal hub. This command

system_profiler SPUSBDataType

in fact lists all the available USB devices, and also the adapter is listed:

Composite Device:
Product ID: 0x2008
Vendor ID: 0x0557 (ATEN International Co. Ltd.)
Version: 0.01
Speed: Up to 12 Mb/sec
Location ID: 0x06200000 / 3
Current Available (mA): 500
Current Required (mA): 100

Anyway, I found this great post, and I followed it from beginning to end. Once I completed the steps, I run again the shell command:

ls -l /dev/cu.* /dev/tty.*
crw-rw-rw- 1 root wheel 18, 3 Apr 10 11:37 /dev/cu.Bluetooth-Modem
crw-rw-rw- 1 root wheel 18, 1 Apr 10 11:37 /dev/cu.Bluetooth-PDA-Sync
crw-rw-rw- 1 root wheel 18, 7 Apr 10 11:55 /dev/cu.PL2303-00002006
crw-rw-rw- 1 root wheel 18, 2 Apr 10 11:37 /dev/tty.Bluetooth-Modem
crw-rw-rw- 1 root wheel 18, 0 Apr 10 11:37 /dev/tty.Bluetooth-PDA-Sync
crw-rw-rw- 1 root wheel 18, 6 Apr 10 11:55 /dev/tty.PL2303-00002006

The last line is the Serial-to-USB adapter. You can double-check that is the adapter by plugging it out and in and repeating the ls command, you will see it appearing and disappearing. Once I identified the adapter and I loaded its driver, I downloaded ZTerm to use it. You need to configure the program in this way:

Go to Settings -> Modem Preferences… and select the adapter as the default modem. Then, go to Settings -> Connections and you will find a screen like this one:

Now you will have to change the parameters as needed, depending on the appliance you need to connect to. Hitting OK, you should see the console:

In my example, it’s a pfSense installation in a Soekris Net4511.

So, at the end even a Network Administrator can use a modern and fashionable Mac, instead of those old and ugly Windows XP notebooks with a serial port

Tweet

ESXi server log management is one of those activites often ignored or poorly managed by many administrators. Usually people are more concerned about hosted virtual machines, and since lately phyisical servers are usually really reliable, as it is ESXi, they end up thinking they will never have to access their logs.

In my opinion, this is a completely wrong behaviour, because it impedes eventually the possibility to analyze events or errors happening in an ESXi servers. And they happen mor frequently than you could think.

While meeting several customers, I found at the end three kind of behaviour, that ends up in three different ways of managing ESXi logs. Let’s see them.

Sweep under the rug

Here, logs are only another bother to manage, so the quickest we remove it the better. On an ESXi server with local storage, this problem does not even surfaces, since by default ESXi saves logs in the /scratch/log directory if it finds a local storage.

More and more frequently, however, new ESXi installations are done on USB or SD memories, and those are not recognized as local storage for logs. In fact, once the installation is complete, we usually find an error like this one:

To solve it, the “sweep under the rug” method provides that the admin selects quickly one of the available shared datastores, and uses it for logs. In fact you can configure the advanced parameter Syslog.global.logDir and set the desired datastore.

If you also want to use the same datastore for all your ESXi servers, you also need to configure Syslog.global.logDirUnique, in order to have different subfolders (once per ESXi server) under the common LogDir directory.

This method has two problems. First, the limited manamanageability of logs; when you need to check them, you can use for example the Log Browser inside the Web Clientand retrieve in real time the logs from a ESXi server:

but this task is really intensive and slow, since every time vCenter needs to retireve logs directly from the ESXi server. The second problem is crystal clear: if the ESXi server is unreachable, how can you read its logs saved inside the failed server itself? In this case in fact you can only retrieve logs if they were saved into an external datastore…

Remote Syslog

A better solution is for sure to send them to a remote syslog where we can safely save them and do searches. In this ways, when you have problems in a ESXi server and its local logs are not available, you will be able anyway to check its logs from the syslog itself. This for sure useful if you are running stateless servers deployed with AutoDeploy.

VMware gives you a syslog server directly in the vCenter installation. It’s easy to install (Jason Boche wrote a nice post about it) and on ESXi server you only need to configure the advanced parameter Syslog.global.logHost with values like tcp://hostname:514 or udp://hostname:514, depending on the the configuration you choose.

If you have several ESXi server, you can even configure this parameter directly into Host Profiles, so every new server gets the right configuration.

Data Analysis?

A simple syslog server, beeing it the VMware’s one or something else, is for sure a great solution, for all the reasons I explained before. But is it enough? Let’s suppose we have a problem in a software component of a ESXi server: maybe we would like to know if in reality the error was happening at regular times since several months, each time at the same hour of the day? Or maybe the alarm appears on all your ESXi servers using that same hardware?

So, a richer data analysis solution probably would be a better fit than a simple syslog. Among the several software available on the market, I usually use Splunk. It’s a powerful solution for log collection (it “swallows” every kind of text you give it, without any problem about format, source or size), but it’s value is in the classification and tagging of every log entry.

Once the data is saved into Splunk, you can do whatever search you want, correlate data, and get results in text form or even in graphical mode, also created automatically and sent to you by mail.

If you would like to test it, there is a free version available. Splunk is licensed on a maximum daily amount of data it can collect, and the free version has a daily limit of 500 Mb. If this seems a high number to you, be aware a single ESXi server can produce almost 1 GB of daily logs, much more than the limit of the free version.

There are many tutorials in the internet about how to install and configure Splunk (it’s really simple), but probably you are more interested about this thread in the Splunk forums, where a user explains how to reduce the daily amount of logs created by a ESXi server so it can be collected with the free version of Splunk.

So?

What level of logging do you use for your logs?

Tweet

Veeam announced yesterday the 5th new feature coming in the next version 7 of Veeam Backup & Replication: tape support!

I’m already hearing many of you, both existing customers or among those evaluating it “oh, finally!”. Well, this can be easily named as THE most wanted missing feature since years. I’m pretty sure that, if it had to be decided only by the internal team at Veeam, this feature would have never seen the light: in virtualized environments backup to disk is a better solution, for restore speed reasons required by demanding businesses.

So, why this new tape support? I think there are at least three reasons:

- tape probably is no more the primary backup target, and has been replaced with disk. But is still a great media for second level copies , and useful for long term retention or backup shipping to remote locations
- Veeam has always been really careful with its customers, so I thin at some point the number of requests about tape has become enough to justify this new feature
- Veeam is a company selling its software, and the lack of tape support has always forced customer to use two different backup software, one beeing Veeam for backing up VMs to disk, and then a second one to save those backup to tape. Tape support allows now Veeam customers to eliminate that second software, so Veeam can become the only backup software in many environments.

For a complete description of this new feature, go and read the official press release. Some of the news there are really interesting, and worth a short analysis.

Files To Tape: Veeam Backup will be able to save not only virtual machines, but also files coming from any windows or linux server. You will only need to register those servers inside the Backup Infrastructure of Veeam, and you will be able to save any file of those servers. That means to support also physical servers!

Second, Veeam Backup Free Edition will support Files to Tape backups, and also restores from old NTBackup backup sets! For all those admins missing NTBackup when Microsoft dropped it in Windows 2008, this new free tool can become the new NTBackup.

Last, every restore option will be available also with tapes. And when I say all, I mean also Instant VM Recovery! I can’t wait and see how Instant VM Recovery will work with tapes, maybe they will use a temporary disk location to store data retrieved from tapes, will see!

There are still two new features to be announced, but this one is for sure one of the most awaited ones. Lastly, here is the official Veeam video about tape support.

Tweet