So imagine a scenario where you have to prep hundreds of different machines, for different customers, with different setups, but with hundreds of potential variations of OS, applications, licensing, and you’d need these all on demand.

Back in the day (up to about 2 years ago, actually), we solved this scenario in our warehouse with a hard drive duplicator setup. We had a massive 16 drive IDE drive duplicator, a smaller 8 drive SATA duplicator, and had recently purchased a 16 drive SATA duplicator that ran it’s own embedded Windows and was quite fancy. We then augmented this deployment solution with the concept of “master drives” that we’d mark as unique, store on a shelf, and pull out to duplicate.

This worked well, until the IDE duplicator died, the smaller SATA duplicator had failed ports, and to be quite honest there was damage being done to screws as the drives were taken out of/back into their machines.

I figured that a network based solution, using the base file type we and our customers use (Symantec Ghost images), would work better, if only due to ease of use if not speed.

Thankfully, I was able to equip our benches with a considerable number of HP ProCurve 2510-G 10/100/1000 managed L2 ethernet switches, some NetGear switches, and an HP tower server. A network was build to accommodate about 200 machines at once, usually 20 to a bench.

Symantec Ghost (the corporate version) supports using multicast technology to push images to many machines at once- and this was the thing that really accelerated our deployments. Benches could be setup, booted into a Windows PE (Pre-Boot environment) OS delivered over the network, then be setup to a session on the server that’d simulatenously push to all connected machines at around 600-700MB/min (or about 10MB a sec)

This setup worked fantastically, and really caught on- no more unscrewing machines, or queuing up for the duplicator, etc. However, the size of our Ghost images (even compressed!) was 20-60GB per image. This quickly blew past the small 170GB RAID that the server had.

To alleviate this, two Seagate BlackArmor NAS 400s were purchased, with a 2.67TB RAID 5 from four 1TB drives. This, combined with the internal RAID, provided a little over 5TB of space.

However, this was not the way they were utilized- one Seagate became the “backup” NAS and the other was the primary. The internal RAID was too small to even be of consequence. As such, the server ended up hosting 6+ Western Digital MyBook devices, with sizes ranging from 1TB to 2TB each, all dangling off the few USB ports a tower server provides.

I was asked to come up with a better, more usable solution to eliminate these USB drives and utilize network storage, and provide much more space. I decided on a 5-bay Synology DS1512+ NAS with five Western Digital 3TB Red drives in a RAID 5 to provide 10.8TB of storage. I’ve had excellent experiences with my own personal 5-bay Synology DS1511+ NAS with five Western Digital 3TB Green drives in a RAID 6 to provide 8.05TB of storage, running 24/7 and providing services since late 2011. I decided that a newer model (DS1512+ has USB 3 ports, providing some utility there) and NAS-ready drives (Red drives are designed to be used in a NAS, whereas Green drives are designed to be “eco-friendly”) would be a great combination.

At first glance, the solution was fantastic. We saw 120MB/sec transfers, basically at the theoretical 125MB/sec cap that gigabit ethernet would have. The Synology environment had some great features I was keen on trying, and all was looking well.

Until we did a transfer not over standard Windows SMB CIFS share, but via the Symantec Ghostcast server application. Now it’s important to understand that Ghostcasting relies on multicasting data to work – and multicasting is a fickle beast. Years ago, when testing a simple point to point multicast on our standard network, a less than 10MB/sec multicast brought the VoIP system to its knees- calls wouldn’t ring, dial tones wouldn’t work, etc. Research into this found that for optimal performance, switches would need to be “multicast-aware” and assist in the multicast process (and sectioned off from the main network via VLAN). The easiest way to know if your switches are multicast-aware is if they provide “IGMP Snooping”. Once IGMP toggles are set in your switches, you can deploy with higher speeds as the multicast blast only goes to ports that are in that exact multicast group – the same logic as how a switch sends data to the correct port whereas a hub blasts it to all ports.

Now, using a SMB CIFS share, over the WinPE environment, using Ghost, we saw an impressive 1800-2000MB/min (30-33MB/sec) performance. Now we’re used to 600-700MB/min, so I was hoping for something around that or higher off the Synology. Unfortunately, we saw performance way under that, unacceptably low numbers – sometimes under 100MB/min.

We saw this issue on multiple servers, we swapped the Synology over to a iSCSI block-level LUN (which was great to learn how to do, and they make it very simple). Unfortunately, nothing we attempted to do worked, and it seemed grim. I sent a request over to support, which was initially met with “the corporate Ghost version is not supported” and some follow-up questions.

So then I tried to salvage the situation by moving the 3TB drives over to a spare server that I resurrected a few years ago – a yellow Google Search Appliance, that actually is just a rebranded Dell PowerEdge 2950 once the Google branding is cleared. This server uses standard SATA drives, and had six bays, perfect for an OS and a 5-bay array.

Setting up the server was easy- until I learned that the PowerEdge 2950 and it’s Perc 5i RAID controller card only saw the first 2TB of the 3TB drives. This meant that, while the drives really were 2.73TB each, the controller would only address up to 2.0TB. So in a RAID 5, I had 8TB of space to use (note: this is more than a RAID 5 of 2TB drives would provide, as a 2TB drive would be more like 1.8TB each). With an 8TB array, we saw high speeds Ghostcasting off the server- showing the drives were not at fault for the speed issues. I had moved the five 250GB drives over to the Synology for testing there, and again saw abysmal speeds – worse than before, most likely due to older, slower drives that were also being scrubbed/validated by the Synology OS.

At this point, I thought I’d reached a middle ground- the server was up, fast, and working, but 2.8TB of space was lost. We would return the Synology. Life would work in a compromise.

But thankfully, Synology support asked for a debug dump from my Synology device. Once they had the binary blob, I was told the switch the Synology was plugged into (a cheap Linksys E2000 device, running Tomato firmware, serving as a wireless access point and connection to the upstream switch with internet) was throwing a wrench into things. By moving the Synology to one of the HP ProCurve switches, performance reached acceptable levels again. The Linksys device has now been downgraded to purely a wireless Access Point, scheduled to be replaced by a more robust Aruba solution in the future.

So then, drives were again swapped back into the Synology, and things look better now. The Linksys switch will be replaced with a HP ProCurve, and the Synology can stay, providing the full 10.8TB of space, serving as a NTP service for the switches, and as a Syslog central storage location. We’re hoping to use it for the storage of the WinPE environment as well using its TFTP server.

The Google box operates now as a second server, allowing for more simultaneous transfers as well as a redundancy for the main server.

It was a dicey week working through the issues, but now I believe our Ghosting environment is robust and can grow – and if we need more space, we know we can get expansion bays for the Synology. And I’m hoping to utilize the Amazon Glacier functionality for off-site backups. I almost gave up on Synology, and I’m glad their support worked with me so that didn’t have to come to pass.

Working through this, Symantec’s stewardship of the Ghost Solution Suite is lack-luster- the product is growing older and older without any updates, rumors and official comments of a new version are hard to believe with each passing quarter. A solution that uses some other software to deploy mass numbers of images is a future R&D project – the most likely candidates will be Clonezilla, FOG, and Acronis SnapDeploy.