What’s the most optimal home-oriented data storage and backup solution for our needs?

Anyone that has a home network of more than a few computers and possibly other devices such as media players (including the Sony PS3 and Microsoft Xbox) likely has a need to easily share files across their network.

In our situation we have approximately 120GB of files to share across the computers in our home. The majority of these files consist of digital audio (MP3) files and photo images. A smaller, but more important set of files represent various personal documents we use to help manage our lives.

A Network Attached Storage (NAS) device seems to be the best way of supporting our local file sharing needs. Since we also need to ensure recovery of our data in the case of a major home disaster, we’ve also included an off-site storage approach in our overall solution.

NAS servers contain one or more hard disk drives (HDDs) and special software that shares the files on the HDDs with other computers on your network. A NAS server is a hardware appliance in that it is designed to perform a single overall function and to be very easy to use – two qualities you would expect of any “appliance”.

As of January 2010, the core part of our solution, Tier 2, the NAS server, will cost you about $575 USD.  A solution that includes almost real-time local backup and easy off-site backup via the rotation of two external drives.

A NAS server will be one of several key components in our network storage and backup solution. The following requirements helped us identify other important components:

• Simplicity and automation: We don’t want to spend time setting up and maintaining a custom solution. Building a Linux-based NAS server from scratch, although feasible, is not desirable.
• Value: There is a wide variety of solutions with dramatic differences in costs and features. We need to find the right balance for our needs.
• Speed of recovery: We shouldn’t be without access to our data for more than a couple of hours to a day or two when the NAS unit or an HDD fails.
• Off-site storage: Recovery speed isn’t critical in this case, but we definitely need to backup all of our data to an off-site location at least once a month.
• Ease of expanding network storage capacity: We expect to significantly expand the storage size within 3 years. We’d rather not have to buy a completely new NAS device in 3 years. We’d rather just replace the HDD with a larger unit.

Multi-tier Solution

Our overall home storage and backup solution is comprised of three tiers of storage:

• Tier 1: Local Storage for Client-specific Needs
• Tier 2: Local Network Storage
  • Tier 2+: Local Backup of Network Storage
• Tier 3: Off-site Backup of Network Storage

Overview of the Solution and Costs

You can read further in this document to learn more about the rationale for this approach and the alternative approaches that we considered, but here’s a summary of our approach:

• Tier 1:
  • No OS Backups: We don’t plan on backing up operating system (OS) installations on our PC. If any of the PC HDD’s experience a failure, we’ll simply purchase a replacement drive and reinstall the OS of interest. We might chose to backup a few application configuration files to the NAS server. e.g. Firefox bookmark files. We certainly don’t expect to expand local HDD storage on our existing PCs.
  • No OS and Application Media Backups: We also don’t plan on making backups of OS and other application media. As long as we’re storing license keys on the NAS server, we can obtain OS and application media if a disaster or some other issue occurs.
  • Expansion of DVR Storage: We have an older 250GB external USB HDD currently acting as a shared drive that we will repurpose by attaching it to our Dish Network DVR to extend the storage of recorded shows.
• Tier 2:
  • NAS server with locally attached external HDD: External HDD acts as a local backup or mirror of the NAS server’s internal drive.
  • We chose the QNAP TS-109 Pro NAS server ($320) plus a Seagate Barracuda model 7200.10 500GB SATA HDD ($139). If you don’t need NFS support, you can save $40 by purchasing the non-Pro version of the TS-109. Since we have several Linux and Mac OS clients, we’re interested in any performance benefits that might be realized with NFS as compared to the Windows-based SMB protocol.
• Tier 2+ and Bulk Tier 3:
  • Two external HDDs rotated between the NAS server and an off-site location: We’ll exchange the external HDDs once a month or so.
  • External HDD Products and Costs:
    • eSATA HDD enclosure: Antec MX-1 for $39 (x2) at CircuitCity
    • Seagate Barracuda model 7200.10 500GB SATA HDD for $139 (x2)
• Selective Tier 3:
  • Online Storage Service for Recently Modified Files: Selectively backup frequently and recently modified files to an online storage service.
  • Manage from NAS Server: We’ll use open source tools to manage these online backups directly from the NAS server itself on at least a daily basis. (This approach requires that the NAS server be somewhat open in that we can add an application to it).
  • Minimal Cost: Online storage for 10GB per month might cost a couple of dollars using Amazon’s S3 ($0.15 per GB per month = $1.50 per month plus any data transfer fees). Alternatively, we can reuse our existing Aplus.Net web hosting account to store this data.

Total cost of the solution: $815 (not including the minimal costs of the selective backup portion of the solution)

This outlay will provide us with:

• 500GB of shared network accessible storage
• Local up-to-the-minute backup of the network storage
• Monthly off-site backup of the network storage

Although $815 might sound expensive, the primary alternative approach looks like this:

• Use a PC as File Server: One external and one internal HDD attached to an existing PC with some backup or mirroring software. This approach would cost at least $300-360 for the two HDDs, but we wouldn’t achieve the off-site storage requirement. Filling that gap would cost another $180 for another HDD that we could store off-site on a rotating basis. So we’re up to $540. The remaining ~$300 is buying us the NAS server which provides the benefits of built-in backups and sharing of our data without needing to constantly run a PC as a file server. We’re also gaining a shared media server function that would cost time and/or money to implement on a PC.
• Reduced Energy Consumption: Compared to running a PC full-time as a filer server, we will probably save $100 or more each year by running a more efficient NAS server (see below for details).

Detailed Analysis

Our analysis of the storage tiers and the details of our approach follow.

.
Tier 1: Local Storage for Client-specific Needs

PC Local Storage: We don’t plan on adding storage to any of our PCs. All of the PCs will depend on the NAS server for access to shared files.

PC Operating System and Customizations: Since we will be storing all shared documents and files on the NAS server, backing up the PC operating system-specific files is a secondary consideration. Although I don’t mind reinstalling an OS if the hard drive fails, you might want to obtain a HDD image backup product to help recover an OS installation and customizations. These solutions could either backup to CD or DVD media or perhaps be directed to backup your drive images to the NAS server. I might research these options after we’ve established the more critical tiers of our local storage and off-site backup solution.

DVR Video Recordings: We plan on adding local storage to our DishNetwork DVR to increase the capacity of recorded programs. We won’t be depending on this increased storage capacity to retain recordings for an extended period of time, but at least we won’t have to watch recording soon after they are made. We’re able to add storage to our ViP 622 DVR because Dish recently upgraded the software of this unit to utilize the USB 2.0 port on the machine. We plan to attach our existing 250GB external USB HDD to the 622 once we get our NAS server deployed. (When this HDD dies it won’t be the end of the world because we will only lose recoded TV shows).

.

Tier 2: Local Networked Storage

Since no form of local storage can address the off-site disaster recovery needs and we don’t have applications that demand continuous data availability, we probably don’t need to invest in a multi-drive NAS server and deal with the added complexity of RAID 5 (redundant array of independent disks). RAID 5 systems require a minimum of three HDDs. A four-drive, RAID 5-capable 1TB Buffalo TeraStation Live costs $650. A 500GB single drive NAS server ($300-500) + 750GB external eSATA or USB drive ($200) will roughly equal this cost. The added flexibility of the external drive and less expensive replacement cost of the NAS server coupled with the simpler configuration of the single NAS server wins out in our minds.

See Tim Higgins’ article on why “Smart SOHOs Don’t Do RAID” for a similar perspective.

Even a two-drive RAID 1 mirroring setup might be overkill for our needs. Limitations of a two-drive RAID 1 capable server include:

• We’re out of luck when the NAS unit itself experiences a failure.
• Slightly higher up front costs as compared to single drive NAS server.
• We have to supply identical HDDs up front and in the future.
• Presence of two internal drives means that it’s harder to swap drives when moving one drive to an off-site storage location (see below for a description of using two external HDDs to rotate to an off-site storage location).
• Real-time mirroring of data between the drives might result in slower performance than writing to a single NAS HDD and relying on the NAS server to backup content to the attached USB or eSATA drive when necessary.

Given these considerations, we’ve decided that our NAS needs are best met by a single drive NAS server with a locally attached, commodity USB or eSATA HDD.

Our NAS storage requirements are as follows:

• Single drive NAS server with a sufficiently large HDD.
  • Some units include a HDD while others require you to bring your own disk (BYOD).
  • Capacity: In our case, since we’re already consuming over 100GB and prices of 500GB drives are pretty reasonable (~$130), we’re planning to obtain a 500GB capacity NAS server.
• USB 2.0 or eSATA connections for a second locally attached external drive.
• Ability to backup data from the NAS drive to either the locally attached external drive or to a second compatible NAS server.
• Gigabit network interface (especially for video streaming).
• UPnP/DLNA multimedia support such that networked media players can stream audio and video from the NAS.
• Act as an iTunes server such that the NAS and its relevant media will appears in iTunes on the PC desktops.
• Solid company with a good support community. We don’t want to be left out in the cold after 3 years.

Another requirement is the ability to run custom programs on the NAS server itself. There are two applications that we’d like to run on the NAS server rather than host on another computer that would have to be constantly powered on and generating heat:

1) Off-site Backup: As part of our overall storage and backup solution, we’d like to run an open source program on the NAS server to periodically backup the recently changed content to an online backup or storage service.

2) Home Monitoring: We have a Perl-based application that constantly captures home sensor and publishes to an external web site. If we could run the Perl application on the NAS server, we could decommission the PC that is currently running 24×7 just to support this application. Some Linux-based NAS server products enable you to add your own applications. Products such as the Buffalo Technology LinkStation and LiveStation NAS servers have Perl already built in and a vibrant user community that has documented how to customize the NAS server to suit a variety of needs.
Using a PC as a Shared File Server

An alternative approach to using a NAS server is to add either an internal or a locally attached HDD to an existing PC and share that storage with other computers on the network. The main drawbacks of this approach are:

• Constantly operating a power hungry, heat generating PC
• PCs have many more points of failure than a NAS server
• Complexity of configuring a PC to play the role of a NAS server

In terms of power savings, a typical PC might consume 300+W during normal operation. Many of the single drive NAS servers consume 15-20W when operating and under 10W when in standby mode. (Adding a second drive for mirroring or backup support will add another 10-15W, but we’d have to add another drive to a PC-based file server to achieve the same local backup feature as described in the next section). As my brother-in-law has pointed out to us, the savings when using a NAS server appliance as compared to an always-on PC could be significant. For example, saving 200W by using a NAS server (probably a conservative estimate) results in a savings of 1747kWh per year which at $0.10 per kWh would equal a savings of $174 per year.

This calculation assumes that it is not easy to put a file serving PC into a reduced power standby mode. If it is feasible, then the kWh difference would need to be reassessed based on the PC’s power consumption in the reduced power standby mode. However, we haven’t stumbled onto any evidence that tells us a suspended PC can be awakened by client computers requesting access to shared drives.

.
Tier 2+: Local Backup of Network Storage

When our NAS server or the HDD experiences a failure, we need to have ready access to our shared data. Although we’ll have offsite backup of our data (see the Tier 3 section), we don’t want to go through the hassle of going to the offsite location to retrieve our most recent complete backup when we could have easily maintained a locally accessible copy of our data. Consequently, we’ve decided to retain a local backup of our shared data.

A local copy of the data housed on the single drive NAS server can best be accomplished by simply attaching an eSATA or USB 2.0 drive (eSATA provides much faster interface) to the NAS unit and configuring the NAS server to backup content from the NAS HDD to the attached drive.

Some NAS devices support the ability to replicate data to another like device on your local network. We are not opting for this solution because using a locally attached drive is more cost effective than investing in a second NAS device.

When the drive in the NAS server experiences a failure, we’ll need to:

1. obtain and install a replacement NAS drive
2. restore the NAS drive content from the locally attached USB or eSATA drive

If the NAS server itself experiences a sever failure, we should also be able to temporarily attach the external USB or eSATA HDD directly to one of our PCs to access our shared data before we obtain a replacement NAS server. We could also grab copies of files from our online backup service.

.
Tier 3: Offsite Backup of Network Storage

As an insurance policy in case of a home disaster in which our NAS server and the drives are destroyed, we need to have offsite backup of our shared data. Several offsite backup solutions were considered:

• Sneakernet 1: Make backups to DVDs or to magnetic tape and store in our safe deposit box
• Sneakernet 2: Make backup to external HDD and store at a friend or relative’s house
• Make online backups to friend’s network
• Make online backups to our web hosting provider
• Make online backups to storage utility services
  
• Use one of many online backup services

Sneakernet 1: DVDs or magnetic tape: We view the use of tapes or DVDs to backup data and the process of manually moving the media to a secure, off-site storage location as far too cumbersome for our needs. We’d initially need a lot of DVDs to represent our baseline backup. After the initial backup, we’d need to handle DVDs to manage the incremental backups. Trying to make sense of lots of DVDs and attempting to recover the files of interest is not attractive to us. It seems like to much manual bookkeeping. For similar reasons, using a tape drive and tape media does not our ease of use requirement.

Sneakernet 2: External HDD: Given the price of HDDs, we could pretty easily backup all of our content once a month to an external HDD and carry the HDD to a friend’s or relative’s home. We’d probably purchase two external HDDs such that we could rotate them. Ideally, at any time, one of the two external HDDs would be stored at the off-site location while the other external drive would be attached to the NAS server and acting as the local backup device. As long as it would be straightforward to unplug this device, move it off-site and reattach the second device from the off-site location and have the backup rebuilt, we could limit the number of external HDDs for our overall solution to two.

Online backups to a friend’s network: Some people suggest making backups across the Internet to another NAS server or HDD on a friend’s home network. Although technically feasible, we don’t want to burden our friends and families with the extra load on their Internet connection and the extra complexity of another device on their network.

Based on the pricing of online backup services, the hassle factor of backing up to a friend’s network might be worth it after all.

Web Hosting Provider: Our web hosting provider and other inexpensive providers offer varying storage sizes with their hosting plans. You can use any number of basic FTP or rsync tools or perhaps even client side backup tools to periodically backup content to the ISP. Bear in mind though that ISPs aren’t providing backup-specific services. Lots of nice backup-oriented features won’t be present when using an ISP. For example, strong encryption of your data may not be available when using your existing ISP account. You would have to encrypt your data and then transfer it.

Aplus.Net offers 170GB of storage for $9.95 per month as part of one of their hosting plans. Although that’s quite a bit less expensive than most online storage utility and backup services, it’s not clear to me that we’d be missing important backup-oriented features. Since we’re already paying for a more basic Aplus web hosting plan to host our personal web content, this might be a very practical route for us to explore.

Online Storage Utilities: Raw online storage utilities such as Amazon’s S3 are attractive, but again, as in the ISP case, they don’t focus on providing backup-oriented features. You’d have to use custom scripts or obtain backup-oriented client software to work with these services.

Online Backup Services: Initially, I was hoping that backing up content from our NAS server to one of the online backup services would be the most cost effective and easiest means to address our off-site storage needs. However, based on our research of online backup services, it’s pretty evident that the cost of backing up 120GB of data won’t be inexpensive. The consumer offerings of the online services don’t usually support mapped drives. This means that you can’t run the backup clients for the consumer-oriented services on a PC that has mapped the NAS drive as a locally accessible drive. For example, MozyPro’s pricing jumps up to$0.50 / GB per month. That’s $60 per month to store our current data. Ugh.

Current Assessment of Off-site Backup Solution

Since our initial hope of using an online backup service to address all of our off-site storage needs has not panned out due to the high costs, we’re looking at the following hybrid solution:

• Sneakernet: two external HDDs to make bulk copies of all of our data that we’d periodically move off-site to a friend’s or relative’s home
• Online backup or storage service to which we’d transmit only the most important, frequently changing files

We still have to itemize the costs of this portion of our solution, but it appears at this stage this hybrid approach is the best value when considering dollar costs and hassle factor.

A couple of external HDDs of sufficient capacity (500GB to start) will cost us about $280. We’d rotate these external drives to reduce the hassle of picking up and dropping them off at the off-site location. Ideally, at any time, one of these two external HDDs would be attached to the NAS server and automatically updated. When we’re ready to move our data off-site, we’d simply disconnect the external HDD and swap it with the other one from the off-site location. Hopefully, we can make this process pretty painless.

Given that our most important, frequently changing files that we have consume perhaps 10GB or less, we can use Jungle Disk + Amazon S3 or one of the other online storage and backup services to schedule periodic backups of this subset of our data. With an Amazon S3-based service, it would cost us about $1.50 per month to store 10GB.

NAS Candidates

There is a wide array of consumer or SOHO (small office/home office) oriented NAS devices on the market. A few of the single drive models that stand out are:

QNAP TS-109 Pro

• See our updated costs as of January, 2010.  A much more powerful QNAP TS-119 unit would replace the TS-109 Pro we purchased several years ago for the original solution.  The TS-119 can be purchased for $280 USD.
• $320 with NFS support and BYOD – bring your own drive. (If you don’t need NFS support, you can buy the TS-109 for $280).
• Huge amount of features, can do RAID-1 to locally attached USB or eSATA drive, etc.
• We’ve spoken to a user that has successfully uses the dual eSATA external HDD swapping technique for off-site backups.

Buffalo LinkStation Live

• $390 with 750GB internal drive
• Wasn’t clear to us that we could easily support the dual eSATA external HDD swapping technique.
• Vibrant hacker community – a plus for us because we want to run a Perl-based application on the device.

Extra Drive Candidates

External HDDs are even more of a commodity than SOHO NAS devices. We basically have two options:

1) Assemble our own eSATA or USB drive
2) Buy prebuilt eSATA or USB drive

In either case, we’re looking at ~$180 to match the capacity of the drive in the NAS server.

Based on an admittedly quick review of eSATA 3.5″ HDD enclosures, we chose the Antec MX-1 product. CircuitCity sells this product for $39.

Again, see the updated costs as of January, 2010. A good quality, fast 1 TB drive can be had for $85 and rather than buying hard drive enclosures, one can purchase a single eSATA-capable drive dock for $40.

Offsite Backup Candidates

There are many online backup services available. Mozy and Carbonite appear to be two of the most popular and backed services. Amazon’s S3 storage utility acts as the back end of some of the online backup services.

As described earlier, based on the pricing structure of these services and the inability of the backup services to backup directly from the NAS, we’ve decided to add custom application our NAS server to periodically backup recently changed files to an online storage service such as Amazon’s S3 for several dollars a month.

Mozy

• a little more than $50 per year of unlimited storage for their consumer, locally attached storage-capable service.
• $0.60 per GB per month for the Mozy Pro service that supports mapped drives (a required feature for backing up the content of a NAS device)
• perceived to have richer feature set than Carbonite.
• EMC just purchased Mozy.

Carbonite

• $50 per year of unlimited storage for their consumer, locally attached storage-capable service.
• Lots of users and popularity. Still independent, but appears to be vibrant.
• One drawback seems to be that many users perceive the Carbonite client to be more invasive than Mozy.

JungleDisk (front ends Amazon S3)

• One-time fee for the JungleDisk client (~$20 or so) plus you sign up for S3 and pay Amazon for the raw storage usage.
• Amazon S3’s costs $0.15 per GB per month and then access fees. Storing our 120GB of data on S3 would cost at least $18 per month or $216 per year. With our revised approach of storing only frequently changing content, we’re probably looking at using 10GB or so of storage. That will cost us a much more reasonable $1.50 per month or so.

Others?

• iDrive supports mapped drives, but their personal plans limits storage to 150GB. Their Pro plan costs $200 per year for 100GB and $500 for 500GB.

No tags