I can’t help but enjoy reading some of the comments, debates and other various blog articles on optical storage technologies, especially on technology durability and media longevity, even more when compared to tape storage for long-term digital archiving or as a slightly active data repository. Back in November 2011, I wrote a blog on the capacity density of the new BDXL Blu-Ray format being denser than LTO5 tapes. But now I want to explore longevity. How long should your data last on a given technology? And I don’t just mean the data on the media – I mean is the media supported today without your datacenter doubling as a technology museum? I’m sure we all have floppy disks, tapes or removable disks that still have data on them, but you’ll never know if that data is any good because, well, there’s nothing left to read them with.
I would like to hear your opinions as to the information in the chart below (my timing might be a little off, but you should be able to get the gist of this post). Fortunately or unfortunately, I’ve been in the technology industry for a long time. I have participated in and implemented optical technology both professionally, and as a “prosumer” (prosumer in this case is a very knowledgeable consumer, or professional consumer). Actually, most of you have as well whether you know it or not.
Let me explain what I am trying to describe in this chart at a high level. The first commercially available Compact Discs (CD) hit the market in the early 1980s. There were eventually two formats released which allowed higher capacity going from 650MB per disc to 700MB per disc. The first commercial use was for music distribution and competed with tapes (cassettes and 8-tracks) and vinyl records. Tapes and records were analog recording technologies while CDs were digitally recorded. The important takeaway here is, digitally recorded music means music was stored as digital data and converted to analog sound. Soon thereafter, CDs were used as a vehicle for other content distribution like audio books, video, documents and multimedia documents, games and software. Today, in 2012 (about 30 years later), CDs are still in use for content distribution, consumer storage, software games and data distribution, etc. During this time, the prices dropped dramatically for both the blank media and the drive devices for reading and writing, yet the quality and features continued to increase. I like to use my example of a used CD I bought at a swap meet for a $1.00 that didn’t work 12 years ago on the drives back then (I knew it wouldn’t work due to the scratches, but “Dark Side of the Moon” for a buck??), but works today on a modern drive. I’m not saying this always works, but in this case, it did.
Around the early to mid 1990s (roughly 12 years later), the follow up to CDs was announced, Digital Versatile Discs, DVD. This time, multiple layers for recording data was incorporated with 4.7GB and 8.5GB capacities. Primary market focus was video distribution competing with VHS videotape and possibly LaserDisc (though no real threat on that front). The follow on markets for DVD included, game distribution, software distribution, music, consumer and enterprise recording for content distribution, backup and archiving, and other multimedia documents. Today, DVD is still the main media distribution vehicle for software, games, consumers, etc. We are currently in the “overlap” era between Blu-Ray (BD) technology and DVD. The overlap era between movies on VHS tape and movies on DVD lasted about 2 to 3 years, and my guess is the overlap era between DVD and BD will be much longer. The cost of DVD blank media and drive devices for reading and writing, have also dramatically dropped like CD technology. One important note here, every DVD drive device today, reader, writers, rewriters, supports both DVD and CD media.
Today, Blu-ray Disc (BD) is the new media introduced around 2006/2007 and became the supported standard around 2007. Blu-ray is still in the overlap era with DVD for movie distribution and as the other content distribution vehicle, but also in the mix are online streaming services, application downloads and “cloud”. In this regard, I’m only going to discuss the BD technology. BD capacities when announced were 25GB and 50GB (dual layer) and famously competed with Hi-Definition DVD (HD-DVD). This consumer-based, public battle was not the replacement of a main staple media like CD’s taking over audio tapes and vinyl records for music, or DVD replacing VHS video, but more like the VHS versus Betamax battle for the video tape standard back in the 1980s. The new BDXL Blu-ray format now supports 100GB and 128GB capacities using both additional recording layers (3 and 4 layers) and a new recording format to increase the per layer density. Projections are that this many layer approach for the Blu-ray technology will continue for some time to increase the capacity density of the media and drive down the bit cost per disc.
Ok, now that the history and background is laid out the way I want it for this article, let’s talk about longevity. Today, if you Google “buy a Blu-ray drive”, you will see internal drive devices for around $60 with a modern SATA interfaces. If you read closely, these drive devices are capable of reading and writing media for CD, DVD, and BD (BDXL support is still new and a little more expensive for now). For a few dollars more, you can even have the drive device write custom labels for you directly on the disc. Think about that, a brand new, mass-produced, commodity modern device manufactured to support a 30-year-old media technology for around $60. The device itself is greatly different from the original CD device from 30 years ago with newer technology, enhanced features, new interface(s), faster, smaller and denser packaging, etc. The oldest CD in my collection that I burned is from March 1995, and this disc still works today in my new MacBookPro (of course, this CD isn’t stored in the bottom of a drawer somewhere), and my first music CD from the 1980s still plays today in these new devices.
In my chart, I tried to illustrate the industry standard interfaces of the time of the introduction of the technology such as SCSI and IDE, and so forth (avoiding the exotic stuff). This shows the evolution of the technology world advancing forward with faster and more economical interfaces, improved manufacturing efficiencies, and so forth, but here’s my take: with the many markets that use these technologies, especially the consumer and high volume markets, the support for legacy media isn’t an option. My guess is, it could be harder to rewrite the firmware to drop support for these media formats than it would be to just keep including it going forward. Bottom-line is 3 generations of media supported with read/write capability today spanning about 30 years with improvements and added features, all with reduced pricing.
Now let discuss tape, specifically Linear Tape-Open (LTO), and more specifically tape for long-term data preservation. From the Wikipedia entry for LTO, a modern LTO tape drive, for example LTO5,
- Can read the current generation tape cartridge (n), LTO5, and the two prior generation tape cartridges (n-2), that is, LTO4 and LTO3 written by those tape’s associated tape drives in their native capacities and format
- Can write a current generation tape cartridge (n), LTO5 drive writes to an LTO5 tape cartridge, and the prior generation tape cartridge (n-1), LTO5 drive writes to an LTO4 tape cartridge at its native capacity and format
- This would apply to all tape generations. For example, an LTO4 tape drive can read LTO4, LTO3 and LTO2 tape cartridges, and can write an LTO4 and LTO3 tape cartridge
Also, according to the LTO generation table below from the same Wikipedia page, LTO dates back to 2000 with new generations of LTO standards introduced on average about every 30 months (2 ½ years).
The next generation LTO release, LTO6, is rumored to be out by the end of this year, 2012. This means that read support for LTO3, released in 2005, and write support for LTO4, released in 2007, will be dropped at that time, or at least for the LTO6 tape drives. This is the 6th generation of this technology in 12 years, so this standard moves very fast, maybe too fast. For long-term data preservation requirements, the idea of technology and media being obsolete so quickly drives Operational Costs (OPEX) higher over the lifetime of the data due to technology and media migration costs, which in many cases is, forever.
Granted, the standard optical storage track (notice at the bottom of my chart the carcasses of defeated technologies throughout this optical storage history) is currently only in its 3rd generation, so there’s no telling what will be supported in the next generation. There are some holographic technologies that promise backwards compatibility, but maybe not for all generations and with a loss of significant advances like capacity, and there are some holographic technologies that might be disruptive to the compatibility track. Then there’s the notion that the market could split for consumer and distribution based holographic technology and support, and a holographic archiving and enterprise class. The trick will be to maintain the price curves using common manufacturing and parts.
Currently, Blu-ray has its pros and cons. Technically, the BDXL format is denser than the current LTO5 tape cartridge when measured in gigabytes per cubic inch, data stored is randomly accessed which is great for non-streaming use cases, has a highly rated durability factor (to be discussed at another blog), and has media longevity options that will last 30 years with standard quality media, and 50 to 100 years for special certified quality media. The technology evolution is slow, but this is not necessarily a negative situation when long-term data preservation requirements need a stable, enduring technology. Stable technology here does not mean static. The technology advances and improves, which applies to the older media as well, but the requirement to support the media going forward has several market segments driving the support requirements for stability while also driving down cost. Stated another way, you cannot upset billions of users and consumers in a highly competitive market, and stay in business long.
Tape is faster for streaming uses cases like backup restores, but is not suited for randomly accessing its stored data. In fact, randomly seeking within a tape shortens its lifespan, as does every tape load operation. It will be interesting to see what the longer-term effect will be with the support for LTFS (Linear Tape File System) for archiving and data repository systems that have a slightly active requirement to them. However, with the short and accelerated compatibility matrix, this may not be an issue, as the underlying technology will require higher tape migration frequencies in order to remain supported and compatible.
What are your thoughts and experiences on this subject? Do you agree with my chart? What’s the oldest CD or DVD you’ve burned that still works? Do you have any predictions going forward?
