In the CHAT digital video preservation plan [3] I presented an overview of digital archives technologies that includes metadata, digital storage, file formats, and repository systems and software. Of these, digital storage technology is the most rapidly developing and changing area, with constant change in price per giga-(tera-, peta-)byte and media formats. In the plan I hint at the fact that optical storage media (DVDs, CDs, etc) fall far short of the storage capacities of currently available hard disk drives and arrays. The gap is quickly closing, however, as improved storage media are announced with increasing frequency. In preparation for a revision of the technology review as a standalone digital video technology primer, I'd like to document some of these recent developments.
Blu-Ray & HD-DVD
Consumer electronics manufacturers continue to wrangle [4] over control and support of these two standards. A PC World article (via Yahoo News) reveals the true nature of the conflict: content. This particular standards "war" demonstrates the quagmire that results when technology and intellectual property, in its current form, collide. This revelation, coupled with the impending release of holographic technologies that will easily dwarf the capacities of both Blu-Ray and HD-DVD, indicates that as far as serious archival and data storage needs go, Blu-Ray and HD-DVD will probably not present any viable mass data storage solutions. It is more likely that these format(s) will end up replacing DVD in the near term as consumer level media. Archivists in the future may have to deal with preserving the end product, but they will do so using their holographic storage contemporaries. If nothing else, the Blu-Ray/HD-DVD fight demonstrates how overactive intellectual property fears truly stifle innovation and development.
As if the competition weren't enough to cause serious concern over the archival viability of either format, Blu-Ray has committed to supporting various DRM technologies [5] (pdf). These include watermarking (ROM Mark), crypto and licensing (AACS), and code updatability (BD+). The effect on pure data applications of Blu-Ray are not yet known, but any forced DRM regime that prevents open access to data would spell the end of any consideration of Blu-Ray for archival storage.
Holographic Discs
Wikipedia [6] has a brief entry for these formats that might prove informative as these technologies develop.
Maxell / InPhase Technologies have announced mass holographic storage.
(via The Register [7] and Slashdot [8])
Current prototypes suggest 300 GB of storage per DVD sized discs with 20 Mbps transfer rate planned for release by late 2006. Five year projections put this version of holographic storage at 1.6 TB per disc with 120 Mbps transfer rate. Anticipated archive life of these discs is estimated at greater than 50 years. Turner Networks is already testing the technology [9] for an anticipated conversion from tape (via ComputerWorld)
This technology is in direct competition with Optware HVD (Japan), a technology originally mentioned in the plan (p. 26).
Subwavelength optical data storage
In a similar vein to Multiplexed Optical Data Storage (MODS), Iomega announced Articulated Optical - DVD [10] (AO-DVD) (via Physorg.com) which promises up to 850 GB using reflective nano-structures. Little else is known beyond the initial announcement and patent filings.
Solid State Memory
Flash Memory Hard Drives / Solid State Discs (SSDs) are increasing in capacity [11] (via New Scientist), though still behind those of the cutting edge in optical technologies. 16 GB capacities are promised soon and intended to replace hard drives in portable and small computing applications. Though these may be more stable than hard drives (vis: no moving parts) they are less reliable in the long term than optical storage (vis: magnetic degradation) and are not likely to overtake holographic technologies for archival use, even if they achieve similar capacities.
Nantero [12] announced that it is developing carbon nanotube based storage called NRAM (Nonvolatile Random Access Memory).
Initial prototypes have reached 10 GB in 13 cm wafers that are about 10 times faster than current flash memory. With refinement, these could provide competition for flash memory, especially in very small applications. but for the same reasons are not likely to overtake mass optical storage.
Mass storage arrays and distributed file systems
A Slashdot posting regarding "home grown" multi-terabyte storage arrays [13] yields some interesting resources for distributed file systems. One or more of these implementations could form the basis for future network-fabric storage. More research is warranted; in no particular order:
Lustre [14]
xFS [15]
PVFS [16]
File Director [17]
GFS [18]
Xsan [19]
AFS [20]
IBRIX [21]
