infoSpace - Storage

Digital Storage Update 2007

tkiehne — Thu, 15 Feb 2007 07:57:45 +0000

It has been well over a year since my last digital storage update, and though there has not been any earthshaking new technology announced within that time, there has nevertheless been some advancement in several areas that I would like to address.

Vertical / Perpendicular Drives

One of the major highlights of the last year has been the introduction of so-called vertical or perpendicular drive technology. Vertical recording aligns the data bits in a vertical, or perpendicular, format with respect to the plane of the the storage media, instead of the traditional horizontal arrangement. Vertical techniques are already in use and have significantly increased storage densities, particularly for compact notebook drives (see Wired: Hard Drives Get Vertical Boost).

New Media Manipulation Techniques

Working around the superparamagnetic limit by recording data perpendicular to the plane of the media is expected to peak at a data density of about 1TB per square inch. Seagate is looking to extend this gain by combining it with other technologies, to the extent that we could see data densities of 50TB per square inch within 10 years. A technique called HAMR (heat-assisted magnetic recording) uses lasers to heat up the disk surface while writing, which later cools to a more stable state. The heat expansion exposes fewer individual grains of disc material to the write process, thus increasing data density. This process is further refined by organizing the grains into a more regular pattern in a process balled bit patterning, where a chemically encoded molecular pattern is infused into the substrate during creation. The combination of these techniques with vertical recording yields a bit of data per grain of magnetic substrate, compared to about one bit per 50 grains that we see now (see Wired: Inside Seagate's R&D Labs).

Hybrid Drives and Solid State Storage

While some manufacturers continue to push for higher data densities, others have improved devices in different ways. Hybrid drives have been developed that combine solid-state flash memory and conventional magnetic discs to increase speed and reliability. Though this development does little to increase storage capacities, it does help with reducing power consumption and portends the elimination of moving parts -- and the corresponding risk of mechanical failure -- as flash memory increases in capacity (see: PC Magazine: Seagate Launches First Hybrid Hard Drive).

Speaking of Flash memory, Freescale has improved on the concept by introducing MRAM (magnetoresistive random-access memory). MRAM boasts faster read/write speeds and better stability than current Flash memory while still holding data after power has been removed from the chip. This technology improves on the upper limit of the lifespan of Flash memory (see: BBC: 'Magnetic memory' chip unveiled).

As if MRAM and Flash are not enough, research is continuing on “phase change” memory that promises more stable storage than Flash memory at as much as 500 times the speed. In addition to faster and more stable storage, phase change chips promise to be much more compact. Initial prototypes of phase change chips have already been introduced by Samsung, and there will likely be production models out within a couple of years (see: Wall Street Journal: Disk Drives Face Challenge If New Chip Comes to Market).

Again, these solid-state technologies do little to increase storage capacities, but improve stability and power consumption, and thus, offer more efficient and stable overall storage and retrieval systems.

Optical Storage

HD-DVD and Blu-Ray media are set to multiply their storage capacities by adding additional layers and increasing the data density per layer. At base specifications, 10 layers on an HD DVD would yield 150GB, assuming 15GB per layer. For Blu-ray, the total over 10 layers jumps to 250GB, assuming the base 25GB per layer. These extra layers are not supported by current readers, but the concept indicates a potentially longer lifespan for standards that initially seemed to be dead on arrival. (see: Daily Tech: Three HD Layers Today, Ten Tomorrow)

Meanwhile, much of the HD-DVD vs Blu-Ray debate has been thwarted by the announcement of a hybrid disc capable of storing data in both formats on one disc. Warner Brothers recently unveiled Total HD Disc, which eschews a standard format DVD layer in order to bundle the two competing HD formats into one disc playable in either type of HD player. This approach is contrasted by the introduction of dual players which have both HD-DVD and Blu-Ray capabilities (see: New York Times: New Disc May Sway DVD Wars).

Even at the higher recording capacities imbued by multiple layers, neither standard will approach the capacities of the terabyte holographic discs that I reported on last year. Given the massive marketing effort behind HD-DVD and Blu-Ray, and the emphasis on applications in entertainment as opposed to mass storage, I have doubts over commercial manufacturers dumping these in favor of holographic media any time soon. The most likely effect of the market battles over the two dominant HD formats is that newer, higher capacity formats will come at a premium for those seeking to implement high capacity data storage solutions. Ars Technica suggests that smaller capacity formats will be exploited first in order to decrease the cost to end users and hasten adoption. But even such decreased capacities are expected to be greater than even the multi-layer HD-DVD and Blu-Ray concepts discussed above (see: Ars Technica: Holographic storage a reality before the end of the year).

While the HD-DVD / Blu-Ray market squabbles continue, yet another terabyte optical technique has been developed. Research at the University of Central Florida developed a 3-D optical system that uses two different light wavelengths to write to multi-layer DVD media that promise more than a terabyte per disc. No plans yet on market potential, but with so many terabyte optical techniques, one or more are bound to arrive soon (see: University of Central Florida: UCF Researcher’s 3-D Digital Storage System Could Hold a Library on One Disc).

Tape Scrolls On

Not to be outdone, new developments in tape technology promise 15 times greater data density in new cassette form factors within five years. This translates to roughly 8 TB per cartridge (see: SpaceMart: IBM breakthrough multiplies the amount of data that can be stored on tapes and Wired: Tape Storage Increases 15 Times). With this sort of density, tape still offers the best price to capacity ratio and still out-carries all storage media short of large magnetic disc arrays. The question of long-term reliability of tape is still debatable, however.

The X-Factor

Moving on to more theoretical realms, scientists at the Max Planck Institute have made a breakthrough on a 40 year old theory that reveals tiny, closed magnetic circuits -- vortexes -- that demonstrate polar properties that could represent data bits. This phenomenon occurs on a scale of about 20 atoms in diameter, which is much smaller than the single grains of magnetic material that Seagate hopes to exploit in the near future (see above). Techniques exploiting this phenomenon are expected to be much more resilient against external disruptions such as heat and magnetic fields, but no word yet on a horizon for practical application or storage densities (see: Max-Planck-Gesellschaft: Magnetic Needles turn Somersaults).

Even further out there is a scheme proposed by a Drexel University professor that claims 12.8 Petabytes in the space of a cubic centimeter! The technique exploits the properties of nano-scale, ferromagnetic wires stabilized by water. Again, commercialization seems quite a ways away, but this should provide good fodder for speculative fiction writers everywhere, at least until the shock of a Petabyte iPod Nano wears off (see: Drexel University: For a Bigger Computer Hard-drive, Just Add Water).

The Future of the Hard Drive

tkiehne — Thu, 12 Oct 2006 21:34:02 +0000

On (roughly) the 50th anniversary of the invention of the hard drive, Tom's Hardware interviews Seagate's Senior Field Applications Engineer Henrique Atzkern (Quo Vadis, Hard Drive? The 50th Anniversary of the HDD). In it, we catch a glimpse of some of the ideas being explored for increasing hard drive density, speed, and reliability, among other things. Parsing through the acronym alphabet soup and surface technicality, one thing remains clear: hard drive manufacturers are not running out of ideas for increasing storage capacity, so we can expect to continue seeing dramatic leaps in storage capacities.

Let's look at this in terms of what we are storing. Most people around my age can remember how any increase in storage capacity seemed to be followed immediately by increases in program size -- developers used the extra space to put more functionality and features into their programs. The storage capacity gap has long since dwarfed the needs of applications and operating systems, but users have since taken the lead. First, users struggled with storing images and audio while developers introduced new compression schemes to accommodate them. Later, video reached the masses and started filling hard drives, even in greatly compressed states.

But the gap keeps expanding as hard drives increase in size. Text documents are not getting any bigger, even though the applications that create them keep bloating. Moving from binary to XML representations has not significantly increased word processing file sizes. Same goes for images and audio -- the bits needed to losslessly represent a 1200 dpi scan have not increased, and the same goes for a 48 kHz digital audio file. In fact, the bits needed to losslessly represent audio have actually decreased with file formats such as FLAC. On the other hand, video storage requirements are still expanding. DV quality is now giving way to HD and I would expect a few more developments before we reach a state where more bits does not yield better quality (for most typical applications).

The only thing left to close the gap for these types of digital media is to have a lot of them. Even then, I expect that the total unused storage, taken across all systems, will increase as dramatically as the storage devices themselves. This can only mean good things for those who want to "save it all."

Audio Encoding Project Resumes (or, a funny thing happened on the way to 300 GB)

tkiehne — Tue, 26 Sep 2006 06:41:22 +0000

It's been a while (almost 8 months, to be exact) since I have updated this forum on the status of my audio encoding project. I could cite the usual life delays and an unusually busy Summer as excuses, but there is more to it.

So, a funny thing happened on my way to 300 GB...

Not long after my last update, steady encoding progress brought me to about 240 GB of encoded music. As far as CDs go not much is left to encode -- perhaps 100 CDs out of the originally estimated 800 -- and I have mostly caught up in creating Ogg Vorbis reference copies. As I worked my way towards filing my 300 GB external drive, however, I began having strange pangs of trepidation centered on the thought: what happens if I lose this drive? Knowing full well that the roughly 240 GB of data represented a significant investment in time and effort, and also knowing full well the fallibility of technology and the loss risk inherent in only one copy of, well, anything, I became reluctant to continue encoding until some of these risks could be mitigated. I cannot say that this trepidation represents anything near as harrowing as what must be felt by an archivist handling rare, unique manuscripts â€“ I have the original objects to re-encode from, and most of them are not unique â€“ but through my meager risk I certainly feel for those who work in such risky situations.

Since halting progress, and having finished the aforementioned busy Summer, I have come into possession of a network attached storage (NAS) server, specifically, a 1 Terabyte Buffalo TeraStation. The prices have recently dropped on these units in the wake of the newer 2 TB versions and, likely, pressure from a spate of competing 1 TB boxes. For the benefit of those who didn't just click the link, the 1TB model contains four 250 GB hard drives and is capable of a variety of RAID configurations and storage capacities. I opted for the relative safety of a 750 GB RAID 5 configuration which, though not absolutely fail-safe, does protect against a single drive failure and quite effectively allays my trepidation over continuing the project.

I've since copied the entire contents of the 300 GB external drive to the TeraServer in preparation for resuming the encoding process, unencumbered by worry. More to come.

New Mass Storage Technology and Research

tkiehne — Tue, 29 Nov 2005 08:07:54 +0000

In the CHAT digital video preservation plan I presented an overview of digital archives technologies that includes metadata, digital storage, file formats, and repository systems and software. Of these, digital storage technology is the most rapidly developing and changing area, with constant change in price per giga-(tera-, peta-)byte and media formats. In the plan I hint at the fact that optical storage media (DVDs, CDs, etc) fall far short of the storage capacities of currently available hard disk drives and arrays. The gap is quickly closing, however, as improved storage media are announced with increasing frequency. In preparation for a revision of the technology review as a standalone digital video technology primer, I'd like to document some of these recent developments.

Blu-Ray & HD-DVD

Consumer electronics manufacturers continue to wrangle over control and support of these two standards. A PC World article (via Yahoo News) reveals the true nature of the conflict: content. This particular standards "war" demonstrates the quagmire that results when technology and intellectual property, in its current form, collide. This revelation, coupled with the impending release of holographic technologies that will easily dwarf the capacities of both Blu-Ray and HD-DVD, indicates that as far as serious archival and data storage needs go, Blu-Ray and HD-DVD will probably not present any viable mass data storage solutions. It is more likely that these format(s) will end up replacing DVD in the near term as consumer level media. Archivists in the future may have to deal with preserving the end product, but they will do so using their holographic storage contemporaries. If nothing else, the Blu-Ray/HD-DVD fight demonstrates how overactive intellectual property fears truly stifle innovation and development.

As if the competition weren't enough to cause serious concern over the archival viability of either format, Blu-Ray has committed to supporting various DRM technologies (pdf). These include watermarking (ROM Mark), crypto and licensing (AACS), and code updatability (BD+). The effect on pure data applications of Blu-Ray are not yet known, but any forced DRM regime that prevents open access to data would spell the end of any consideration of Blu-Ray for archival storage.

Holographic Discs

Wikipedia has a brief entry for these formats that might prove informative as these technologies develop.

Maxell / InPhase Technologies have announced mass holographic storage.
(via The Register and Slashdot)
Current prototypes suggest 300 GB of storage per DVD sized discs with 20 Mbps transfer rate planned for release by late 2006. Five year projections put this version of holographic storage at 1.6 TB per disc with 120 Mbps transfer rate. Anticipated archive life of these discs is estimated at greater than 50 years. Turner Networks is already testing the technology for an anticipated conversion from tape (via ComputerWorld)
This technology is in direct competition with Optware HVD (Japan), a technology originally mentioned in the plan (p. 26).

Subwavelength optical data storage

In a similar vein to Multiplexed Optical Data Storage (MODS), Iomega announced Articulated Optical - DVD (AO-DVD) (via Physorg.com) which promises up to 850 GB using reflective nano-structures. Little else is known beyond the initial announcement and patent filings.

Solid State Memory

Flash Memory Hard Drives / Solid State Discs (SSDs) are increasing in capacity (via New Scientist), though still behind those of the cutting edge in optical technologies. 16 GB capacities are promised soon and intended to replace hard drives in portable and small computing applications. Though these may be more stable than hard drives (vis: no moving parts) they are less reliable in the long term than optical storage (vis: magnetic degradation) and are not likely to overtake holographic technologies for archival use, even if they achieve similar capacities.

Nantero announced that it is developing carbon nanotube based storage called NRAM (Nonvolatile Random Access Memory).
Initial prototypes have reached 10 GB in 13 cm wafers that are about 10 times faster than current flash memory. With refinement, these could provide competition for flash memory, especially in very small applications. but for the same reasons are not likely to overtake mass optical storage.

Mass storage arrays and distributed file systems

A Slashdot posting regarding "home grown" multi-terabyte storage arrays yields some interesting resources for distributed file systems. One or more of these implementations could form the basis for future network-fabric storage. More research is warranted; in no particular order:
Lustre
xFS
PVFS
File Director
GFS
Xsan
AFS
IBRIX