infoSpace - Audio Encoding Project

Software Activation, DRM, and Implications for Digital Preservation

tkiehne — Fri, 03 Aug 2007 23:00:39 +0000

It's time again for another installment in my ongoing audio encoding project saga. For some time now I have been on the verge of the next phase of the project, which involves encoding the remaining analog sound objects in my collection, specifically cassette tapes and vinyl records. Procrastination, combined with a serious dose of being busy with other things, has delayed my progress on this phase of the project, but one technical aspect has also proved crucial.

In order to digitize the analog sound objects I require a software platform for encoding the analog input into digital objects that is also capable of cleaning-up the analog input of analog artifacts, such as tape hiss, pops, clicks, scratches, etc. There are many software packages that are available on the market for sound recording and processing and, fortunately, I already "own" one of them: Sonic Foundry's Sound Forge.

So, what's the technical problem, you ask? Well, I purchased version 5 of the software in 2001 as part of a special introductory promotion at a very reasonable price. Unfortunately, Sonic Foundry transferred ownership of the entire Sound Forge product line, as well as a few other key products, to Sony in 2003. Normally this wouldn't mean a thing, except for the fact that professional-level software like Sound Forge is protected by an online registration/activation scheme. In a nutshell, the software will install and run just fine for a 30 day trial period. During that period, you are expected to perform one of a set of procedures to register the product with the vendor which, when completed, will eliminate the 30 day countdown and give you full, unlimited access to the program. As you can guess, the transfer to Sony complicated the process in that the online registration routine built into the original program could no longer find the registration server — these functions had been transferred to Sony while the software remained unchanged.

Not being satisfied with only 30 days of the program at a time, and unwilling to shell out the bucks to upgrade, I embarked on a search to figure out the new registration procedures. I'll spare the details, except to say that it took some Googling, several failed customer service contact attempts, numerous user forum searches, and a call to a number that I finally managed to track down, which implored me to visit a chat application on their Web site in order to get to the information I needed to reactivate "my" software.

In the end, no big deal, right? But, my experience exposes some very important digital preservation issues. Sound Forge is not itself a particularly important piece of digital information in itself. It is a toolkit used to create the artifacts in which we are interested; in this case, sound artifacts. The same could be said about Photoshop, or any of an increasing number of professional media toolkits. Perhaps the furthest extent that a person in the future might need current or past versions of these software tools would be to regenerate projects that one might have created using them, or to analyze detailed technical aspects of the software. But, again, it is the products of these programs that will most likely interest future users, archivists, and the like.

But consider this: the registration and activation process used in software like Sound Forge is conceptually identical to the license management process in Digital Rights Management (DRM) schemes used to protect digital information, particularly music, movies, and other copyrighted works. Having reviewed my account above, one could imaging that instead of activating software that I purchased, that I might have been trying to access a DRM-encoded sound or video file that I had purchased in the past. The same issues with license servers, transfer of ownership/responsibility, changes in the license registration schemes and so on are just as pertinent in this new situation.

Everything managed to turn out alright for me in this case, but imagine if Sonic Foundry had simply disappeared instead of selling off it's product line? Or what if I had tried to install this software 10 or 15 years later, after the market had decided that the software no longer held enough value to justify supporting it? All discussion about ownership of digital information aside (a discussion of which would explain my liberal use of scare quotes), it seems apparent from this example that if left to the market (as governed by long copyright terms and far-reaching copyright legislation), we stand to lose not only the right to preserve digital information, but the technical ability to do so. Conveniently enough, I've treated on this situation before.

Stay tuned as I embark on the more complicated phases on my encoding project.

Audio Encoding Project Milestone

tkiehne — Mon, 19 Feb 2007 00:05:25 +0000

This week I achieved a major milestone in my personal audio encoding project -- after a longer period of time than I planned. Other than a few stragglers, I have managed to completely encode all of my compact discs, including creating use copies and fully describing the objects using embedded metadata. These are all stored on a 0.75 TB NAS appliance configured in a RAID 5 array. Additionally, I have ingested (using the term rather loosely) the use copies into an access system, Ampache.

When I embarked on this project, I estimated that my entire collection -- CDs, albums, and cassettes -- would amount to about 300GB in total. The lossless formats in the corpus currently comprise approximately 230GB across 7500 distinct objects. Once I finish encoding the analog formats I expect to more than exceed my 300GB initial estimate. Incidentally, the total corpus including use copies amounts to about 280GB across 14,500 objects*.

The next step, other than sweeping up the straggler CDs, is to move to encoding my cassettes. I have fewer cassettes than vinyl records and, since there are fewer noise reduction and quality issues to address, I figure that encoding them is the logical next step.

*Note: Although one might think the number of objects would simply be double the number of lossless objects, some releases were live or mixed which I encoded as one track for the use copy, rather than as individual tracks as they appeared on the CD. This approach preserves the work as a unitary effort for access while maintaining the original order and structure in the lossless format.

Metadata Quality Control

tkiehne — Wed, 17 Jan 2007 05:20:54 +0000

As I near the end of the first phase of my audio encoding project I feel the need to share some of the metadata quality control observations that I have collected.

Although ripping my CDs to digital media has been time consuming, it has not been near as laborious as checking and correcting the metadata that was automatically gathered during the process. FreeDB as an automatic metadata gathering service has been very helpful, but as I reviewed the corpus of encoded audio, I found many disturbing errors: misspellings, typos, missing articles, missing fields omission or misrepresentation of international characters, and, of course, the usual discrepancies in case handling, title formating, and normalized forms..

To find and correct these errors, I relied on one or more separate discography databases and integrated metadata management software. Discogs.com has been an invaluable resource for confirming apparent discrepancies, as well as helping me find and correct release dates, many of which were not found in the FreeDb data. Discogs does some rather stringent data normalization, often deviating from what is present on the actual releases, so that they can eliminate redundancy and excessive cross-linking between records. This issue has been the source of heated debate on the site's forums, as well it ought to be. Having submitted information to Discogs for many of my rarer CDs helped me to understand the compromises that they have made in their system so that I could understand why deviations from the original objects occurred and make an informed decision as to whether to apply changes. In the absence of information from Discogs, label, band, and fan sites have also come in handy for verifying information.

The most important tool I've used is an integrated metadata management program called Tag & Rename. This particular program merges a Windows explorer-like interface for viewing directories and files with an embedded metadata viewer that is capable of extracting and manipulating all of the major audio metadata formats (id3, ogg vorbis, aac, ape, etc.). The software provides a middle ground between the file system and the content which greatly increases the speed at which I can update embedded metadata.

In fact, there seem to be many such tools for this purpose for all sorts of digital object types. Another one I have come across is Exifer, a program that allows editing of embedded EXIF information in digital photos. I expect this program will come in handy when I begin processing my seven years worth of digital images.

Between these two programs, I have come up with a general list of essential characteristics for embedded metadata editors:

Filesystem integration: Functions such as copy, move, delete, rename, create directories, and so on. This feature ensures that you can stay within the metadata editing environment which saves time wasted in program switching. One thing that has been missing in my experience which could be useful is having multiple filesystem views so that you can jump between directories or volumes without leaving your working directory. This idea is insipred by my preferred code editor, Homesite.
Metadata listed in directory views: Selected metadata should be shown as part of the file list to allow a quick appraisal of the contents of the embedded metadata. Like a file list in the operating system, the list should be sortable and/or filterable by metadata field.
Ability to manipulate many different metadata standards: The program should be able to manipulate all applicable formats for the target object type (image, sound, text, etc.). Additionally, an ideal program would be extensible such that new metadata and file types could be added as needed.
Automated or batch editing: Manual, object by object editing is an expected feature, but the greatest time saver is the ability to modify entire directories or lists at once. Additionally, the ability to transfer between one metadata format to another applicable format (e.g.: id3v1 to id3v2) is essential. Copying tags directly from one field to another in the same file, swapping tags, and copying tags from one file to another have also been essential features. Finally, extraction of metadata from the filesystem, such as regular expression or pattern conversion of filenames into metadata, and vice-versa, has also come in handy.
Ability to create or access authority files: Tag & Rename allows me to create a list of genres for music files and exposes that list in edit dialogs, although it does not apparently have the option to force me to use only this list. In the absence of pre-coordinate lists, input masks should be available, especially for date and time fields. An added bonus for more detailed metadata formats could be accessing authoritative Web services for standard entries, such as a LCSH service for subjects, though I am not certain that such things yet exist.
Aggregate and summary views: This feature does not exist in Tag & Rename, but having brought all of my encoded music into an access system I have found the feature sorely lacking. Essentially, there should be a way to see the total number of objects marked with specific data, for example: grouped by genre. By browsing a list of all genres returned by my access system I was able to see outliers or variants that were present (e.g.: Synth Pop vs. Synthpop) and find them so that I could go back to the metadata editor and normalize as needed. It would be ideal to have this capability within the editor; although simply conforming to an authority list of genres would have prevented this particular problem, there may be situations where a strict authority list is not desirable.

.This is by no means an exhaustive list, and is perhaps too general to fit all object types, but the basic concept is clear. As a rule ,the less typing one does, the more accurate the metadata, but as I have experienced, even external databases have errors. Over the course of thousands of files or records, small error percentages accrue quickly. I can only imagine the headaches that would have arisen were my project to take place in a larger organization, with many people participating in the encoding and preservation process, let alone with a much larger corpus. It is clear that quality control of metadata, whether hand-entered or not, is crucial. These software tools

Audio Encoding Project Resumes (or, a funny thing happened on the way to 300 GB)

tkiehne — Tue, 26 Sep 2006 06:41:22 +0000

It's been a while (almost 8 months, to be exact) since I have updated this forum on the status of my audio encoding project. I could cite the usual life delays and an unusually busy Summer as excuses, but there is more to it.

So, a funny thing happened on my way to 300 GB...

Not long after my last update, steady encoding progress brought me to about 240 GB of encoded music. As far as CDs go not much is left to encode -- perhaps 100 CDs out of the originally estimated 800 -- and I have mostly caught up in creating Ogg Vorbis reference copies. As I worked my way towards filing my 300 GB external drive, however, I began having strange pangs of trepidation centered on the thought: what happens if I lose this drive? Knowing full well that the roughly 240 GB of data represented a significant investment in time and effort, and also knowing full well the fallibility of technology and the loss risk inherent in only one copy of, well, anything, I became reluctant to continue encoding until some of these risks could be mitigated. I cannot say that this trepidation represents anything near as harrowing as what must be felt by an archivist handling rare, unique manuscripts â€“ I have the original objects to re-encode from, and most of them are not unique â€“ but through my meager risk I certainly feel for those who work in such risky situations.

Since halting progress, and having finished the aforementioned busy Summer, I have come into possession of a network attached storage (NAS) server, specifically, a 1 Terabyte Buffalo TeraStation. The prices have recently dropped on these units in the wake of the newer 2 TB versions and, likely, pressure from a spate of competing 1 TB boxes. For the benefit of those who didn't just click the link, the 1TB model contains four 250 GB hard drives and is capable of a variety of RAID configurations and storage capacities. I opted for the relative safety of a 750 GB RAID 5 configuration which, though not absolutely fail-safe, does protect against a single drive failure and quite effectively allays my trepidation over continuing the project.

I've since copied the entire contents of the 300 GB external drive to the TeraServer in preparation for resuming the encoding process, unencumbered by worry. More to come.

Audio Encoding Project: On Genre Description

tkiehne — Sun, 29 Jan 2006 03:30:27 +0000

First, a status update on the project. At this point, I have lost track of exactly how many discs I have encoded. This is probably because the ripping environment has been working virtually flawlessly since I finished troubleshooting, but, a rough estimate puts me at around 200-250 discs encoded. Now, to move on to an issue that has been in the back of my mind for a while: genre description.

Specifically, I am finding myself increasingly annoyed by the lack of depth in genre description allowed in ID3-type metadata. To expound: most digital audio formats support some sort of embedded metadata, one of the most common being the ID3 tag block used by MP3s and FLAC. The ID3 specification allows for a single field to describe the genre of the object. Since the ID3 tag is embedded within each unitary object, this allows for record-level description. Unfortunately, I have been encoding a disc at a time and the software I am using only allows descriptive metadata to be defined at the disc level, which is then copied into each file upon encoding. This is fine for fields such as album name, release year, or artist, but is quite frustrating for its tendency to stereotype artists or releases as a whole. To make things worse, the centralized database of CD information, FreeDB, limits the genre field to a choice of among only 11 -- and not a single one of them represents any type of electronic music. For a collection such as mine that is dominated by electronic and abstract styles, this limitation is unacceptable. Fortunately, I can override the FreeDB defaults for purposes of encoding.

Determining the most appropriate descriptive term for the genre of an object is a problem that is not at all new to descriptive cataloging. Any object can have different semantic uses and/or meanings depending upon the attitude and understanding of the describer and the user. Furthermore, using a pre-coordinate description precludes the notion that new understandings or uses for the content (hence, new genre descriptions) could become apparent at a later time. As a result, I have been selecting the most specific possible genre term that can help identify the musical genre within a fairly broad tolerance, avoiding overly obscure or transient terms. For some works, the best term is obvious, but compendiums of music with very little in common among songs or artists or eclectic compositions confound any attempt at detailed description without record-level control.

So, a combination of technological limitations and the theoretical limitations of description have conspired to limit the genre choices I may make while encoding. I can overcome the record level constraint by going back through my encoded collection with an ID3 tagger, but I am still limited to one, single term. This may suffice on some general level, but is highly unsatisfactory to me personally, and this is whyâ€¦

Envision, if you will, a media player or other system that provides access to my corpus of encoded music. The system in question could access by artist, title or year with high recall. But, imagine that I want to use this system to generate playlists on-the-fly based on various content semantics, the content being the music itself. The single-term genre field will yield high recall for many types of music, but recall suffers for types of music that cross genres or are equally applicable to several at a time. For example, much of my music could be termed â€œambient,â€ implying slow to no beats or rhythm and a generally softer or quieter composition. Some ambient tracks, however, lend themselves well to a more traditionally industrial genres, or downtempo/chill, or experimental â€“ all of which are genres that can stand in their own right apart from ambient. If we were to visualize this graphically, imagine a Venn diagram with all of these genres overlapping with ambient (and in some cases, a bit with each other). A single term is unable to capture this depth, thus, the recall of automatically generated playlists is limited.

Additionally, the genre description does little to capture two other semantic aspects of song content: tempo and (what I will refer to as) energy. Any song, no matter what genre, can be classified according to its rhythmic speed with very little disagreement among users. This additional level of description would enhance playlist generation by preventing the sudden acceleration or deceleration between tracks that is so prevalent among streaming Internet radio stations. A smooth, consistent feel can be projected across a whole playlist of between groups of songs, or algorithms could be devised to create a change in tempo across the playlist in a myriad of creative ways. One may also see a similar role for key and time signature â€“ all three of these could be determined automatically with great accuracy during the encoding process.

Energy is a bit less definitive. What I mean by energy is a description that takes into account the emotional states that may be experienced by the listener. There is an inherent bias that is transferred by the describer, but I feel that, like the genre description, energy could be described consistently enough to be used in an advisory capacity. Genre has been used to encompass energy to some degree. For example, I have seen CDDB descriptions such as â€œdark technoâ€ or â€œambient industrialâ€ that do the work of describing both the energy and technical style. Unfortunately, the result is that the genre term is devalued as ever more granular descriptions that can become lost over time as collective definitions of genre morph and change. As with tempo, energy can be used to prevent abrupt transitions in automatically generated playlists. Unlike tempo, which uses a linear scale, energy is much less definite and will require a thesaurus to determine appropriate transitions and relationships.

Regardless of the depth of description that is possible, it is clear that a single descriptive genre term is not sufficient. A simple modification to the ID3 specification could allow multiple genre terms to be stored at the record level, thus improving recall for access systems.

Audio Encoding Project: Software Status Update

tkiehne — Sun, 08 Jan 2006 05:29:08 +0000

Last we heard, I was having problems with the Winamp/FLAC combination for ripping CDs. As it turns out, I was not only unsuccessful in figuring out what the exact issues were, let alone resolving them, but it turns out that the ripping process was not being entirely transparent about non-obvious errors. I discovered upon listening to some of the encoded files that there were occasional errors in the form of digital audio glitches, and, in some cases, truncated (prematurely ending) files. The ripping process did not inform me that there were any problems outside of the major errors that I occasionally encountered (and which prompted me to investigate in the first place). Fortunately, these hidden errors were not too frequent.

Frustrated by these persistent problems with Winamp, I began to look for a better solution. Some Googling and freeware searches later, I discovered dbPowerAMP, a free ripper/converter whose makers claimed to have been frustrated about error-prone rippers in much the same way as I. dbPowerAMP meets all of the specifications that I enumerated at the outset of the project, so I decided to give it a try.

Initial tests were encouraging at first. The program ripped to FLAC quickly and easily. A checksum is computed for each track which is checked against an online database that verifies an error-free rip. Such a distributed error-checking method is useful, but the single point-of-failure inherent in the centralized database is a bit worrying for long-term viability of the solution. I need to verify that there is a fallback in case the database is not accessible, in other words, that there is some means of error-checking or prevention present in the local system. I am also a bit disappointed that this technical metadata is not automatically appended to the file metadata -- I may either suggest this feature to the developers or figure out a way to script it myself.

The problem I ran into, however, was that the FLAC codec did not set the ID3 file metadata. After both my first and second run, the ID3 tags were empty. Later, however, I updated the FLAC codec to a more recent version which seemed to resolve the issue. FreeDB metadata is automatically appended to an ID3 tag, as desired.

The only remaining benefit that the Winamp solution holds over dbPowerAMP is that Winamp had the ability to automatically create a playlist file for the disc; the current solution does not. To compensate for this lost information, I set the filenaming macro to include the track number along with the artist and track name. This way, the directory structure will retain the original order, which will allow me to retroactively generate playlist metadata.

This is the current environment:

Software:
dbPowerAMP version 11.5, Windows 2000 SP 4
FLAC codec for dbPowerAMP version 5.3 (using FLAC 1.1.2)

Hardware:
Dell Latitude C600, Pentium III 700 MHz, 512 Mb Ram, 32x DVD/CD combo drive
Seagate 300 Gb USB/Firewire combo external drive

At this very moment, I have completed 8 rips in a row, with only one reported (and properly handled) error. Ripping speed averages 4-5 times realtime which translates to 10-14 minutes per full-length CD. I could probably see significant improvement in speed were I using a faster computer (decreases encoding time) and connecting to the external drive using firewire instead of USB 1.1 (I cannot use both my firewire and network cards at the same time). If we use 10 minutes as a baseline average for encoding, the remaining 750 CDs will take approximately 125 hours (5 24-hour days) of linear effort to encode.

Wish me luck!

Ruminations on Generating Project Metadata

tkiehne — Fri, 02 Dec 2005 04:27:17 +0000

Although I am still debugging the CD ripping problems I have been having, I have enough of a music corpus to begin thinking about second stage metadata generation. Additionally, I already have a corpus of 1700+ digital photos that I can also begin thinking about describing.

At this point, I have a small amount of metadata in the form of ID3-style tags embedded in the music files, playlist files describing relationships between music files, and possibly EXIF or other digital camera data embedded in the images. There are also latent attributes such as color depth, resolution, color profile for the images and compression profile, playing time, and filesize extent for the music which can be extracted wit the proper tools. The music has been described thus far using metadata extracted from the CDDB and may not prove to be accurate in some cases (I have a rant about genre description coming up, so stay tuned). All this metadata must be proofed before extending it into separable metadata objects and quite a bit more must be added, especially in terms of describing the contents of and subject indexing the photos.

Knowing the types of information that are currently available and having an eye towards the long term requirements of the collections, I can begin formulating a plan for metadata representation. One popular metadata standard is Dublin Core -- a simple, straightforward descriptive scheme. Unfortunately, DC is quite weak when it comes to encoding detailed technical or structural data, both of which are important for preservation. In all, DC is something of a cop-out to me -- something to use only in a situation where time is of the essence and description is of primary (sole?) importance.

METS, on the other hand, is a highly extensible container framework that can accommodate many other schemas. Having researched METS in the past, I know that the structural portions of the METS schema are particularly attractive for the projects I am working on. Many extensions and versions of METS already exist to handle a wide variety of situations, including photographs and sound recordings. Some of the possible extensions that I may use or derive from are:

For images:
UCB/Model Imaged Object Profile: These are probably overkill for born-digital images but could serve as a starting point.
If I am able to automatically extract more technical data from the image itself (EXIF, etc), the MIX extension (developed in partnership with the NISO Technical Metadata for Digital Still Images Standards Committee) could be of use here.

For music:
Library of Congress profile for Audio CDs.
MODS for description.

Unfortunately, METS is a fairly complex scheme. Having created a schema by hand, I know that the process of manual encoding is time consuming and error-prone. I know that much of the existing metadata described above can automatically be harvested and placed into the correct areas of the schema â€“ IF a tool exists for the purpose and IF there is a mapping of external data to the appropriate place(s) in the schema. Should these conditions be met, however, all that would be needed of me is to review the automatically generated data, add unique descriptions (or choose them from previously used values), and let the tool do the dirty work of creating the XML and saving the data to disk.

From what I know, however, this process is the biggest roadblock to thorough description of digital objects for all types of projects. I have come across many "roll-your-own" systems used by various institutions and academic groups that I know from my Web development experience would be hard pressed to extend beyond the specifics of the environment for which they were produced. In other words, the tools created for these projects are not portable, probably not terribly scalable or extensible, and thus, of little practical use to others. This reminds me of the Web applications development environment some 6-7 years ago, where every new e-commerce or content delivery idea generated a new set of code, standards, and procedures. May I dream of an imminent development of standard frameworks for metadata generation tools in the spirit of current Web application frameworks?

For the time being, I might have to do the deed and roll-my-own as well. So far, I have found a METS Java toolkit that shows promise for developing a custom tool. With any luck, what I create might be something I can release into the wilds for other intrepid researchers to use (and critique).

For the digital images, I might be able to extend existing image management software. For some of my images, I use a framework called Gallery for sharing on the Web. So far, I have not come upon a suitable metadata extension for this application, but I do not see why I can't create one. There are already extensions used by Gallery for image manipulation that could be used for generating some technical metadata. Additionally, description is as easy as a Web form and structure can be inferred from the application itself (the hierarchy within photo albums). Once a means of encoding metadata is established, I can envision adding an OAI service as well. The main concerns I have with this approach (not so much for my sake, but for the greater Gallery user base) is that of authenticity. In other words, with the image manipulation capabilities of the application, one may be led to believe that they are describing and making available an original object, when in fact they are working with a lesser-quality copy. Perhaps I can revisit this in more detail later â€“ it is not enough to dissuade me from the notion that such extensions to this widely used application are a good idea all around.

Audio Encoding Project: Initial Progress Report

tkiehne — Sun, 27 Nov 2005 05:15:21 +0000

I made some progress this week on my audio encoding project. In a total of three sessions I have ripped the contents of about 50 CDs to a lossless digital format with basic embedded metadata. This entry recounts the requirements, procedures, and observations on the project thus far.

The overall requirements for the project involve encoding all of my physical music media to a compact yet lossless digital format for long-term preservation. Additionally, the encoded objects will be accompanied by as much descriptive metadata as possible and provide a testbed to experiment with developing structural and technical metadata (see Gilliland-Swetland "Setting the Stage"), namely, describing relationships between individual objects and the technical data regarding their encoding and creation.

I have defined a set of software requirements to help me to meet these high-level requirements. There are a plethora of ripping programs available for free, as shareware or commercial products, or bundled with hardware. Some of these are specifically designed to support only certain codecs, formats, or hardware devices, but many are format agnostic. The software I require for this project should ideally allow format agnostic file extraction at a cost that is commiserate with the quality of its interface, and its stability and reliability.

In addition to the basic features of the encoding software listed above, it should also support appropriate audio codecs. Popular digital audio file formats, such as MP3, Ogg Vorbis, and Windows Media do well at compressing an audio file to a very small size, but do so at the expense of fidelity to the original recording (for more on audio codecs see the NDIIPP formats page). The very principle behind long-term preservation and archiving of any object demands high-fidelity with the original. For binary formats such as images and audio, this means using a lossless codec â€“ one that allows all of the original recording's features to be reconstructed upon each subsequent access, without loss of data, whether perceivable by humans or not. In addition to being lossless, the selected codec must also be open (as in Open Source) and well documented. The reasoning here is that should a closed, scantily supported, or otherwise obscure codec be used, the risk of obsolescence, and thus loss of access in the future, increases (see my description of transparency concerns in my digital video preservation plan).

Whatever software is selected, it should support metadata creation. Most current digital audio formats will support at least some amount of embedded metadata â€“ that is, descriptive data encoded with the object itself, usually artist, title, date, genre, etc. The ideal solution will support metadata retrieval using either CDDB or FreeDB. These online databases have already encoded much of the descriptive metadata required, thus saving untold hours of labor and error in manual reconstruction. Some programs will also support the creation of other types of metadata such as playlists, which, though not necessarily standards-based may help create more sophisticated structural metadata at a later time.

Finally, an additional requirement on my part is support for metadata configurable filenaming, including path references. In addition to the explicit metadata described above, I, like most people, like to use filesystem metadata to assist in search and retrieval. This is simply a way of saying that the location of an object within the filesystem indicates some basic metadata that is used for retrieval and description, such as artist name, album name, and song title. The ideal program will be able to use metadata from CDDB/FreeDB to create filenames and directory paths that are descriptive of the object's contents.

Among the software that I evaluated, Nullsoft's Winamp offers an extensible platform with a wide variety of third-party plug-ins that seems to best match the requirements stated above. Winamp supports CDDB queries as well as CD ripping, metadata configurable filenaming, and a variety of codecs. In addition to meeting these requirements, the program is not beholden to supporting specific formats or encumbered with DRM, and is inexpensive (approximately $20 US for the full version).

Winamp natively supports encoding to WAVE format, which is a widely used audio codec. Despite being openly documented, WAVE is nonetheless a proprietary format developed by IBM and Microsoft. Although it is unlikely that the format will endure catastrophic technical changes in the near future, the legal landscape could conceivably change in deference to future lossless audio formats that these vendors might wish to promote. For this reason alone, a pure open format is desired. Fortunately, a codec named FLAC (Free Lossless Audio Codec). In addition to comparable sound quality, FLAC appears to support a more flexible metadata structure. Unfortunately, Winamp does not natively support encoding or decoding to/from FLAC. To enable this, I found third-party plug-ins developed by Michael Facquet that enable Winamp to support FLAC.

It is with this combination of software that I began encoding my collection. Within the space of time it takes to encode 50 CDs, however, some problems are already evident. Some mixture of circumstances has resulted in not infrequent instability including program crashes and encoding failures. Though not frequent enough to warrant aborting the effort, it has taken some undue attention to manage something that should be nearly effortless. Most of the problems occur upon loading a new CD into the drive and during Winamp's response to the change. The worst crashes have occurred just after the CDDB query has completed, within a few seconds of the CD being inserted. This could be the result of program control failure (failing to pass from one state to the next in the correct sequence), hardware communication issues (the CD player not communicating with Winamp correctly), or potentially some other system or program instability as a result of the current configuration.

The second type of failure occurs during encoding. Apparently some sort of error condition encountered while encoding will cause Winamp to attempt to play one of the CD tracks before the encoding process is complete. The resulting conflict causes the encoding process to fail, usually to the effect of an incompletely ripped track. A more minor variation of this problem results when a dirty CD (usually foggy due to age, not so much scratched or damaged) causes the encoder to quit the track in question, also resulting in an incomplete track. Neither of these situations is caught or reported by the FLAC encoder; the former being caught by Winamp ("CD currently in use"), the latter not reported at all (I accidentally discovered the incomplete track while listening to it). The lack of error recovery is a disturbing development. Otherwise, however, the encoding and metadata gathering process has worked as expected.

Before continuing, it is obvious that I must determine the cause of the crashes. I have already varied the CD ripping speed (configurable within Winamp) to no avail. Next, I will begin to disable unused plug-ins within Winamp, especially those that were installed prior to my upgrading to the full version. I may also need to upgrade or re-install Winamp and/or the FLAC plug-in. Finally, I need to do some research to determine if this problem has occurred with other users. At the least I should notify the FLAC plug-in creator about the problems with error correction and checking in the hopes that he can improve future releases of his product.

If these efforts prove unsuccessful, I will need to search for a more stable FLAC plug-in or an entirely different software platform. The combination of features in Winamp will be hard to beat, though, which keeps me hopeful that I may find the solution to the problems that I am encountering.

A Personal Audio Encoding Project

tkiehne — Wed, 23 Nov 2005 00:33:01 +0000

I've recently embarked on a personal digital archives project involving digital audio. I've recently embarked on a personal digital archives project involving digital audio. This is one of a couple of projects I have in mind to help me explore the intricacies of managing digital objects with an eye on experimenting and understanding the challenges the individuals and small groups face when managing digital objects.

I will be encoding my CDs, records, and cassette tapes to a lossless digital format over the next few months. The choice of music as the corpus of study serves two goals: it provides a collection of objects that is not trivial in number or file size, and the emphasis of description is on the object level, thus reducing the emphasis on content search and granular description that are so prevalent in textual objects. Furthermore, my collection is not insignificant in size and contains a number of rare and unique items, so there is a preservation imperative. As for the legalities, any RIAA members out there can take a look at what I have to say about the subject and review portions of Title 17 of the US Code, particularly Section 107.

At this point in the project, I have elected to encode the works into a lossless, open source digital file format called FLAC. Fellow archivists need no explanation for the reasoning behind lossless and open source, but others may need to know that this choice of format ensures that nothing is effectively lost from the content value of the original objects (the music), and that by using an open source format, I can ensure that the documentation about how to decode the objects will be preserved along with the objects themselves.

The size of the encoded collection will likely reach or exceed 300 GB, which is not large at all compared to that which may be encountered in many organizational repositories, and indeed, many personal ones. However, the size is consistent with the amount of storage that an individual could require for his or her needs. I am not seeking to push the boundaries of digital storage just yet, although I recognize the need.

What I hope to achieve, besides having a backup of a large personal investment in music, is a full understanding of what tools are already available to assist with encoding and describing digital music objects, what tools may still be needed to facilitate self-archiving, and what potential problems could occur when one self-archives. These findings will inform my greater goal of developing a public infrastructure for digital preservation. I will post my observations as I encounter them.