infoSpace - Metadata

Guarding the Guards: Archiving the Electronic Records of Hypertext Author Michael Joyce

tkiehne — Fri, 28 Jul 2006 18:27:45 +0000

In June of 2006, Thomas Kiehne and Catherine Stollar were selected to present the results of work performed the previous year at the Harry Ransom Center in Austin, Texas to a colloquium assembled by the Society of American Archivists and the National Archives and Records Administration. The following is the case study that was presented.

Update: This text was re-published by SAA in the proceedings from the "New Skills for a Digital Era" colloquium.

Abstract

In 2005, the Harry Ransom Center at the University at Austin acquired the fonds of hypertext author Michael Joyce. The major emphasis of the Ransom Center's collections is the study of literature and culture in the late 20th and early 21st century of the United States, Great Britain, and France. Michael Joyce's groundbreaking work in hypertext poetry and fiction make his papers a desirable addition to the Ransom Center holdings.

The Michael Joyce Papers are mostly composed of electronic records with an additional 60 manuscript boxes of paper -based materials This is the first mostly electronic archive the Ransom Center has acquired and new strategies for preserving digital content were employed. This case study discusses the techniques and skills utilized to preserve the electronic records of Michael Joyce as a model for processing future digital manuscripts at the Ransom Center.

Scenario

Established in 1957 by University of Texas Vice President and Provost Harry Huntt Ransom, the Harry Ransom Humanities Research Center at The University of Texas at Austin incorporated a strategy for collecting older rare books and manuscript collections with a new initiative to collect literary, photographic, and theatrical works by modern artists. Some of the authors whose works are included in the Ransom Center's collections are Norman Bel Geddes, Don DeLillo, T.S. Eliot, James Joyce, Ernest Hemingway, Norman Mailer, D.H. Lawrence, Ezra Pound, Anne Sexton, Isaac Bashevis Singer, and Tennessee Williams. Michael Joyce's work as perhaps the most influential hypertext poet and author fits nicely into the Ransom Center's contemporary author collecting policy.

Our case study to preserve Michael Joyce's digital manuscripts resulted from collaboration between the School of Information at the University of Texas at Austin and the Harry Ransom Center. Three students, Thomas Kiehne, Vivian Spoliansky, and Catherine Stollar, from Dr. Patricia Galloway's Problems in Permanent Retention of Electronic Records course offered at the School of Information undertook a semester long project to develop a strategy for archiving an initial accession of electronic materials saved on 371 3.5" floppy disks (totaling 211 KB) from author Michael Joyce. Upon completion of the project, a second accession of electronic and paper-based materials, including the contents of three hard drives (totaling 8.38 GB) and 60 manuscript boxes, was acquired by the Harry Ransom Center and is currently being processed by staff archivist Catherine Stollar according to the strategy developed during the class project. Our case study discusses strategies for file recovery, migration, preservation, arrangement, and description developed working with both accessions of Joyce's materials. The electronic records are currently maintained in a DSpace repository administered by the School of Information, however, in the future the Joyce records will move to a DSpace repository controlled by the Ransom Center and the General Libraries of the University of Texas.

The lack of Ransom Center staff with skills in digital archivy provided the impetus for the Ransom Center to partner with the School of Information on the Michael Joyce Papers. Although the Ransom Center employs talented archivists and IT professionals, no staff member possessed skills necessary for archiving digital manuscripts. The Ransom Center sought advice from Professor Galloway and agreed to use the Joyce materials as a case study in the Problems in Permanent Retention of Electronic Records course.

Some audio and video migration preservation projects were already in progress at the Ransom center in the Department of Photography and Visual Collections to preserve audio and video works, but there were no concerted efforts to preserve born digital manuscripts. Policies and Procedures for migration of audio and video content to new media were unsuited for born digital manuscript preservation and policies for preserving digital manuscripts were inadequate to capture the complete behavior of the original digital record. Previously, the few electronic manuscripts and correspondence already in the Ransom Center's manuscript collections were printed and organized in boxes like paper records. Because digital records are entirely unlike paper-based records, a preservation strategy based in printing records preserved very little of the original document. Electronic media were saved, but researchers were prevented from viewing original disks and no access copies were created.

The main component of our preservation strategy is to ingest electronic records and associated metadata into an institutional repository. DSpace, created from a joint project between MIT and Hewlett-Packard, is the institutional repository we used and will continue to use for electronic record preservation. At the heart of DSpace, like most open archival information systems (OAIS), is a database populated by individual digital objects supported by content, context, and structure metadata. We used DSpace, instead of FEDORA or another institutional repository, because it was already the established as the repository of choice for the School of Information. Although we had issues with the web user-interface for ingesting, viewing, and accessing materials within the repository, we plan to work with a talented ISchool student with Java programming skills to make our installation of DSpace more user-friendly.

Partnerships were key components to our case study's productivity and success. The initial group of students participating in the first part of the case study each represented a different background. Thomas Kiehne brought a wealth of Information Technology skills to the project, including programming and operating systems knowledge. Catherine Stollar shared her knowledge of archival theory and practice during the case study. Vivian Spoliansky viewed the case study through the lens of preservation and shed light on aspects of authenticity and desired levels of service for object preservation. Working with a variety of subject specialists on the project enabled participants to learn key skills from the others that will be useful on future digital record preservation projects.

Processing as Digital Archeology

A number of digital materials, including emails and published articles, within the archive had a paper-based counterpart, demonstrating that he created both digital and analog records while performing the same activities. Both formats of records were created synchronously, and at an institution like the Ransom Center that preserves not only works that have influenced the arts and humanities fields, but also preserves the context in which those works were created, we determined it would be desirable to reflect synchronous creation in the arrangement. We did not originally understand relationships between Joyce's digital and paper materials because our first portion of the case study only dealt with electronic records from the first accession of 371 floppy disks. One of the more unique aspects of this project involved the processing of 371 3.5â€ floppy discs that contained the digital objects of the first accession. The provided floppy discs were mostly from the Macintosh â€œclassicâ€ era, some of which date as far back as the mid to late 1980s. The assumption at this stage is that the original storage media is not stable or reliable and the information that they hold must be moved quickly and efficiently. Otherwise, little was known about what to expect in terms of specific technological issues or challenges.

At the outset of the project, we had only a general idea of the process of moving the digital files from the source media to a repository, and as such, we could not express specific requirements for software tools and utilities that might be needed. In order to minimize project overhead in terms of time and resources, we desired to use only open source, shareware, or freeware tools that are readily available in order to assist with the extraction process. This approach allowed us to assess the suitability of tools that are currently available and their ability to interoperate. In the absence of suitable free tools, we intended to find commercial software or create our own programs or scripts to perform the required tasks as we identified them. In the course of processing the first accession of discs, we quickly elucidated a more detailed procedural framework that can be abstracted and applied to future projects.

The general process implemented during the processing of the discs is as follows:

Receive and identify physical media
Catalog the physical media
Copy files to newer physical media
Perform initial file processing
Create an item-level index of all recovered files
Create and process working copies of all files while retaining the original bitstream copies

Technical metadata is collected at each step in the process not only to facilitate the work in progress, but to support provenance and authenticity. Each operation performed on the bitstream â€“ every copy and access â€“ provides the opportunity for inadvertent loss or alteration, so careful recordkeeping is as essential as careful handling. Additionally, all personnel involved in processing must thoroughly understand the procedures involved in order to prevent duplication of effort or discontinuities in results. In many cases, software can automate these processes, thus reducing the chance of errors, but the extent to which software can mitigate such risks is limited by the assumptions made by the creators of the software and how well the personnel making use of this software understand these limitations.

Given that time was of the essence, we opted to use text entries in Microsoft Excel spreadsheets to create the initial disc catalog and the associated metadata. This approach allowed us to leverage existing proficiency with spreadsheets and the availability of the software to eliminate the time needed to create a custom database application or to learn project management software. Unfortunately, the conspicuous absence of relational or workflow aspects in the spreadsheet format made us vulnerable to recordkeeping errors, making quality control a primary concern.

The copy functionality of the computer operating systems involved were sufficient to perform the movement of digital files from floppies to hard rives and removable media. Unfortunately, the differences between Macintosh and Windows in the management of file system metadata became significant. Creation dates are handled differently between these two operating systems such that a copy made in Windows takes on the date of the copy operation, not the creation date of the original from which it was made. Additionally, file system metadata for Macintosh files are stored as separate, invisible resource forks that are notorious for becoming corrupted. As a result, we often could not trust the dates ascribed by the operating system and had to refer to external resources, such as Michael Joyce's curriculum vitae, to confirm or provide date metadata at a later time. Issues with Macintosh resource forks also affected file downloads from DSpace after ingest.

At many points during the processing, we encountered technical difficulties in the form of file or disc errors. These errors can occur for a number of reasons, including damaged media, exposure to magnetic or other hazards, dirty data surface areas, and so on. In the case of dirty surface areas, several attempts were needed to overcome a copy error. It is suggested to have a drive cleaning kit available and use it periodically to prevent build up of debris on the drive head. For other errors, it was necessary to have available software utilities that can attempt to recover from file copying errors. Windows provides such capabilities within the operating system (e.g.: Scandisk), but Macintosh does not. For our purposes we were able to discover an older version of a commercial program, Norton Utilities, which allowed us to recover many files that could not be copied initially. Virus checking was also a preeminent concern. Errors and crashes must be met with persistence as they are often surmountable, which implies at least a minimum degree of technical knowledge.

In moving the digital files to other media, we created a filesystem hierarchy that mimicked the physical arrangement of the discs. Such hierarchical arrangement allowed us to use file system tools to generate some of the metadata automatically. There are a number of freeware, shareware, and commercial applications for Macintosh that will catalog a file volume and produce reports. We used a shareware utility called CatFinder to index the copied files and export a report to a delimited format that was imported into Excel. This report formed the basis of our item-level metadata, including fields for filename, file size, kind (document or folder), Macintosh file type (analogous to the Windows file extension), Macintosh creator code, creation date, and modification date. To this basic report we added a comments field for use during appraisal and to collect technical notes.

MD5 file hashes were also generated for each file. Having an MD5 hash for each file allowed us to do two important things: to identify and/or eliminate redundant files, and to support provenance auditing during the repository ingest process. A freeware PERL application called Integrity automatically created MD5 hash calculations and exported the results to a delimited text file. Unfortunately, integrating the MD5 hashes into the CatFinder index was not trivial due to differences between the two applications in file name recursion and handling of hidden files.

Having created a unified index of filesystem metadata, augmented with processing notes and MD5 hashes, we were able to more accurately assess the extent of the digital files and facilitate arrangement and appraisal. Unfortunately, the index was in no way tied to the digital files and presented us with a significant information management problem. For example, any movement of files was not automatically noted in the index, nor was any change or deletion in the index reflected in the filesystem. We can envision a workflow-oriented system that stands between the filesystem and a metadata database that would greatly increase the speed and reliability of processing large bodies of digital documents.

Arrangement

(Note: More detail about our project can be found in a forthcoming article by Catherine Stollar about processing the Michael Joyce Papers in Provenance.)

After recovering most of the unique digital content from the first accession of floppy disks, we began the process of archival arrangement. In the beginning, we asked ourselves some questions. Can and should digital files be arranged like paper-based records? Should we heed traditional archival arrangement practices or follow newer theories of arrangement based on item-level metadata? Do electronic records have a natural hierarchy that can be expressed in a traditional arrangement? Should physical housing for digital materials be kept? If so, where? Our answers to these questions are not definitive, but we came to a compromise incorporating basic tenets of archival theory with features of on-demand, flexible file arrangement using item-level metadata.

A number of digital materials, including emails and published articles, within the archive had a paper-based counterpart, demonstrating that Michael Joyce created both digital and analog records while performing the same activities. Both formats of records were created synchronously, and at an institution like the Ransom Center that preserves not only works that have influenced the arts and humanities fields, but also preserves the context in which those works were created, we determined it would be desirable to reflect synchronous creation in the arrangement. We did not originally understand relationships between Joyce's digital and paper materials because our first portion of the case study only dealt with electronic records from the first accession of floppy disks. We initially arranged the files into 5 series: Works, Academic Career, Correspondence, Storyspace, Third-party Works, and Personal. After surveying the paper-based materials and the second accession of electronic materials, we had to alter our originally arrangement to include the newly accessioned materials. The final arrangement we created is Works and Related Materials, Academic Career, Correspondence, Storyspace, Journals and Appointment Books, Personal, Works by Other Authors, and Published Materials.

Institutional repositories like DSpace can facilitate digital object arrangement into our specified series by using the community, sub-community, collection, sub-collection, and item level hierarchies. DSpace's hierarchies relate to traditional archival hierarchical levels: communities equate to archival fonds, sub-communities to series and sub-series, collections as other layers of granularity within a series, and item-level entries relate to digital objects. In an additional level of granularity, items composed of multiple sub-components or related files, i.e.: websites with multiple linked HTML files can be ingested as bundled files.

After determining how to arrange the paper and digital materials, we decided how to arrange the physical housing (jewel cases, magnetic media, paper holders, plastic cases, etc.) from Joyce's electronic works. Previous policies and procedures at the Ransom Center dictated that electronic media should be physically housed in Hollinger boxes separate from the rest of the paper-based materials. This separation policy apparently arose out of concern for potential damage to other materials caused by degrading electronic media and to limit access to the electronic materials by researchers. No studies on electronic media degradation have found any instances of off-gassing or other damaging effects of filing electronic media with paper-based materials, so we determined physically integrating paper-based material and digital media would be the best policy for physically arranging the Michael Joyce Papers. The Ransom Center will still limit access to files saved on original media because researchers will have access to the files via DSpace.

Although we integrated Joyce's digital objects into a functional group arrangement similar to his paper-based records, we also took advantage of the flexible, non-linear nature of digital object arrangement by enabling on-demand, user-controlled arrangement by item-level metadata. Preservation of digital objects depends on item-level metadata used to document, migrate, emulate, and preserve the objects. Item-level metadata recorded for preservation in DSpace's database also enables flexible arrangement of digital objects. Digital arrangement allows archivists, and users, multiple options for organizing objects depending on the parameters set by the user interface, such as file name, title, author, date created, subject, or other metadata element. Arrangement is limited only by the skills of the programmer developing the user interface used to access the OAIS database and the precision of metadata recorded for each object.

Arrangement is also affected by how we ingested objects into DSpace because our method of ingest affected what metadata fields we included. Although manual metadata assignment of all files within the Joyce archive was laborious, certain metadata fields were impossible to record automatically. Content metadata, such as subject and title of work, had to be entered by hand because automatic tools to accurately extract content were not available.¹ We found it difficult to use file names within the archive to associate files with published titles because the file names were not specific or standardized.

We incorporated methods for traditional archival arrangement and strategies for on-demand item-level arrangement while processing digital objects within the Michael Joyce Papers. Together, both methods allow users to browse records according to functional series and create new arrangements based on any metadata available for individual objects.

¹ Literary text comparison tools designed for use with small numbers of digital works were not sufficient for our large collection of files. Apparently text-mining tools could serve our purposes to compare large bodies of records with each other. We have not utilized any text-mining tools to date.

Challenges

In addition to the challenges we encountered developing a strategy to preserve Joyce's text and graphic files, we faced unique challenges associated with preserving Joyce's most influential creative works written using specialized software called Storyspace. Storyspace, created by Michael Joyce, Jay David Bolter, and John B. Smith, as a format type presented (and continues to present) challenges for media migration, ingest, and file use. Hypertext works written in Storyspace are composed of multi-faceted texts linked by guard fields (words within texts that enable direct links to other nodes, usually under specific conditions) and can only be viewed using Storyspace software. To complicate matters, we originally thought the latest version of Storyspace was backwards compatible and could read works written in the first version of Storyspace. Unfortunately, this is not entirely the case as older Storyspace documents do not degrade gracefully. For example, the text from files written in Storyspace 1.5 can be read in Storyspace 2, but the individual nodes and links are missing, making the Storyspace 2 rendering of a older work vastly different from the original.

New Skills

A thorough grounding in the various operating systems. The profusion of technical difficulties and operating system inconsistencies required an intuition about the various platforms that can only be gained by direct experience. While processing digital files it is essential to have an understanding of the environment in which they were created. Computer literacy in more than one platform and with networked environments will be ideal traits for archivists of the future.
A basic understanding of the structure of digital documents. Knowing how a digital file is created and stored, including such basics as the difference between binary file formats and textual formats such as ACSII and UTF helps provide an understanding of what happens during processing. Furthermore, an intimate knowledge of the types of formats and how they might be identified (e.g.: file type extensions or creator codes), accessed, and converted is essential. Understanding digital formats offers clues to where to find item-level metadata (e.g.: document properties embedded in word processing files, ID3 tags embedded in MP3 audio files, etc.) and suggests migration paths for long-term preservation.
Proficiency with and trust in new tools. Familiar means of handling physical documents are not present with digital documents. Software tools and operating systems augment the functions of our senses in the digital world and mitigate some of this loss, but not completely. Integrated toolkits and processing systems are needed and must be developed so that they can be trusted to conform to the expectations of archival practice.
Establishment of new workflows and procedures. The intangible nature of digital information makes documentary evidence crucial to processing. Many institutions have established procedures for document processing, including audio/visual materials, but these cannot be assumed to be sufficient for digital objects. Operating systems alone cannot document processes, so new systems that function according to sound processing policies are necessary.
Ability to monitor current trends in digital preservation including metadata standards, crosswalks between encoding standards, available tools, storage systems, file format repositories, national and international research initiatives, user expectations, and published best practices guides.
A thorough understanding of traditional archival theory and practice. Archivists who work with digital records should be able to extrapolate traditional theory and apply it to electronic record preservation, but must be flexible enough to create new standards for archival practice. What we do as archivists will change (practice), but why we do it will not (theory).

Discussion Questions

Would automatic content management be more time consuming or less time consuming that manually arranging the digital manuscripts? Would it result in a better arrangement?
Is it feasible to devise a one-size fits all processing toolkit?
How can authors, who may deposit their materials in an institution like the Ransom Center, implement a digital preservation strategy at home, closer to the point of document creation?
How desirable is it to keep most files in proprietary formats that are the current de facto standard? (i.e. Microsoft Word, Adobe PDF, etc.)
Should files be arranged at all or should they be indexed and sorted using search engines using item-level metadata?
Is DSpace a viable option for smaller repositories and organizations?
How will DSpace integrate into existing points of access? (i.e. OPACs, website, EAD consortium sites)
How do archivists best obtain the skills we are advocating they have? Classes? Projects? Workshops? Conferences?

Attachment	Size
4_Stollar_Kiehne.pdf	134.9 KB

An OAIS Ingest Metadata Specification

tkiehne — Tue, 14 Dec 2004 06:40:48 +0000

Problem Definition

For this exercise, we will prepare a digital object for submission to a digital archive for long term preservation. The digital object in question is an HTML text with an in-line image and links to several other HTML texts. The objects must be readable, but the specific look and feel of the rendered text is not important. We are to generate a metadata set that will conform to ingest requirements for an Open Archival Information System (OAIS) according to a Submission Information Package (SIP) agreement. Additionally, we will consider the process for converting a SIP into an Archival Information Package (AIP) and extend the metadata set with additional elements for the conversion process, as needed.

OAIS Ingest and SIP

The OAIS model outlines a system-agnostic framework to ensure reliable, long-term preservation of information. The process begins with the submission of an information package from a producer of information (publisher, author, researcher, etc.) to the archive. The information is submitted as one or more SIP objects that conform to the archive's SIP agreement. For a digital archive, the information is sent electronically along with descriptive metadata. The components of an OAIS information package are shown in Figure 1 (CCSDS, 2002, §4).

Figure 1: OAIS information package (CCSDS, 2002, p. 4-31)

As shown above, the information package is comprised of Content Information (CI) and Preservation Description Information (PDI). In the current exercise, the content information consists of the HTML code and image file, or pointers to these resources, and the representation information necessary to decode the content of the digital objects. The Packaging Information binds the CI and PDI by some means, including universal identifiers, encoding (such as used for a CD-ROM), or some other system-specific method.

Descriptive information is derived from the PDI or declared prior to submission. PDI contains the metadata needed for conversion into an AIP and further storage within and access from the repository. Four functional areas are defined in the PDI as summarized in Table 1

Provenance	Content	Reference	Fixity
Source Description and pointer to original object(s) Copyright/legal restrictions Access restrictions Authority to modify representation information or migrate Agreements with external organizations History Change history Pointers to other versions Custody since origination	Relationship to other objects Description Pointers to other objects Purpose/Reason for creation Reason for creation Reason for archiving Encoding environment Software Languages Character set(s)	Unique identifier(s) URI, ID Number, etc. Bibliographic description Creator(s), organization(s) Date of creation Title(s) Etc.	Authenticity indicators Checksum, CRC, MD5 hash Digital signatures Encryption Quality of service requirements Specification of integrity preserving mechanisms Error protection specifications

Provenance

Content

Reference

Fixity

Source

Description and pointer to original object(s)
Copyright/legal restrictions
Access restrictions
Authority to modify representation information or migrate
Agreements with external organizations

History

Change history
Pointers to other versions
Custody since origination

Relationship to other objects

Description
Pointers to other objects

Purpose/Reason for creation

Reason for creation
Reason for archiving

Encoding environment

Software
Languages
Character set(s)

Unique identifier(s)

URI, ID Number, etc.

Bibliographic description

Creator(s), organization(s)
Date of creation
Title(s)
Etc.

Authenticity indicators

Checksum, CRC, MD5 hash
Digital signatures
Encryption

Quality of service requirements

Specification of integrity preserving mechanisms
Error protection specifications

Table 1: Elements of PDI (CCSDS, 2002, pp. 4-27 – 4-29)

A Model SIP

When approaching the problem of metadata creation, two basic approaches may be used. First, we may generate a new metadata set that conforms to the archives requirements. Generating a new set allows the specific project requirements to be addressed in detail, but requires a significant amount of labor to produce the DTD or schema and the tools to use them. Furthermore, a new metadata set does not leverage existing standards and practices for interchange. The second, and preferred, approach is to use or extend an existing metadata specification.

A review of existing metadata specifications reveals that the Metadata Encoding and Transmission Standard (METS) provides a framework for many of the necessary elements for a SIP (Library of Congress, 2004). The first two columns of Table 2 show a basic mapping of METS container elements to OAIS ingest information. Since METS does not provide detailed descriptive metadata elements, other metadata schemes must be used to complete the SIP. Source metadata schemes and sets for detailed metadata are shown in the third column of Table 2. External metadata sets can be linked from the METS container using <mdRef>, or embedded within using one of the metadata types allowed in <mdWrap>.

OAIS Ingest Information	METS Element Location(s)	Source for Specific Metadata
Preservation Description Information	all elements nested under <mets> root
Provenance
Description of original object(s)	<amdSec> <sourceMD>	Metadata Object Description Schema: <relatedItem type=”original”>
Rights management information	<amdSec> <rightsMD>	Rights Declaration Extension Schema: <RightsDeclarationMD>
Access restrictions	<amdSec> <rightsMD>	Rights Declaration Extension Schema: <RightsDeclarationMD> <Context>
Agreements with creator(s)	<amdSec> <rightsMD>	Rights Declaration Extension Schema: <RightsDeclarationMD> <Context>
Agreements with external organizations	<amdSec> <rightsMD>	Rights Declaration Extension Schema: <RightsDeclarationMD> <Context>
Change history	<amdSec> <digiprovMD>	Metadata Object Description Schema: <originInfo> <dateModified>
Pointers to other version(s)	<amdSec> <digiprovMD>	Metadata Object Description Schema: <relatedItem type=”otherVersion”>
Custody since origination	<amdSec> <digiprovMD>	Metadata Object Description Schema: <note type=”bibliographic history”>
Content
Relationship to related objects	<amdSec> <digiprovMD>	Metadata Object Description Schema: <relatedItem type=”XXX”>
Pointers to related objects	<amdSec> <digiprovMD>	Metadata Object Description Schema: <relatedItem xlink=”XXX”> <location>
Reason for creation	<amdSec> <digiprovMD>	Metadata Object Description Schema: <note type=”bibliographic history”>
Reason for archiving	<amdSec> <digiprovMD>	Metadata Object Description Schema: <note type=”conservation history”>
Original encoding/technical environment	<amdSec> <techMD>	Schema for Technical Metadata for Text: <byte_order>, <charset>, <encoding>, <markup_language>
Reference
Unique identifier(s)	<dmdSec>	Metadata Object Description Schema: <identifier>
Bibliographic description	<dmdSec>	Metadata Object Description Schema: <titleInfo>, <name>, <subject>, <language>, <typeOfResource>, <genre>, <abstract>, <originInfo>, <physicalDescription>
Fixity
Authenticity indicators	<fileSec> <fileGrp> <file CHECKSUM=”XXX”>	N/A
Quality of service requirements	<amdSec> <techMD>	Schema for Technical Metadata for Text: <viewingRequirements>
Content Information	all elements nested under <mets> root
Data object(s)	<fileSec> <fileGrp> <file> <FLocat> <FContent> <binData> \| <xmlData>	N/A
Packaging Information	all elements nested under <mets> root
Relationships between data objects	<structMap> <div> <mptr> \| <fptr> <structLink>	N/A

Table 2: Basic OAIS ingest and METS container mapping (derived from: Tingle, 2004; Library of Congress, 2003; Library of Congress, 2004b)

Augmenting the SIP to Facilitate Conversion to an AIP

Figure 2 illustrates the complete OAIS ingest process. Once all of the component SIPs have been received and checked for conformance to the archive's ingest specifications, the SIPs are enhanced and reformatted according to the archive's technical standards, as necessary. Workflow procedures may require an administrative audit of the submitted information package. Once these processes are complete, the AIP is created, descriptive metadata and “browse products” are extracted, then the AIP is sent to data management.

Figure 2: OAIS ingest process (CCSDS, 2002, p. 4-5)

In the process of converting SIPs to an AIP, new metadata is created. To ease the conversion process, the SIP format should include element definitions for these items of information. Additional metadata for the SIP to AIP conversion are shown in Table 3 along with the METS container location and prescribed metadata element source.

Additional Information	METS Element Location(s)	Source for Specific Metadata
Ingest history	<amdSec> <digiprovMD>	Metadata Object Description Schema: <note type=”acquisition”> <note type=”conservation history”>
Quality assurance (QA) results	<amdSec> <techMD>	Schema for Technical Metadata for Text <processingNote>
History of formatting and encoding changes made during conversion process	<amdSec> <digiprovMD>	Metadata Object Description Schema: <note type=”conservation history”>
Administrative audit reports	<amdSec> <digiprovMD>	Metadata Object Description Schema: <note type=”admin”>
Additional descriptive or classification data	<dmdSec>	Metadata Object Description Schema: <classification>, <subject>
Relationship to archive objects or collections	<amdSec> <digiprovMD>	Metadata Object Description Schema: <relatedItem>
Internal identifier(s)	<dmdSec>	Metadata Object Description Schema: <identifier>

Table 3: Additional metadata required for SIP to AIP conversion (derived from: Tingle, 2004; Library of Congress, 2003; Library of Congress, 2004b).

Example SIP Profile

The appendix contains a METS Profile Schema document describing the requirements enumerated in Tables 2 and 3. At the end of the profile is an <Appendix> element containing an example ingest package encoded according to the profile specification.

References

Consultative Committee for Space Data Systems (CCSDS) (2002). Reference model for an Open Archival Information System (OAIS). CCSDS 650.0-B-1 BLUE BOOK. Retrieved on 28 November, 2004, from http://ssdoo.gsfc.nasa.gov/nost/wwwclassic/documents/pdf/CCSDS-650.0-B-1.pdf.

Library of Congress (2003). METS news and announcements: Draft rights declaration schema is ready for review. Retrieved on 11 December, 2004, from http://www.loc.gov/standards/mets/news080503.html.

Library of Congress (2004). METS: An overview & tutorial. Retrieved on 7 December, 2004, from http://www.loc.gov/standards/mets/METSOverview.v2.html.

Library of Congress (2004). Metadata Object Description Schema (MODS). Retrieved on 11 December, 2004, from http://www.loc.gov/standards/mods.

McDonough, J. (2003). Schema for Technical Metadata for Text. Retrieved on 11 December, 2004, from http://dlib.nyu.edu/METS/textmd.html.

Tingle, B. (2004). METS 1.3 schema documentation. Retrieved on 9 December, 2004, from http://ark.cdlib.org/mets/schema_documentation.

Attachment	Size
mets_sip_example.xml	18.98 KB

TEI Lite History and Evaluation

tkiehne — Tue, 30 Nov 2004 05:57:09 +0000

New and disparate ways of digitally encoding texts were developed as computing became available to scholars of the humanities in the 1980s. The encoding of textual objects into a digital form creates opportunities for examining old and rare texts simultaneously and without the risk of wear or damage to the original object. Additionally, an encoded object permits new ways of interacting with the text, such as concurrent views of different versions and viewing subsequent editorial or annotations. The lack of standard methods for encoding and describing texts made it difficult for researchers to exchange objects and diminished the benefits that the digital format offers.

The Text Encoding Initiative (TEI) was conceived in this disjointed digitization environment. TEI is a successful and influential metadata encoding standard that is primarily concerned with the encoding of textual objects, but is flexible enough to apply to many other types of information objects. The standard is customizable and extensible. One such customization is TEI Lite, a subset of the TEI specification. In this essay we will examine the development and history of TEI Lite as well as the role it plays in documenting the lifecycle of digital objects. TEI Lite's relationship to other metadata initiatives will also be explored. Finally, an evaluation will be made of how well TEI achieves its purpose and some of the problems the specification faces.

History of TEI Lite

TEI Lite shares its formative history with its superset, TEI. Work on TEI formally began in 1987 with the meeting of a group of 32 scholars from North America, Europe, and Asia held at Vassar College in Poughkeepsie, NY. The initial meeting was convened by the Association for Computers in the Humanities and funded by the National Endowment for the Humanities with the purpose of beginning work on the problems facing digital text encoding (Mylonas & Renear, 1999, pp. 3-4).

At the close of the conference, the group issued a closing statement to provide direction for the development of guidelines. The statement, known as the “Poughkeepsie Principles,” directed that the forthcoming guidelines should (Burnard & Sperberg-McQueen, 2002, p. 1):

suffice to represent the textual features needed for research;
be simple, clear, and concrete;
be easy for researchers to use without special-purpose software;
allow the rigorous definition and efficient processing of texts;
provide for user-defined extensions;
conform to existing and emergent standards.

After the meeting in 1987, three organizations participated in forming the guidelines: the Association for Computers in the Humanities, the Association for Literary and Linguistic Computing, and the Association for Computational Linguistics (Mylonas & Renear, 1999, p. 3). Draft versions of the TEI Header and Guidelines were completed and distributed in 1990 (MIT Libraries, 2004). After several years of refinement, the final draft (version P3) was released in 1994. “Guidelines for the encoding and interchange of Machine-Readable Texts” spanned 1300 pages and defined over 600 elements of Standardized General Markup Language (SGML). The TEI specifications defined an extensible set of elements that could be customized by user communities for their specific needs. One of these customizations is TEI Lite, which defines a subset of TEI meant to serve as a “starter set” of core elements to assist in learning the extensive TEI set (Burnard, 2000).

The P3 guidelines underwent several minor revisions between 1994 and 2001, mostly to clarify varying interpretations and practices (Burnard & Popham, 1999, p. 39). During this time, however, the success of TEI as a metadata specification informed and influenced the development of the eXtensible Markup Language (XML) (DeRose, 1999). TEI was subsequently converted to XML and released as version P4 in Summer 2001 (MIT Libraries, 2004). Development of TEI Lite continues in parallel with TEI, the next revision of which (P5) is expected to be released at the end of 2004 (TEI Consortium, 2003, How to participate - Next version).

Formal structures were developed to guide future development as participation increased. An Executive Committee formed in the mid-nineties that included representatives from each of the three sponsoring associations and two influential researchers, Michael Sperberg-McQueen (University of Illinois at Chicago) and Lou Burnard (Oxford University). By 1996, a Technical Review Committee was established to conduct the development and maintenance of the guidelines in a manner similar to the International Standards Organization (ISO) (Burnard & Light, 1996, pp. 25-26).

In 1999, the Executive Committee was petitioned to create an international membership organization that could better handle the TEI's increasing administration and development responsibilities. The petition resulted in the formation of a non-profit corporation (Burnard, 2000). Membership in the consortium includes dozens of agencies from the humanities, education, computing, linguistics, and librarianship. Members elect a technical council that oversees development of the guidelines and funding for the organization. The consortium's first Council was elected in 2001 and met for the first time in 2002. Members may also participate in the various special interest groups or workforces that develop the guidelines (TEI Consortium, 2004, How to participate). The Consortium relies on its members to expand TEI's user base and has chartered a special interest group for training to support their efforts (TEI Consortium, 2004, How to participate – Special interest groups).

TEI is hosted by four universities and is sponsored by the three associations originally responsible for initial development of the guidelines. Significant support is provided by the U.S. National Endowment for the Humanities (NEH), Directorate XIII of the Commission of the European Communities (CEC/DG-XIII), the Andrew W. Mellon Foundation, and the Social Science and Humanities Research Council of Canada (TEI Consortium, 2004).

The Functional Role and Structure of TEI Lite

TEI and TEI Lite intend to define a framework for the encoding of texts that facilitates the interchange of digital objects. The specification defines a common and extensible language that different software platforms can understand and use to render the digital object in consistent ways. Although the development of TEI has focused on the encoding of texts, particularly capturing non-digital texts, the framework is applicable to the description of non-text objects such as images and sound (Burnard & Sperberg-McQueen, 2002, p. 1).

The description and interchange goals of TEI implies fidelity to the structure and content of the object being encoded. As such, much of the focus of TEI is on the structural description of textual objects, while the TEI header supports most of the lifecycle metadata functions (see Table 1). Elements of the TEI header provide creation, appraisal and descriptive metadata and, to a lesser extent, transfer/authenticity and preservation metadata. Accession and usage metadata are much less apparent, but may be augmented by the information system that stores the digital object. Rights metadata is simply represented in regard to the original object. In fact, the distinction between metadata about the digital encoding and metadata about the original object is difficult to discern from the element definitions and likely results in differing practices. In general, the file description (fileDesc) describes attributes of the original text while the encoding description (encodingDesc) concentrates on aspects of the digital implementation

Creation	Appraisal	Transfer/Authenticity	Accession
<fileDesc> <titleStmt>(all) <profileDesc> <creation> <revisionDesc>	<fileDesc> <editionStmt>(all) <seriesStmt> <sourceDesc> <encodingDesc> <projectDesc> <samplingDecl> <editorialDecl>	<fileDesc> <extent> <publicationStmt> <publisher> <distributor> <authority>	<fileDesc> <publicationStmt> <authority> <notesStmt> (also defined by containing system)
Descriptive	Preservation	Usage	Rights
<fileDesc> <titleStmt> <title> <editionStmt> <edition> <seriesStmt> <sourceDesc> <profileDesc> <textClass>	<fileDesc> <extent> <encodingDesc> <tagsDecl>(all) <refsDecl> <profileDesc> <langUsage>	<revisionDesc> (also defined by containing system)	<fileDesc> <publicationStmt> <publisher> <availability> <distributor> <availability> <authority> <availability>

Table 1: Metadata life-cycle roles of TEI header elements. (derived from Burnard & Sperberg-McQueen, 2002)

A complete TEI Lite document contains a header and text body (see Figure 1). The header, as indicated above, contains metadata related to the digital object and the original information object. The header is separable from the encoded body, which allows it to serve as a description for non-text objects stored separately from the header. The TEI header is analogous to the title page of a text. It has up to four parts: a description of the electronic file (fileDesc), an encoding description (encodingDesc), a non-bibliographic description of the text (profileDesc), and a revision history (revisionDesc)(Burnard & Sperberg-McQueen, 2002, p. 6). Of these, only the file description is required, the elements of which can be related directly to MAchine Readable Cataloging (MARC) fields. Unlike MARC, elements of the TEI header are not required to conform to a controlled vocabulary such as described by the Anglo American Cataloging Rules (AACR), although such rules may be applied at the encoder's discretion (Pouchard, 1998).

Figure 1: Structure of a TEI Lite document (HTML Writers Guild, 2001)

For textual objects, structural encoding is defined within the text element. A TEI text may contain a single, unitary work, or a group of works as realized in a series or anthology. For the latter case, the text element may contain an arbitrary number of group elements, each containing a text body with optional front and back matter. Additionally, multiple TEI objects may be grouped as a corpus, analogous to a collection of texts (Burnard & Sperberg-McQueen, 2002, p. 7).

The range of elements and the relatively relaxed markup rules allow for varying granularity depending on the intended usage. The body of an encoded text is structured by p and div elements, similar to those in HyperText Markup Language (HTML), that represent chapters, sections, and subsections of a text. Text within these structures may be further encoded using a myriad of markup that indicate layout and appearance. Furthermore, elements are available for defining alternate appearances or versions of text and editorial markup or annotations as applied to the original object. Additionally, elements such as unclear allow for the indication of unintelligible or damaged areas of text (Burnard & Sperberg-McQueen, 2002). Such elements enhance textual analysis by allowing the encoding of multiple version of a text within the same electronic file.

Relationship to Other Metadata Initiatives

TEI was one of the first metadata initiatives, predated only by MARC and the International Standard Bibliographic Description (ISBD), AACR, and SGML standards (Burnard & Light, 1996). The TEI header and the descriptive fields of later versions of MARC closely resemble the functional structure of the ISBD. Despite the structural similarities with MARC and ISBD, however, TEI does not require the use of controlled vocabulary and as such does not readily convert to either standard. Early TEI development eschewed strict cataloging requirements in the expectation that non-catalogers would use the specification. The decision to conform to standard cataloging practices is left to the creating agency (SCHEMAS Registry, 2002). Such a flexible approach favors ease of use over uniformity in order to facilitate a wider adoption of the standard (MIT Libraries, 2004) - an approach that Dublin Core has also uses.

Metadata initiatives developed subsequent to TEI have benefited from TEI's success and derive structures from TEI Lite. Encoded Archival Description (EAD) borrowed TEI's header concept (Burnard & Light, 1996, p. 13). Other metadata initiatives are domain specific applications of TEI. The Consortium for the Computer Interchange of Museum Information (CIMI) uses the TEI framework for the description of museum resources (Burnard & Light, 1996, p. 15). Another derivation is the Spoken Text Markup Language (STML), a text to speech markup language inspired by TEI (Sproat, 1997). Similarly, the Music Encoding Initiative (MEI) was based on TEI (Roland, 2002).

Most notable of TEI influences was in the development of XML. TEI represented the first and most precise SGML implementation at the time of XML's development. As a result, developers of TEI were closely involved in defining XML. Especially useful to the nascent XML specification was TEI's extended pointer language which served as a prototype for XLink and Xpointer (DeRose, 1999).

Evaluation of TEI Lite

TEI Lite was created to present a useful subset of TEI that provides the elements necessary for most common encodings. The 140 elements of TEI Lite represent only a fraction of the hundreds of elements available in TEI and its extensions. The majority of the subset, besides those in the header, define basic structural and perceptual attributes necessary for textual objects, but not so many as to become overly granular. Additionally, the use of a lesser number of elements restricts the size of the metadata vocabulary that different agencies need to have in order to understand conventions used during encoding and markup. The subset represents a lowest common denominator of sorts that is compliant with and upgradeable to the full TEI specification.

There are a number of criticisms with TEI encodings. First, as mentioned previously, the header lacks a controlled vocabulary for bibliographic elements. There is a compromise between usability from the perspective of creation and accessibility in terms of resource location. Free text bibliographic descriptions, however, could prove to be more useful for scholars of ancient texts which, by their unique character, require more detailed descriptions than those afforded in library cataloging (Pouchard, 1998). The upcoming P5 version of TEI will allow external metadata and namespaces to be included in TEI documents (TEI Consortium, 2004, Guidelines - P5 status). Embedding MARC encoded data may offer a solution to controlled vocabulary problem, although it is uncertain how such features will cascade into TEI Lite.

Second, texts may overlap semantic and organizational structures. XML and TEI are hierarchical languages that require inelegant procedures to represent such overlapping structures. The overlap problem is especially pertinent to representing variant structures beyond the word or character level such as macro-level versions and variations (Smith, 1999).

Third, the reduced set of elements available in TEI Lite reduces the chance of over-granular structure, but divergent encoding practices are still possible. The basic structural elements (p and div) and their attributes may be used differently and result in confusion when encoded documents are exchanged. Numerous “best practices” standards have been created to help alleviate variation within institutions (TEI Consortium, 2004, Tutorials). The loosely prescribe structuring rules, however, demand that TEI rendering tools be just as flexible and not beholden to a particular encoding practice.

Finally, the basic assumptions underlying the use of structural elements creates problems for representing the physical structure of a work. TEI is based primarily on encoding the intellectual structures of a text, such as chapters, acts, volumes, and other semantic containers. Such assumptions preempt encoding structures based on physical attributes of the container, such as the sequence of formes in early printed texts (Bauman & Catapano, 1999). The scope of this problem may be beyond the capabilities of TEI Lite and require use of the larger element set of TEI.

Despite these criticisms, TEI Lite successfully achieves its goal of providing a readily adaptable point of entry to TEI. Furthermore, TEI Lite sufficiently addresses the domain problems that TEI was meant to solve. We can judge the 1987 Poughkeepsie Principles in terms of the current implementation of TEI Lite: TEI Lite provides for simple, clear, and concise representations of textual objects; Expression in XML allows for efficient processing, the use of non-specialized software, and conforms to existing standards; Structural definitions in TEI Lite are not as rigorous as TEI, and the user may not extend TEI Lite freely, however, upward compatibility between the specifications provides a solution.

In addition to basic principles, we may judge success by the degree to which TEI Lite has been adopted. The Oxford Text Archive and the Electronic Text Centers at the University of Virginia and the University of Michigan use TEI Lite to encode their holdings. The TEI Consortium uses TEI Lite in its technical documentation (Burnard & Sperberg-McQueen, 2002, p. 2). Additionally, a significant number of the projects listed on the consortium Web site use TEI Lite (TEI Consortium, 2004, Projects using TEI) and a cursory Web and journal search reveals that TEI Lite is frequently used for encoding projects and research.

Conclusion

TEI Lite is an introductory subset of TEI, one of the earliest metadata initiatives. The encoding standard blends a flexible implementation with established descriptive principles. The result is a metadata set that is easy to apply and capable of describing many types of objects. The success of TEI, representing the efforts of scholars worldwide, has informed the development of many subsequent metadata standards and influenced the development of XML. Development of the standard continues as does its increased use in projects for a variety of domains.

References
(see also pathfinder & annotated bibliography)

Bauman, S. & Catapano, T. (1999). TEI and the encoding of the physical structure of books. Computers and the Humanities, 33(1/2), 113–127.

Burnard, L. & Sperberg-McQueen, C. (1995, updated 2002). TEI Lite: An introduction to text encoding for interchange. Retrieved on 18 September, 2004, from http://www.tei-c.org/Lite/teiu5_en.pdf.

Burnard, L. & Light, R. (1996). Three SGML metadata formats: TEI, EAD, and CIMI: A Study for BIBLINK Work Package 1.1. Retrieved on 18 September, 2004, from http://www.ifla.org/documents/libraries/cataloging/metadata/biblink2.pdf.

Burnard, L. & Popham, M. (1999). Putting our headers together: A report on the TEI header meeting 12 September 1997. Computers and the Humanities, 33(1/2), 39–47.

Burnard, L. (2000). Text encoding for interchange: A new consortium. Ariadne, 24(21 June 2000). Retrieved on 16 September, 2004, from http://www.ariadne.ac.uk/issue24/tei/.

DeRose, S. (1999). XML and the TEI. Computers and the Humanities, 33(1/2), 11–30.

HTML Writers Guild (2001). An introduction to the Text Encoding Initiative (TEI), DTD. Retrieved on 26 November, 2004, from http://gutenberg.hwg.org/teidtds.html.

MIT Libraries (2004). MIT metadata reference guide: TEI (Text Encoding Initiative) metadata. Retrieved on 16 September, 2004, from http://libraries.mit.edu/guides/subjects/metadata/standards/tei.html.

Mylonas, E. & Renear, A. (1999). The Text Encoding Initiative at 10: Not just an interchange format anymore – But a new research community. Computers and the Humanities, 33(1/2), 1–9.

Pouchard, L. (1998). Cataloging for digital libraries: The TEI scheme and the TEI header. Katharine Sharp Review, 6(Winter 1998). Retrieved on 18 September, 2004, from http://alexia.lis.uiuc.edu/review/6/pouchard.html.

Roland, P. (2002). The Music Encoding Initiative (MEI). Musical Applications using XML (MAX) 2002 Conference. Retrieved on 21 November, 2004, from http://dl.lib.virginia.edu/bin/dtd/mei/maxpaper.pdf.

SCHEMAS Registry (2002). Activity reports: Text Encoding Initiative. Retrieved on 16 September, 2004, from http://www.schemas-forum.org/registry/desire/activityreports.php3
?field=filename&value=TEI_D29D35(RDF).rtf.

Smith, D. (1999). Textual variation and version control in the TEI. Computers and the Humanities, 33(1/2), 103–112.

Sproat, R., Taylor, P., Tanenblatt. M. & Isard, A. (1997). A markup language for text-to-speech synthesis. 5th European Conference on Speech Communication and Technology, Rhodes, Greece, September 22-25, 1997. Retrieved on 21 November, 2004, from http://www.talkingheads.computing.edu.au/resources/documents/serge/
Sproat/A%20Markup%20Language%20for%20TTS%20Synthesis-Sproat.pdf.

TEI Consortium (2004). Text Encoding Initiative. Retrieved on 16 September, 2004, from http://www.tei-c.org.

Attachment	Size
TEI-Lite_pathfinder.pdf	235.58 KB