Digital preservation as a service

This post is based on a presentation to eResearch NZ in March 2015.

Digital Preservation as a Service (DPaaS) is a joint project of National Library of New Zealand and Archives New Zealand. The current phase of the project is to determine the demand/need for a whole of country approach to digital preservation leveraging government’s investment to date in the Library and Archives.

What’s digital preservation?

It is:

  • ‘the active management of digital content over time to ensure ongoing access’ (Library of Congress)
  • a ‘series of managed activities necessary to ensure continued access to digital materials for as long as necessary’ (Digital Preservation Coalition)
  • despite ‘the obsolescence of everything’ (Michael Factor)

And it’s not:

  • backup and disaster recovery – these are short term business functions supporting the ‘ability to maintain the continuity of the organization, focusing on the restoring of data in its current state’ (Marshall Breeding) and for current use. Backup and disaster recovery do not ensure, over time, availability of software to access the file, dealing with obsolete file formats, rights issues, issues of authenticity and provenance, context, semantic understanding
  • only about access or ‘open access’ – although digital preservation is necessary for ongoing access in the digital world - ‘without preservation, there will be no access, open or otherwise’ (Blue Ribbon Task Force)
  • ‘an afterthought’ (Edward M. Corrado)

Digital preservation requires interaction with all the organisation’s processes and procedures and institutional support for appropriate resources, human or capital. It ought to matter to everyone.

What loss looks like

What are we preserving against? In 1986, the BBC Domesday project set out to record the economic, social and cultural state of Britain, i.e. ‘how Britain looked to the British in 1986’. The technology used was 12’’ videodiscs. Today, those disks cannot be read, unlike the Domesday Book itself, written a thousand years earlier.

The following real scenarios are less high-profile, but extremely common. In an incredibly short time, millions of dollars and years of work go to waste because systems were built without considering long-term digital preservation.

  • An engineer responds to external requests for their engineering and financial data and models. They receive a request for data that was stored on a 7 inch floppy disk, but there is no longer a drive to run it on. A nation-wide search fails to find one: the data remains inaccessible.
  • Hydrology modelling software is used to create thousands of input and output files. The software is upgraded to a new version. Unfortunately, staff find they cannot access data that was created on previous versions and have to send it back to the vendors to open, costing significant time and money.
  • An organisation has footage stored on video tape and DVDs. This information is used to inform budget projections, and to show where pipes and culverts are located. They can no longer access videos, because they have nothing to view them on. Where they can’t access the footage, they have to contract people to re-shoot it.
  • An organisation has an established practice of migrating digital information through systems as technology changes. The Infrastructure Group recently moved to a new asset management system. During the migration, 10 years’ worth of service data disappeared. This data is needed to log repairs against, and predict future trends. By lucky coincidence, paper copies of reports covering some of this period have been printed off.
  • In the above example some raw data has been backed up, but it can’t be manipulated without specialist software. The software in the legacy asset database has not been upgraded. Compounding this, vendors have ‘time-locked’ this software, meaning that after a period of time the organisation is not able to manipulate their data as they need to. The replacement system also has time-locked software.
  • When an earlier EDRMS was implemented, the system was configured to save all emails as .VMBX files instead of .msg. files. .VMBX is a proprietary file format, and when information was migrated across to the new EDRMS, they could not be viewed with current Office applications. The consequence is that there are 6,000 business emails that can only be opened in a legacy environment which the organisation no longer wishes to support.
  • An organisation has cabinets full of CCTV footage of sewer pipes on a variety of media, including reel-to-reel film, VHS video tapes, DVD and CD ROMS. It is estimated that this footage cost 2 million dollars to create. They still have a player for the reel-to-reel, but it is in poor condition and they are also concerned that if they attempt to check the conditions of these recordings the tapes will snap.

Digital preservation in New Zealand

Digital preservation is not an unknown problem locally; government has invested $40m in funding for this purpose in the National Library and Archives NZ. However, we don’t know the nature or extent of the risk to digital stuff on a national basis. Sustainability issues will only increase in volume and complexity as time passes.

But, given the economies of scale involved and the government’s current investment, New Zealand has a very rare opportunity to model digital preservation at a national level. Our Digital Preservation as a Service project (DPaaS) is designed to provide evidence of the demand and need for a national approach to digital preservation.

Digital preservation as a service (DPaaS)

Very few other institutions in New Zealand will have the financial or people resources to undertake digital preservation. A nation-wide approach will:

  • ensure the long term safekeeping of a greater range of New Zealand’s social, cultural, scientific and economic digital assets
  • leverage investment to date
  • reduce duplicate investment
  • support a strategic response to issues related to data use and re-use – there is currently real dissonance between the rhetoric of use and re-use and the need to address the long-term sustainability of digital assets required to ensure use and re-use

Digital preservation is becoming increasingly important for government, and as a national information policy issue.

Why’s it a good idea?

Services that provide enhanced access to digital information are now core to the social, cultural, and economic well-being of the nation. Their development is a fundamental component of a high functioning digital environment, at a national level.

Providing the foundations that support robust and well-integrated access to evidence of the record of government and wider society matters to both the general public, and to government.

By working at a national scale, we can provide the digital preservation capability and capacity that’s not currently available. This is critical to central government’s strategy and direction – such as being digital by default, sharing our investment and capability, and understanding and managing information as an asset.

Once we have that scale, we’ll also be able to provide better support to the wider community, with information management and preservation of assets. We’ll be able to do it more cheaply, more efficiently, and more productively.

What a high-functioning digital environment looks like

Diagram showing how a knowledge-rich society with widespread digital literacy and a culture of creation and innovation relies on a solid legislation and regulation, technical capacity for transmission and storage, and a sustainable and secure environment. It shows that some required aspects, including elements of legislation like the National Library Act and stable data centres are already in place, while copyright and privacy legislation still needs work, and other elements like a national research infrastructure have not been started.

Losing our stuff is a pain

Losing our valuable stuff is expensive.

Let’s not let it happen to us.

We all know there is probably a lot of digital stuff out there that is not being kept in a safe manner.

We need a sustainable safekeeping model for digital assets – is a national level digital preservation service the answer?

If you would like to tell us about your digital preservation needs, email Steve.Knight@dia.govt.nz.


Teaser image by Greig Roulston for HeritagePreserve. Derived from Eph-F-MEAT-Gear-035.

By Steve Knight

Steve is Programme Director Preservation Research and Consultancy at the National Library. He runs the digital preservation programme.

Post a Comment

(will not be published) * indicates required field

Be the first to comment.