In Denmark, all public institutions are required by law to deliver certain data and information to the Danish National Archive for posterity. The role of the archivist is to determine what needs to be kept in the national interest and to select the information that needs to be stored and retained for future use.

The Danish National Archives is responsible for setting the technical standards for how public institutions must deliver this data, and these institutions must comply with these standards.

These standards are necessary in order to ensure that the information and data will be sustainable and accessible over the long term. However, this means that institutions must export and convert the data from their various systems to one specific preservation format, and this can be a complex and costly process.

Currently, the Danish National Archives can only offer guidance to the institutions about the format in which the data and information is to be exported, and yet can only accept the incoming data and information if it all complies with the technical standards set. The E-ARK tool tested in this pilot can attach to the most common systems used by the institutions and can export to the preservation format developed in E-ARK. As the E-ARK tool is very similar to the preservation format already used in the Danish National Archives, the tool allows institutions to export to the Danish preservation format. This means that, by making the E-ARK tool tested in this pilot universally available, it will become cheaper and easier for historic data to be delivered to the Danish National Archives.

The final product will also be accessible to other countries’ archives meaning that small nations can benefit from the tools arising from the collaboration and the technological expertise developed as part of the E-ARK project.

This pilot focused on the initial stages data and information transfer from the producer to the archive. Before the data can be moved, it has to be organised and packaged in a way that meets the standards of the archive where it will be processed further. The pilot consisted of three scenarios with various types of content of different size and form.

(1)  Database content

(2)  Content related to the Noark-4 standard

(3)  Content related to the Noark-5 standard

The format of the content was of interest to the E-ARK project inasmuch as it had to be validated according to local standards. Three types of software developed by one of the other partners (ES Solutions AB) in the E-ARK project were used:

(1)  ESSArch Tools Producer (ETP). This is aimed at data producers.

(2)  ESSArch Tools Archive (ETA). This services the front end of the repository/depot.

(3)  ESSArch Preservation Platform (EPP). The repository/depot solution.

The tools all use international metadata standards such as OAIS, METS and PREMIS for the description of structure and actions on the content. During the pilot, the overall success of applying the tool support was evaluated. At a more detailed level, how size considerations versus time consumption, and details regarding the OAIS, METS and PREMIS implementation etc. were also assessed.

The process of moving information from government agency record management systems to national archives where it can be sustained and made accessible can broadly be divided into four phases:

(1) Selection     (2) Extraction     (3) Transfer     (4) Acceptance

Usually, this is information which needs to be retained for an indefinite period of time and is relevant to both the agency that created it and to the nation as a whole. Each country must follow similar pathways yet naturally, there is much variation, which had led to a lack of a common approach or methodology.

The movement of material of archival value and the related metadata was tested in this pilot using a series of tools which implement the E-ARK information packaging specifications. It can be hard to persuade institutions to develop their own processes for doing this. By involving multiple partners and stakeholders in E-ARK, a universally available standard process has been created that means that individual agencies and archives will no longer need to find bespoke, customised solutions.

Often, the cost of migrating information to a newer system can prove complex and prohibitively expensive. When commercial or private sector data is stored or left in databases that are then untouched for significant periods of time or become out of date, retrieving and reusing information can be difficult.

Companies need solutions to extract information that is still valuable though not in active use, but which needs to be kept for a variety of reasons – legal, compliance, business – and then reused. In this pilot the National Archives of Estonia worked alongside the Estonian Business Archives. As a service company to the private sector, the latter has extensive experience of and are familiar with the needs and requirements of the private sector.

This pilot tested the E-ARK Database Preservation Toolkit (DTP) at the SIP (submission information package) creation stage, allowing information to be transferred from old platforms to an interim standardised format which is then digestible by newer systems. The SIARD (Software Independent Archiving Relational Databases http://www.digitalpreservation.gov/formats/fdd/fdd000426.shtml) tool was piloted to understand how to keep the data alive and sustainable for future use. It is hoped that SIARD will become the de facto industry standard in due course.

This Pilot developed and tested guidelines and tools for archiving digital geographical maps which have in the past been used as a basis for decision making in government and other official procedures (e.g. granting planning permission and building permits, assessing where to levy taxes and in monitoring pollution).

Focusing on digital geographical data in vector format, the pilot produced guidelines that can be used for archiving the digital geodata and the required documentation, and which will ensure a proper understanding of and reuse of these records in the future. It also demonstrated that the integration of existing open source tools and the tools developed in the E-ARK Project can support the archiving process and access to such data.

Users benefited by gaining access to a wider range of geodata and by, using location-based search tools, by having the opportunity to use and compare additional descriptions of the required documents for the interpretation of and, significantly, the re-use of the geodata.

The life cycle of a record that needs to be archived usually falls into three phases:

(1) When the metadata and data are ‘live’ (being used and modified regularly).

(2) When the metadata and data are archived for a short period (up to 5 years).

(3) When the metadata and data are moved to a long-term repository for permanent conservation.

The aim of this pilot was to assess the efficacy of the E-ARK Information Package Specification. This defines how metadata and data should be packaged in order to move records between these phases. The pilot worked on a way to ensure that no relevant data or metadata was lost in the process. To accomplish this, a special integration tool was developed that implements the specification and orchestrates the entire transfer process.

The information package used proved effective in terms of its flexibility and ability to cope with multiple data types. The pilot transferred about 2 TB of data between the archival service and the long-term preservation service. The institution’s archival service is not yet prepared to face the challenges of long-term digital preservation, so the records that were selected for permanent conservation were then transferred to a long-term digital repository (the third phase). The pilot worked with a public institution whose records were initially produced and managed in an electronic records management system (ERMS). These were mandated to be archived after their “live” phase and the first part of the pilot focussed on the assessment of the adequacy of the Information Package structure and the efficiency of the integration tool in handling the data and metadata (and supporting documentation) as it moved from the original system to the institution’s own internal archival service (the second phase).

The pilot also examined the earlier stages of the transfer of data and information and tested the E-ARK tools in that phase.

When information is initially stored in databases used in offices and institutions, it is categorised according to the nature of the information. For example, individual tables are made of generic data such as names, addresses and so forth and are linked to other tables of relevant information. Multiple sets of data in these operational databases, where the original material is of interest to the originator and the archivist, are stored in a normalised form in order to maintain consistency.

When this data is moved to longer-term storage and access, the structure for storing the data is simplified into fewer datasets, grouping together sets extracted from the original database. This simpler structure is intended to be beneficial by making the data clearer and easier to interrogate, and for access to be more straightforward.

These are the data warehouses and they exist to ensure long-term sustainability for the data and to allow for interrogation by researchers requiring access to the data. This pilot tested the E-ARK ESSArch tool. This is designed to ensure that all the data is retained and that there is no loss of function as a result of the transition from the original relational database to the data warehouse.

Furthermore, this pilot also tested the efficacy of freely available web-based applications, such as Oracle APEX, in allowing researchers to get access to and use the data stored in the data warehouses. In some cases, companies and institutions such as banks, tax offices and public institutions only have their records in data warehouses, rather than in operational databases, so it is important that they are easy to understand and meet the requirements of those needing access.

In ‘rendering’ information, data needs to be provided in relevant ways and according to the specific requests of the researcher. Various tools are available for this with varying degrees of sophistication and at varying costs.