E-ARK Tools - E-Ark Project

WP1 - Project Co-Ordination

2015-11-26T12:34:34+00:00

Work Package Lead

University of Brighton (UoB)

What the Work Package did

E-ARK comprised a complex series of inter-dependent activities, which delivered a large number of outputs over a period of 3 years. These were managed to ensure that each output was delivered on time and to budget, as well as in a way that interoperated with the other parts of the project.

The Project provided regular (annual) progress reports and financial statements to the European Commission to explain and justify the co-funding with which they are helping to support the project to achieve its objectives.

We managed E-ARK in accordance with the suite of methodologies known as the Swirl™ Suite originally created by the UK Office of Government Commerce and now managed by Axelos:

PRINCE2® - Project Management
MSP® - Programme Management
M_o_R® - Risk Management and
MoV® - Management of Value

We managed the project’s relationship with the European Commission, liaising with their Project Officer(s) and Finance Department as required. We were responsible for the submission of the project’s work to the European Commission.

We had overall responsibility for the financial management of the project. We arranged for the submission of financial statements and costs claims to the EC, including ensuring that audits are carried out where necessary. We dealt with any questions which the EC had regarding our returns and then arranged for the distribution of grants when these are received.

At the end of each year we produced an Annual Report on the project, a summary of which is published on this website.

We also organised and lead the Annual Scientific Review conducted into the project's work by the EC.

In addition to having responsibility for the structures by which the project is governed, we also provide support for a number of Advisory Boards representing the Archive, Technical and Commercial Supplier Communities with whom we are co-operating in our work.

Highlights

Agreement of overall Description of Work with the EC
Successful completion of Year 1, 2 and 3 Reviews
Preparation of Year 2, Year 3 and end-project reports

WP5 Archival Records Access Services

2015-10-19T09:51:33+00:00

Work Package Lead

Danish National Archives

What the Work Package did

This work package defined an E-ARK Dissemination Information Package (E-ARK-DIP) format, and provided access methods and interfaces for structured and unstructured archival records. Referring to the OAIS reference model the consumer gets access to the information in the archive by using an access and presentation system. The consumer requests information from the archive by using the access system, which requests the database of the archive to retrieve the matching archival information packages (AIPs). These AIPs are disseminated by the system into Dissemination Information Package DIPs. These DIPs are used by the presentation system to present the contained information to the consumer.

Highlights

GAP report
E-ARK DIP draft specification
Access Component overview diagram
Access use cases and tool requirement
Search facilities available
Prototypes of order management tool, AIP-DIP converter; DIP presentation tools

Public Deliverables

D5.3) E-ARK DIP pilot specification (revision of D5.2)
D5.4) Search, Access and Display Interfaces

Conference Papers/Publications

Thirifays A., Hougaard K., What’s E-ARK: The DIP format (abstract and presentation), ICA Annual Conference, Reykjavik, Iceland, 28 September, 2015.
Thirifays A., Hougaard K., Towards a common approach for access to digital archival records in Europe (article and presentation), iPRES2015, Chapel Hill, North Carolina, USA, on November 2-6 2015.

WP4 Archival Records Preservation

2015-10-19T09:51:33+00:00

Work Package Lead

Austrian Institute of Technology (AIT)

What the Work Package did

This work package specified an archival information package format (AIP) and implemented the conversion of a submission information package (SIP) into an AIP that complied with the format specification.

The AIP format specified a directory hierarchy providing a clear separation between the actual content to be preserved, the metadata describing the content, and the preservation metadata recording digital provenance information.

The result is a highly flexible and extensible structure together with a best practice approach of using metadata for structuring and preserving archival information packages over the long-term. Special attention was paid to the requirement of being able to handle very large data sets which have to be split into several parts because it is not possible to archive them as a single coherent unit.

The implementation of the SIP to AIP conversion represents a prototype implementation of the specified AIP format and is based on architectural principles allowing for scalable and parallelised information package transformation. In this sense, the work package develops a SIP to AIP conversion software based on modular conversion components in order to enable information package transformation most appropriate for each information package type (e.g. ERMS, database, GIS, etc.).

Highlights

AIP format specification published
Vocabulary Manager deployed and online
MoReq2010 Validator deployed and online
Service for validating a MoReq SIP
Ability to map MoReq exports with E-ARK SIPs and consequently with E-ARK AIPs
SIP to AIP conversion as part of the integrated prototype
Recommendations for OLAP related data transformations during SIP to AIP conversion

Public Deliverables

D4.1) Report on available formats and restrictions
D4.2) E-ARK AIP draft specification
D4.3) E-ARK AIP pilot specification
D4.4) Final version of SIP-AIP conversion component

Conference Papers/Publications

WP4 Presentation, DLM-Forum Members Meeting, Luxembourg, Luxembourg, 14-15 October, 2015.

WP3 - Transfer of Records to Archives

2015-10-19T09:51:33+00:00

Work Package Lead

National Archives of Estonia (NAE)

What the Work Package did

The overall objective of this work package was to ensure that records and their metadata could be exported from source systems, prepared for transfer and ultimately transferred and ingested into archival repositories in a way which is efficient, reliable and applicable across all European countries.

The first stage was to gather information on current practice, methodologies and tools. This information was then be used to develop best practice and standardisation through the creation of:

A central methodology and model requirements for the export of records and their metadata, building largely on the acknowledged MoReq requirements;
A pan-European SIP format specification which provides sufficient standardisation for automated solutions, working with scenarios for archiving both individual records and full records systems, the initial version of which will be circulated among stakeholders for feedback before finalisation; and
SIP creator tools that are compatible with the new specification based on existing tools from ES Solutions and the Danish National Archives. Stakeholder feedback will also be important to the development of these tools.

Highlights to Date

SIP specification defined
SIP creation tools

Public Deliverables

D3.1 – Report on Available Best Practices
D3.2 – E-ARK SIP Draft Specification
D3.3 – E-ARK SIP Pilot Specification
D3.4 – Records Export, Transfer and Ingest Recommendations and SIP Creation Tools

Conference Papers/Publications

WP2 - Use Cases and Pilots

2015-10-19T09:51:33+00:00

Work Package Lead

National Archives of Hungary (NAH)

What the Work Package did

The overall objective of this work package was to ensure that the scenarios implemented at the pilot sites are both realistic and relevant. That is, that they brought together a meaningful subset at each site of the use cases that could provide the foundation for the general model of the E-ARK service. Use cases were defined for workflows across the record lifecycle and presented functional and non-functional requirements. The accompanying pilot scenarios defined the business and operational contexts against which the pilot will be evaluated. Both were defined in UML and plain language to allow ease of communication both within the project and with relevant stakeholders.

The following tasks were undertaken:

Definition of the general model and use cases with reference to the OAIS model. This included the common framework for the scenarios that was taken into account during the project as well as clearly defining the complex scenarios involved and ensuring that they are defined at the same conceptual level.
A comprehensive survey of the legal and organisational framework under which European records management, preservation and access takes place. This included key issues such as requirements for data protection, open government, seamless access, management of hybrid data and collaboration with agencies, with the hope of encouraging greater harmonisation in both understanding and approach.
Support for tool developers by providing necessary information including technical documentation and definitions of workflows.
Definition of the pilot scenarios including tools to be deployed and the level of activity required to bridge the gaps of currently existing solutions. In addition to the 7 full-scale pilots there were also shorter ‘stretch’ pilots and a ‘beta’ program to allow external validation.
Support and execution of the pilots was undertaken according to a methodological framework that specified the input/output points and the uniform principles to be applied across different areas to ensure comparable results.
Production of the final report on the pilot activities, listing resources used and providing an evaluation of the progress and final results against the project objectives and milestones.

Highlights

General model for E-ARK services and pilots
Report on Legal Issues
E-ARK Pilots
Final recommendations for best practice

Public Deliverables

D2.1 - General Pilot Model and Use Case Definition
D2.2 – Legal Issues Report for Pilot
D2.4 – Pilot Documentation
D2.5 – Recommended Practices and Final Public Report on Pilots

Conference Papers/Publications

WP6 - Archival Storage, Services and Integration

2015-10-19T09:51:33+00:00

Work Package Lead

Austrian Institute of Technology (AIT)

What the Work Package did

Highlights

HDFS storage service
MapReduce ingest workflow
Faceted search interface
Access interface
Web User Interface
Cluster installation at NAH
Data mining showcase
Benchmark results

Public Deliverables

D6.1) Faceted Query Interface and API
D6.2) Integrated Platform Reference Implementation
D6.3) Data Mining Showcase

Conference Papers/Publications

Delve J., Aas, K, Schmidt R., Database Archiving in the E-ARK Project (presentation), Preservation and Archiving Special Interest Group Meeting, Karlsruhe, Germany, 17 September, 2014.
Aas, K, Delve J., Schmidt R., Long-term access to databases the meaningful way, DLM Forum Triennial Conference 2014, Lisbon, Portugal, November 10-14, 2014.
Schmidt R., Data Valorisation and the Application of Big Data Techniques in Archives (presentation), E-ARK Advisory Board Meeting, Lisbon, Portugal, November 10-14, 2014.

WP7 - Evaluation and Assessment

2015-10-19T09:51:33+00:00

Work Package Lead

Instituto Superior Técnico (IST)

What the Work Package did

This work package developed and promoted tools, methods and practices to support the consolidation of Information Governance knowledge and references. Internally to the project the work package is responsible for evaluating and assessing the solutions being developed in E-ARK.

To achieve its objectives, work package 7 developed a maturity model for information governance and used the same model to assess the initial and final levels of the pilots. Additionally, the work package implemented a knowledge centre consolidating existing best practices, standards and other references for information governance.

Highlights

Vocabulary Manager deployed and made available online
MoReq2010 Validator deployed and made available online
Initial assessment of pilots published
Knowledge Centre prototype deployed and made available online
Maturity Model published
Final assessment of pilots published
Knowledge Centre published

Public Deliverables

D7.1) A Maturity Model for Information Governance – initial version
D7.2) Initial Assessment and Evaluation
D7.3) Prototype of the Knowledge Centre Service
D7.4) The Knowledge Centre Service
D7.5) A Maturity Model for Information Governance – final version
D7.6) Final Assessment and Evaluation

Conference Papers/Publications

Aas, K., Delve, J., Vieira, R., King, R., Integrating e-government systems with digital archives, 11th International Conference on Digital Preservation, Melbourne, Australia, October 6-10, 2014.
Proença, D., Vieira, R., Borbinha, J., A Maturity Model for Information Governance, DLM Forum Triennial Conference 2014, Lisbon, Portugal, November 10-14, 2014.
Vieira, R., Ragageles, L., Borbinha, J., One consolidated view of information management references, DLM Forum Triennial Conference 2014, Lisbon, Portugal, November 10-14, 2014.

About E-ARK

2014-05-30T12:27:41+00:00

Archives provide an indispensable component of the digital ecosystem by safeguarding information and enabling access to it. Harmonisation of currently fragmented archival approaches is required to provide the economies of scale necessary for general adoption of end-to-end solutions. There is a critical need for an overarching methodology addressing business and operational issues, and technical solutions for ingest, preservation and re-use.

In co-operation with commercial systems providers, E-ARK created and piloted a pan-European methodology for electronic document archiving, synthesising existing national and international best practices, that will keep records and databases authentic and usable over time.

The methodology was implemented in an open pilot in various national contexts, using existing, near-to-market tools, and services developed by the partners. This will allow memory institutions and their clients (public- and private-sector) to assess, in an operational context, the suitability of those state-of-the-art technologies.

Our objective has been to provide a single, scalable, robust approach capable of meeting the needs of diverse organisations, public and private, large and small, and able to support complex data types. E-ARK has demonstrated the potential benefits for public administrations, public agencies, public services, citizens and business by providing simple, efficient access to the workflows for the three main activities of an archive - acquiring, preserving and enabling re-use of information.

The practices developed within the project will reduce the risk of information loss due to unsuitable approaches to keeping and archiving of records. The project has been public facing, providing a fully operational archival service, and access to information for its users. The project results were generic and scalable in order to build an archival infrastructure across the EU and in environments where different legal systems and records management traditions apply. E-ARK has provided new types of access for business users.

E-ARK piloted an end-to-end OAIS-compliant e-archival service covering ingest, vendor-neutral archiving, and reuse of structured and unstructured data, thus covering both databases and records, addressing the needs of data subjects, owners and users. The pilot and methodology also focused on the essential pre-ingest phase of data export and normalisation in source systems. The pilot integrated tools currently in use in partner organisations, and provided a framework for providers of these and similar tools ensuring compatibility and interoperability. A core component of the project was the integration platform which uses the existing ESSArch Preservation Platform (EPP) application as an Archival Information System, which was already in productive deployment at the National Archives of Norway and Sweden. In order to achieve scalability, E-ARK adopted a data management and storage layer for this tool on top of the proven open-source Cloudera CDH4 distribution of Apache Hadoop, enabling storage and computational power to be seamlessly added to the system.

The pilot ran in several national archives, each of which provided data to run in the pilot instance by agreement from an associated government data owner (e.g. national or regional / federal).

The project outputs will be sustained over the long term by project partner The DLM Forum, comprising 22 national archives and associated commercial and technical providers. Using the open Apache licensing model, commercial suppliers will be able to incorporate the project outputs (particularly the open interfaces for pre-ingest, ingest, archival, access and re-use) into their own systems, enhancing their longevity. National archives running E-ARK pilot instances will serve as exemplars for others wanting to adopt up the new e-archiving open system.

In addition, project partner, The Digital Preservation Coalition will promote best practices in this area, as will our dedicated government institution partners.

E-ARK Closing Conference

2014-05-30T12:27:41+00:00

The E-ARK Project final conference and workshops took place at the National Archives of Hungary from 6^th-8^th December 2016.

About the Conference

The E-ARK Conference brought together inspiring keynote speakers with presentations on the innovative standards and tools developed by the project. Over the three days attendees gained a detailed understanding of the practical solutions offered by E-ARK outcomes. On the third day, there was an opportunity to gain direct hands-on experience of the E-ARK tools and solutions.

Day One took a strategic overview of the issues faced within the e-government arena, including presentations on the E-ARK pilots and maturity model, as well as a panel session on sustainability featuring a number of industry experts.

Day Two focused on the project’s technical outcomes including preservation of databases, scientific data and other “big data”. Finally, Day Three offered a smaller group of attendees the chance to gain a hands-on introduction to deploying E-ARK solutions in a series of workshops.

Speakers included:

Alina Senn - Project Officer at the European Commission
Dr Angela Dappert - Project Manager, British Library
Krystyna Ohnesorge - Swiss Federal Archives

There will also be presentations by the E-ARK teams responsible for designing and implementing the tools and services in E-ARK.