Web Curator Tool (WCT)
One of our innovative tools is Web Curator Tool. The Web Curator Tool is used for harvesting websites, web pages, and other documents on the internet.
What is the Web Curator Tool?
The Web Curator Tool is designed for use in libraries and other collecting organisations. It supports collection by non-technical users while still allowing complete control of the web harvesting process. The tool supports:
- Harvest authorisation — obtaining permission to harvest web material and make it accessible.
- Selection, scoping and scheduling — deciding what to harvest, how, and when.
- Basic description — adding unqualified Dublin Core metadata and web-specific notes.
- Harvesting — downloading the selected material from the internet.
- Quality review — ensuring the harvested material is ready to archive.
- Archiving — submitting harvest results to a digital archive.
Selective web harvesting programme
The National Library of New Zealand runs a selective web harvesting programme using the Web Curator Tool. Web harvesting is the term we use to describe selecting, copying and archiving of websites found on the internet. Websites harvested by this method are deposited into the Library’s National Digital Heritage Archive (NDHA) using Rosetta.
Diagram of the workflow of a web harvest into a preservation system.
National Library web harvesting
Development of the Web Curator Tool
The Web Curator Tool was developed in 2006 as a collaborative effort by the National Library of New Zealand and the British Library. The project was initiated by the International Internet Preservation Consortium (IIPC).
In December 2018, the Web Curator Tool 2.0 was released. This release is the product of a collaborative development effort started in late 2017 between the National Library of New Zealand (NLNZ) and the National Library of the Netherlands (KB-NL).
British Library
International Internet Preservation Consortium (IIPC)
National Library of the Netherlands
Web Curator Tool is open-source
The Web Curator Tool is written in Java and designed to run in Apache Tomcat. It has a flexible architecture, allowing the components of the tool to be distributed over multiple servers.
Web Curator Tool is available under the terms of the Apache Public License. The Web Curator Tool was released as open-source software and can be downloaded from GitHub.
Before you use the Web Curator Tool we recommend you read the Web Curator Tool Documentation. Documentation relating to the older version of the Web Curator Tool are available on GITHub.
Web Curator Tool documentation — current Web Curator Tool documentation.
Older versions of the Web Curator Tool documents on GITHub — the documents on GITHub relate to previous versions of the Web Curator Tool.
Web Curator tool 2.0 handout (pdf, 110KB) — find out what is new in the Web Curator Tool 2.0.
Back to top