Digital Scholarship Conversations: Exploring the Institutional Newspapers Dataset

Join researchers from the Institutional Data Initiative at Harvard Law School Library and librarians from Boston Public Library for an in-depth look at Institutional Newspapers, a structured dataset derived from public domain digitized newspapers: http://www.bpl.org/news/boston-public-library-and-harvard-idi-release-open-dataset-from-135-years-of-historic-newspapers/. The dataset, released in August 2026, contains images and text from over 1.4 million pages published between 1795 and 1930, including 83 million individual image crops (or segments) and 16.3 billion o200k_base tokens of VLM OCR text. This presentation will cover the origins of the collaboration between BPL and IDI; the inherent challenges of working with historical newspapers, design decisions made during the project's lifecycle, a detailed overview of the processing steps and output, and the implications of the release of the pipeline code and dataset for future research. This release represents a first step in an ongoing collaboration to scale the project, broaden and deepen our understanding of historical newspapers, and expand computational access to these materials.

Matteo Cargnelutti is Principal Engineer and Engineering Lead at the Institutional Data Initiative at the Harvard Law School Library. His work sits at the intersection of digital preservation, open knowledge, and the data that underpins AI. Eben English serves as the Digital Repository Services Manager at Boston Public Library, and technical lead for DigitalCommonwealth.org: https://www.digitalcommonwealth.org/, which provides access to digitized primary source materials from hundreds of Massachusetts libraries.