Data Analytics Approaches for Web Archives

Gregory Wiedeman
University Archivist
M.E. Grenander Department of Special Collections & Archives

Collecting the Web at UAlbany

• Preserve and manage permanent public records
• Crawling and preserving albany.edu since 2012
• Began outside collecting in 2016
• 1.5 TB of Web Data
• Collect, preserve, provide access, and encourage use

Twitter Data

• Detecting Russian bots: it`s
• Ed Summers apostrophe dataset 40 accounts
• Collected 14,217 tweets with Twarc

Jonathan Albright, “The #Election2016 Micro-Propaganda Machine”

Working with WARCs

• .warc file ISO standard
• Tremendous volume of data
• Mess of HTML, CSS, JavaScript
• Unclear provenance
• Standard derivative datasets
• Tools becoming easier to use

Funding availability

• Mellon Foundation Scholarly Communications

• NEH Digital Humanities Advancement Grants (Due in January 2018)

• NEH Institutes for Advanced Topics in the Digital Humanities (for training programs)

• IMLS National Digital Platform
• National Archives NHPRC supports access and use of records

Collecting the Web at UAlbany

• We need partners!
• Smaller research-focused collections
• Work with Internet Archive to get older data
• How can we improve?

Links

• Github repo
• UAlbany Web Archiving Program
• Doc Now (http://www.docnow.io/)
• Archives Unleashed Toolkit, Warclight
• Jonathan Albright, “The #Election2016 Micro-Propaganda Machine”
• An Open-Source Strategy for Documenting Events: The Case Study of the 42nd Canadian Federal Election on Twitter, in Code4Lib
• Live Visualizations
  • NY19 Congressional District
  • Links top 10 domains
  • Links all domains