When Web Crawling Backfires?


Gregory Wiedeman
University Archivist
University at Albany, SUNY
Twitter logo @GregWiedeman

Web Archives at UAlbany


Web Archives at UAlbany


Appraising a Racist Website


Crawling a Racist Website

Crawl hosts report with 1,849,670 out of scope documents.



"/mejs." 
"/ADSENSE/" 
"/groups/" 
"/friends/" 
"/favorites/" 
"/mentions/" 
"/notifications/" 
"/messages/" 
"/settings/" 
"/Captions/" 
"/js/index.php" 
"/audio/" 
"/video/" 
"/profile/" 
"/eventEmitter/" 
"/get-style-property/" 
"/doc-ready/" 
"/matches-selector/" 
"/fizzy-ui-utils/" 
"/outlayer/" 
"/isotope/" 
"/masonry/" 
"/layout-modes/" 
"?mode=list" 
"?mode=grid" 
"/wp/v2/" 
"/circle.background"

Things to think about


Breaking things


Web Archives and Oppositional Collecting


When Web Crawling Backfires?


Gregory Wiedeman
University Archivist
University at Albany, SUNY
Twitter logo @GregWiedeman