When Web Crawling Backfires?

Gregory Wiedeman

University Archivist

University at Albany, SUNY

@GregWiedeman

Web Archives at UAlbany

• Archive-It partner
• Crawling albany.edu since 2013
• NY State Politics since 2016
  • Extension of physical collecting
• Some Experimenting
• No known active use

Web Archives at UAlbany

• We have and want to build collections on NY State politics and public policy
• Rockefeller College of Public Affairs & Policy
• No real existing relationship
• Faculty member who researches right-wing extremist networks on the web
  • wget –recursive and TBs of raw html
  • Trouble with some sites

Appraising a Racist Website

• Purposely document conflicting views
  • Conservative Party of NYS
  • Ernest Van Den Haag
• Listening to at-risk groups
• Is conservatism underdocumented?
  • Maybe in quantity, but not in research value
  • Content-wise this is garbage
  • Underdocumented records are not on the web
• SPLC already adding to Wayback
• Can directly aid research

Crawling a Racist Website

• Recursively appending garbage to URLs?
• Millions of docs
• Bad wordpress plugins?
• obfuscation or incompetence?

"/mejs." 
"/ADSENSE/" 
"/groups/" 
"/friends/" 
"/favorites/" 
"/mentions/" 
"/notifications/" 
"/messages/" 
"/settings/" 
"/Captions/" 
"/js/index.php" 
"/audio/" 
"/video/" 
"/profile/" 
"/eventEmitter/" 
"/get-style-property/" 
"/doc-ready/" 
"/matches-selector/" 
"/fizzy-ui-utils/" 
"/outlayer/" 
"/isotope/" 
"/masonry/" 
"/layout-modes/" 
"?mode=list" 
"?mode=grid" 
"/wp/v2/" 
"/circle.background"

Things to think about

• Do they sell ads? Am I funding them?
  • Turn off add blocker
  • Is scopeing them effective?
• Why is it appending numbers as params?
  link/to/image.jpg?pas=20312538471902072133

Breaking things

• Crawls didn’t get user activity pages
• Tried to crawl just that section
• Suddenly required login for major parts of site
  • is that good? </br>
• “I might have made your research data impossible to get, but here’s a WARC you don’t know how to use.”

Web Archives and Oppositional Collecting

• I think there are underdocumented things
  • How racist or other ideas affect political power?
  • Are these records on the public web?
• Web Archives are extremely limited for this type of collecting
• Broader questions about collecting and power

When Web Crawling Backfires?

Gregory Wiedeman

University Archivist

University at Albany, SUNY

@GregWiedeman