Breaking the open web won't stop the bad bots
Web page · https://techpolicy.press/breaking-the-open-web-wont-stop-the-bad-bots
First seen by Oriel on .
AI summary
The article argues that proposed federal and New York laws targeting anonymous web crawlers would chill legitimate journalism, research, archiving, and security work without addressing the harms of abusive scraping.
- The Stealth Bot Prohibition Act could impose FTC penalties of up to $53,000 for conduct deemed reasonably likely to burden a website, but the bill does not define that standard.
- The New York bill would let covered news publishers subpoena crawler identities without showing harm, even though disclosure does not limit request volume or compensate publishers.
- Examples of beneficial scraping include the Wall Street Journal’s analysis of 31,000 financial disclosures, the Wayback Machine’s archive of more than one trillion webpages, and accessibility audits of over one million websites.
- The authors warn that compelled identification and vague liability could deter anonymous public-interest research and burden the First Amendment right to receive information.
- They recommend interoperable technical standards for managing machine access and, if legislation is adopted, liability based on identifiable harms such as server overload, fraud, or prohibited data collection.
#web scraping #ai policy #open web #internet archive
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Points
Score reported by the source.
- Hacker News: 323, latest observation ; +316 over observed period since
Comments
Comment count reported by the source.
- Hacker News: 147, latest observation ; +147 over observed period since
Aliases
| Source | External ID | URL |
|---|---|---|
| Hacker News | 50263837 | https://www.techpolicy.press/breaking-the-open-web-wont-stop-the-bad-bots/ Read discussion |