Proposals at the IETF could limit web crawling and scraping, harming journalists, researchers, and the open internet. These tools help access public information and support vital online work. The free web is under pressure from new rules.
The free and open web relies on automated tools that let people access public information easily. Crawling and scraping power many useful services. They help find data, preserve history, and support analysis across the internet.
Journalists use these methods to report stories, while researchers spot security issues and study social problems like discrimination. Non-profit groups such as the Internet Archive depend on crawling to save web pages for future generations. Without it, much online knowledge could be lost.
Now, discussions at the IETF aim to change how these tools work. New limits might block access for good causes. This shift threatens the core benefits of an open internet and could reduce transparency for everyone.
Original Author: Tori Noble | Source: EFF
Comments