Web Page Crawler Revealed - tracking down requests from Contabo GmbH
Web page · https://bruceediger.com/posts/contabo-crawler
The story is a technical breakdown attempting to trace a series of crawlers from Contabo GmbH. Edited the title to be more clear. This feels relevant here but I struggled with tagging this. I welcome suggestions for how to tag it better!
First seen by Oriel on .
AI summary
Apache logs from a generated binary-tree site reveal a coordinated crawler operating from Contabo IP ranges, repeatedly traversing every linked page at a nearly constant rate.
- The PHP site generated 1,048,579 pages with parent and child links; each major crawl series requested almost every page once by following hyperlinks.
- Across six traversals, the crawler averaged about four requests per second, with 7,690 IP addresses involved across four series and 2,964 appearing in all four.
- Request sequences show concurrent traversal by binary-tree rank, while the absence of unrequested-parent cases supports link-following rather than URL enumeration.
- A Go simulation of a conventional FIFO queue with concurrent workers did not reproduce the observed ordering; a stack of queues better approximated concurrent rank traversal but remained less consistent.
- The author infers centralized coordination and possibly distributed queues, but the crawler's internal scheduling mechanism remains undetermined.
#web crawling #apache logs #go #distributed systems
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Points
Score reported by the source.
- Lobsters: 8, latest observation ; +6 over observed period since
Comments
Comment count reported by the source.
- Lobsters: 0, latest observation ; 0 over observed period since
Aliases
| Source | External ID | URL |
|---|---|---|
| Lobsters | eyrbup | https://bruceediger.com/posts/contabo-crawler/ Read discussion |