Oriel — AI & technology worth noticing, every day.

Web Page Crawler Revealed - tracking down requests from Contabo GmbH

Web page · https://bruceediger.com/posts/contabo-crawler

The story is a technical breakdown attempting to trace a series of crawlers from Contabo GmbH. Edited the title to be more clear. This feels relevant here but I struggled with tagging this. I welcome suggestions for how to tag it better!

First seen by Oriel on .

AI summary

Apache logs from a generated binary-tree site reveal a coordinated crawler operating from Contabo IP ranges, repeatedly traversing every linked page at a nearly constant rate.

  • The PHP site generated 1,048,579 pages with parent and child links; each major crawl series requested almost every page once by following hyperlinks.
  • Across six traversals, the crawler averaged about four requests per second, with 7,690 IP addresses involved across four series and 2,964 appearing in all four.
  • Request sequences show concurrent traversal by binary-tree rank, while the absence of unrequested-parent cases supports link-following rather than URL enumeration.
  • A Go simulation of a conventional FIFO queue with concurrent workers did not reproduce the observed ordering; a stack of queues better approximated concurrent rank traversal but remained less consistent.
  • The author infers centralized coordination and possibly distributed queues, but the crawler's internal scheduling mechanism remains undetermined.

#web crawling #apache logs #go #distributed systems

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Points

Score reported by the source.

  • Lobsters: 8, latest observation ; +6 over observed period since

Comments

Comment count reported by the source.

  • Lobsters: 0, latest observation ; 0 over observed period since

Aliases

SourceExternal IDURL
Lobsterseyrbuphttps://bruceediger.com/posts/contabo-crawler/ Read discussion