Crawler policy

Identification

Our crawler identifies itself in the User-Agent string and fetches only publicly reachable pages over HTTP and HTTPS.

Politeness

We read robots.txt before crawling and obey its directives, including crawl-delay. We limit concurrency per host, cap pages and depth per visit, back off on errors and never attempt to bypass authentication, paywalls or rate limits.

We never submit forms, never log in, and never fetch internal or private network addresses.

Blocking us

Add a disallow rule for our user agent in robots.txt and we will stop crawling within 24 hours. You can also write to us to have an existing report removed from public listings.