domainrating.org

DomainRatingBot

The fetcher behind site-owner verification and the “About this domain” facts on domainrating.org. It follows no links and indexes no content — but it does make a small number of automated fetches, so this page spells out exactly which, how often, and how to opt out.

User agent

DomainRatingBot/1.0 (+https://domainrating.org/bot)

What it fetches

  • Verification checks (user-initiated). When a site owner clicks “Check now” in their dashboard, we fetch https://<domain>/ or /.well-known/oar-verify.txt once to look for their verification proof. Verified proofs are re-checked about every 30 days, one page per domain.
  • Domain facts (background). When someone views a domain’s public page here, a background job may fetch that domain’s /robots.txt and homepage once to record the page title, declared language, and redirect behavior — at most once per domain per 30 days. The same job looks up DNS records (via public DNS-over-HTTPS resolvers), registration data (via the registry’s public RDAP service), and performs one TLS handshake to read the certificate your server presents to every visitor. Those three don’t request any page from your site.

How it behaves

  • It requests at most one page (plus up to 3 redirects) per check, reads at most 512 KB, and times out after 10 seconds.
  • Background fact checks are globally rate-limited (at most 100 per hour across ALL domains) and budget-capped, and each domain is checked at most once per 30 days.
  • It follows no links and indexes nothing — it is not a search crawler.
  • Verification checks are additionally rate-limited per account and per domain.

Opting out

Disallow DomainRatingBot in your robots.txt to stop the background homepage fetch:

User-agent: DomainRatingBot
Disallow: /

We check robots.txt before every background homepage fetch and record the opt-out (your domain’s page then says the homepage wasn’t checked). To be honest about scope: robots.txt governs fetching pages, so the DNS lookups, RDAP registry query, and TLS handshake — none of which request a page from your server — still run. Owner-initiated verification checks also still work, since the site owner explicitly asks for those.

Questions

If you see unexpected traffic from this user agent, contact [email protected].