Crawler policy
ClawHarborBot
What this crawler does, how often it does it, and how to stop it.
User agent
ClawHarborBot/1.0 (+https://clawharbor.io/bot)What it fetches
ClawHarbor reads a public website twice, for two different reasons.
- Once, at onboarding. A crawl records the site's shape: colour tokens, type scale, page structure, and the CSS selectors that identify a listing. The output is a manifest describing that shape. It contains no page content — no listings, no descriptions, no images, no font files.
- Live, per request. When someone asks an assistant a question, the page that answers it is fetched at that moment, cached briefly, and discarded. Nothing is accumulated into a database.
Politeness
robots.txtis fetched and honoured before anything else, and re-checked on every crawl.- Requests to one origin are serialised behind a per-origin delay — never parallel, never bursty. The ceiling is one request per second.
- Responses are cached, so repeated questions cost the origin nothing.
- If an origin fails, we serve the cached answer labelled stale rather than retrying in a loop.
- Nothing behind a login is ever fetched, and no generated tool writes, books, pays or authenticates.
How to block it
Add this to your robots.txt and the crawler stops on its next check:
Blocking by user agent or IP at your edge also works and needs no notice to us. If you would rather an interface be removed outright, use the removal form — that is faster and it is honoured without argument.
Per-host detail
The crawler runs against one host at a time and each published interface states the origin it reads and the rate ceiling it observes. The live list is served by the application host at mcp.clawharbor.io/bot, which reads it from the tenant registry rather than from this page.