The PlaceDatum crawler

We read the public pages of businesses to build their listings on PlaceDatum.

How to recognise it

We send this user agent on every request:

PlaceDatumBot/0.1 (+https://placedatum.com/crawler)

How to verify it

Anyone can copy a user agent, so check the signature. We sign every request with an HTTP message signature, under Web Bot Auth, and send a Signature-Agent header of https://placedatum.com. We publish our public keys there as a JWKS, at https://placedatum.com/.well-known/http-message-signatures-directory, and at the same path on every country host. To check a request, read its keyid and verify the signature with that key.

We publish a new key before retiring an old one, so an old signature still verifies. [pending: the founder names the verified bot program the crawler is listed with, once the listing is accepted at the first deploy]

What we read

On first contact we send one request at a time, in this order:

1. A DNS lookup. If the host doesn't resolve, we stop.
2. /robots.txt, read for the group named PlaceDatumBot.
3. Other rules about access: /llms.txt, /ai.txt, and the content-signal and X-Robots-Tag headers.
4. The home page, read only to find the link to the terms of use.
5. The terms of use, from that link or the usual paths.

If any of that restricts us, or a term forbids us to read the site, we stop. First contact takes at most 8 requests, and we take nothing from a page before we've read the terms.

If they all permit it, we read the home page again and the pages it links to that a visitor would open, including bookings, services, contact and hours. If a page needs its scripts to run, we read it once more through a managed browser on our platform, and that platform sends its own identity headers.

On its own interval we also read the sitemap, /.well-known/ucp and AGENTS.md, only to see what a site publishes. We read the robots file, those rules and the terms again every 90 days on a business's own site, and every 30 days on a booking or listing service.

The pace

We send one request at a time to a host, then wait 3 seconds before the next. If a site blocks us, we wait 30 days, then come back with the same user agent and headers.

If robots.txt disallows us, we stop.

What we never do

• Sign in, hold a password, key or credential, solve a CAPTCHA, or work around a paywall, login or block.
• Come back after a block or disallow under another user agent, address or headers.
• Fill in a form, send an enquiry, make a booking or pay. We check booking links, and stop before any step that commits.
• Keep a copy of a page. We store the facts, a request log, a content hash and an outline of its links and structured data.
• Search for information about a person, test the security of a site, or read to train a model.

How to stop it

Add this to robots.txt:

User-agent: PlaceDatumBot
Disallow: /

We stop the next time we read that file.

To stop us sooner, or if the pages are on a host you don't control, use the contact page and pick the request kind that stops us reading. If you show us proof that you own the business, we answer within 7 days. The listing stays, with its dates, and shows that we stopped reading. To remove it too, see data takedown.