LocalBeatBot

The crawler behind LocalBeat's event listings.

What it fetches

LocalBeatBot reads public event listings: venue calendars, iCal and RSS feeds, and event pages that a venue or organizer has configured as a LocalBeat source. It fetches only the pages needed to keep those listings current, and it does not crawl links beyond them.

How it identifies itself

Every request carries this User-Agent:

LocalBeatBot/1.0 (+https://localbeat.com/bot.html; [email protected])

Every request is also signed with an HTTP message signature (RFC 9421, Web Bot Auth). The public keys that verify those signatures are published at:

https://localbeat.com/.well-known/http-message-signatures-directory

That directory response is itself signed with the same keys, so you can verify it came from LocalBeat before trusting what it lists.

A request that claims to be LocalBeatBot but does not carry a signature that verifies against one of those keys is not from us.

robots.txt

LocalBeatBot reads your robots.txt before fetching and follows it, including Crawl-delay. Its robots.txt token is localbeatbot (case does not matter). It follows your User-agent: * rules unless you give it a block of its own, in which case only that block applies.

To keep it off part of your site, or to slow it down:

User-agent: localbeatbot
Disallow: /private/
Crawl-delay: 10

To allow it into one area only:

User-agent: localbeatbot
Allow: /allowed-here-only/
Disallow: /

To keep it off your site entirely:

User-agent: localbeatbot
Disallow: /

When no crawl-delay is set it still waits at least one second between requests to the same host, and it backs off whenever a server answers 429 or 503.

Questions or concerns: [email protected]