Skip to main content

DoggsLifeBot

DoggsLifeBot is the crawler of Doggs.Life, a site where dog owners find groomers, vets, trainers, daycare, boarding and pet stores near them. It identifies itself as:

DoggsLifeBot/1.0 (+https://doggs.life/bot)

Why it visits your website

Many listings on Doggs.Life are built from openly licensed map data, and the business behind a listing may not know it exists. To tell a business about its free listing, and how to claim, correct or remove it, the crawler looks for the email address that business published on its own website. It visits only the website already shown on a listing.

What it fetches

  • robots.txt first, and nothing else if it asks crawlers (or DoggsLifeBot) to stay away. If robots.txt cannot be read, for any reason but a plain 404, the site is skipped.

  • The homepage, and at most 2 more pages linked from it that look like a contact, privacy or about page: never more than 3 pages per site.

  • At least one second between requests, or the Crawl-delay your robots.txt sets (a site asking for more than 30 seconds is skipped instead). Redirects are followed only within the same website.

  • It never submits forms, signs in, runs scripts or downloads files, and it does not come back to a site for 180 days.

What it keeps

  • Only email addresses written out on those pages, on the business’s own domain or a free mailbox the site shows, together with the address of the page each was found on. Someone at Doggs.Life reviews every address before it is used.

  • It does not decode disguised addresses (such as “name [at] example [dot] com”), guess or complete addresses, copy page content, or use anything it reads to train models.

  • It takes nothing from a website that says it does not share, sell or transfer email addresses, or that forbids collecting them. It also honours noindex and nofollow in robots meta tags and the X-Robots-Tag header.

  • Any email we send says where we found the address and carries a one-click way to opt out.

How to block it

Add these lines to the robots.txt at the root of your site. The crawler reads the file at the start of every visit, so the change applies from its next one.

User-agent: DoggsLifeBot
Disallow: /

To keep it from taking anything from one page, or following that page’s links, add:

<meta name="DoggsLifeBot" content="noindex, nofollow">

Other requests with this user agent

  • A single page, when a member shares its link in a chat, to show a preview.

  • A business’s homepage, when its owner asks us to check the verification tag they added to it.

Opting out and contact

To stop email from us about your business, or to correct, hide or remove its listing, use the request form or the opt-out link in any email we sent.

Questions, or a request to stop visiting a site regardless of robots.txt: bot@doggs.life.