BadgerflyBot

What BadgerflyBot is

BadgerflyBot is the web crawler of Badgerfly, an SEO platform used by agencies and in-house teams to check websites they work on. It visits a site when a Badgerfly user asks for something about that site: a site audit, a check that a backlink is still in place, a look at a directory listing for a business, a comparison of competing pages. It does not crawl the web at large, build a search index, or collect personal information from the pages it visits.

How to recognise it

BadgerflyBot identifies itself in the User-Agent header in one of two forms:

Mozilla/5.0 (compatible; BadgerflyBot/1.0; +https://badgerfly.com/badgerflybot) BadgerflyBot/1.0 (+https://badgerfly.com/badgerflybot)

The first is used by site audits (including the pages it renders in a headless browser), page fetches for content and cannibalization research, backlink checks and the backlink monitor. The second is used for business directory listings, link placement checks, domain signals and directory sitemaps.

We do not publish a list of IP addresses, because the addresses our requests come from can change. Identify BadgerflyBot by its User-Agent.

robots.txt

BadgerflyBot reads /robots.txt before it fetches a page and follows the group for BadgerflyBot, or the * group when there is none. For site audits it also honours Crawl-delay (up to 10 seconds between requests). If robots.txt cannot be fetched, the site is treated as having no rules.

To keep BadgerflyBot off your whole site:

User-agent: BadgerflyBot Disallow: /

To slow it down:

User-agent: BadgerflyBot Crawl-delay: 5

A site audit records pages that robots.txt blocks as "blocked by robots.txt" and does not fetch them. One exception during a site audit: the images, scripts and style sheets used by a page that was allowed are requested once each to check that they load, as a browser would request them.

How hard it crawls

• One site audit visits at most the page limit its user set (200 by default, 5,000 at most), with at most 5 requests at a time to one host.
• Each request waits at most 15 seconds and follows at most 5 redirects. Pages larger than 5 MB are noted by size and not read.
• If your server answers 429 (or 503 with a Retry-After header), BadgerflyBot waits for the time you give in Retry-After, or 5 seconds doubling on each further refusal (up to 90 seconds), retries at most three times, and slows every later request to your host (1.5 seconds between requests, growing to 10 seconds while you keep refusing).
• A sample of up to 30 pages per audit is opened in a headless browser to compare the rendered page with the raw HTML, one page at a time.

Fresh fetches and PageSpeed

An audit is meant to show your site as visitors get it today, not as a cache remembers it. So, when it fetches pages of the site being audited (not images, scripts or other sites), BadgerflyBot:

• sends Cache-Control: no-cache and Pragma: no-cache, and
• adds a query parameter _bfnc=<random value> to the URL, so a CDN or page cache returns a fresh copy.

The same audit asks Google PageSpeed Insights for a fresh measurement of up to 20 of the site's pages (mobile and desktop) instead of reusing a recent one. These requests come from Google's PageSpeed service, not from BadgerflyBot.

If the extra parameter causes trouble on your site, tell us at contact@geeksandme.com and we will look at it.

Blocking or contacting us

• To stop BadgerflyBot, use robots.txt as above. A site audit reads robots.txt once, at its start; other checks re-read it at most once an hour, so a new rule can take up to an hour to apply to them.
• To ask a question, report a problem or ask us to stop visiting a site: contact@geeksandme.com. Include your domain and, if you can, the time and the User-Agent from your logs.

BadgerflyBot is operated by Geeks and Me LLC.