RivalScoutBack to site

RivalScout crawler

Last updated 9 August 2026

Identity

Every request the RivalScout crawler makes carries this user-agent string, unchanged and without alternatives:

RivalScout/1.0 (+https://rivalscout.io/bot)

If a request reaches your server without that string, it did not come from RivalScout. Requests originate from Cloudflare Workers infrastructure, so source addresses vary.

What it does

RivalScout is an AI market analyst. Customers name the companies they want to follow, and on each scheduled run the crawler reads a small number of public pages belonging to those companies so that changes between runs can be described.

Typical pages fetched:

  • Home page and product pages
  • Pricing pages
  • Changelog, release notes and status pages
  • Blog, news and press index pages, plus their RSS or Atom feeds
  • Careers and job listing pages
  • /robots.txt and /sitemap.xml

What it never does

  • Fetch a path disallowed for our user-agent or for * in your robots.txt.
  • Sign in, submit forms, present cookies to enter a gated area, or work around a paywall or login wall.
  • Attempt to reach private or internal network addresses.
  • Follow more than five redirects, or download more than 5 MB from a single page.
  • Execute scripts, click adverts, or generate analytics events beyond the page request.

Rate and frequency

The crawler makes at most two concurrent requests to one hostname and leaves at least two seconds between requests to the same host. A Crawl-delay directive is honoured up to thirty seconds. Most sites are read once per day at most, and many once per week, depending on the customer’s cadence.

We send If-None-Match and If-Modified-Since where you have given us an ETag or Last-Modified value. A 304 Not Modified saves us both the transfer, so those headers are the fastest way to reduce our load on your server.

How to block it

Add the following to /robots.txt at the root of your site. The change is picked up within twenty-four hours.

User-agent: RivalScout
Disallow: /

To exclude only part of the site:

User-agent: RivalScout
Disallow: /private/
Crawl-delay: 10

You may also block by user-agent string at your CDN or firewall. We treat a 403 as a permanent refusal and stop fetching the source after repeated rejections.

Contact

Questions, complaints, or a request to stop crawling entirely: bot@rivalscout.io. We reply to crawler requests within two business days and act on removal requests immediately.