CompanyMapBot

1. What it is

Company Map UK publishes a map of UK companies built from the free Companies House register. The register records a company's name, number and registered address, but it does not record a company's website. CompanyMapBot is how we work out which website belongs to which registered company.

It identifies itself with this User-Agent string:

Mozilla/5.0 (compatible; CompanyMapBot/0.1; +https://companymap.co.uk/bot.html)

2. What it actually fetches

For most sites, a single request to the homepage and nothing else. It reads the visible text looking for a UK company registration number, a GB VAT number, a "registered in England and Wales" statement, or a UK postcode — the details a company is generally required to publish anyway.

If a homepage carries a UK signal but no company number, it may follow up to a few more pages that normally do: the privacy, terms, legal, about or contact page, and any linked certification document. It never fetches more than about nine pages from one site, and for the overwhelming majority it fetches one.

It does not submit forms, does not attempt to log in, does not fetch images or video, and ignores anything behind a paywall or login.

3. What we keep

Where we can match a site to a registered company, we keep the domain, the company number, and the headings and paragraph text of the pages above — so we can describe what the company does without fetching the site again. Where we cannot, we keep a single line recording that the domain was checked, so we do not re-check it needlessly.

We do not collect personal data as a goal. Business contact details published on a company's own website (a general enquiries address, a switchboard number) may be captured incidentally as part of the page text. If you want your domain and anything derived from it removed, email crawler@companymap.co.uk.

4. How often, and how fast

Any individual site sees a handful of requests once, not a repeated crawl. We re-check a domain at most a few times a year. Requests to one site are sequential, never parallel — the crawl gets its speed from visiting many different sites at once, not from hitting yours hard.

The crawler runs from AWS in Ireland (eu-west-1). Addresses are ephemeral, so we cannot publish a fixed IP list; please match on the User-Agent above.

5. How to block it

CompanyMapBot fetches and obeys robots.txt before requesting anything else. To exclude it entirely, add:

User-agent: CompanyMapBot
Disallow: /

To allow the homepage but keep it away from part of your site:

User-agent: CompanyMapBot
Disallow: /members/
Disallow: /admin/

A blanket User-agent: * rule that disallows the paths in question works too — we honour those the same way. Changes take effect the next time we look at your site; we do not cache robots.txt between runs.

6. If something goes wrong

If CompanyMapBot has caused a problem — unexpected load, a misbehaving request, anything appearing in your logs that does not match the description above — please tell us and we will investigate and stop. Include your domain and, if you can, the log lines.

Operated by: Optional LTD
Crawler contact: crawler@companymap.co.uk
Website: companymap.co.uk