Skip to content
crawlforgeEspañol

For site administrators

You found this in your logs

Then something requested your pages with this User-Agent, and you want to know what it is and how to make it stop. Both answers are on this page, the second one near the top.

CrawlForge/1.0 (+https://crawlforge.org/bot/)

How to stop it

The standard way, and it works: robots.txt. CrawlForge reads it before anything else and obeys it.

User-agent: CrawlForge
Disallow: /

It also honours Crawl-delay, and honours it properly: a host that declares one is crawled with a single request in flight and that much time between the start of one request and the next.

One exception you should know about, because we would rather tell you than have you find it: the tool has a flag that ignores robots.txt. It exists because the people who use this audit sites they own, and an owner sometimes needs to see what a search engine cannot. It is off by default, it has to be typed on purpose, and the pages it reaches are marked in the report as blocked — the point is to see the block, not to get around it.

What it is

A technical SEO auditor. Someone points it at a site, it crawls that site, and it reports what is wrong with it: broken links, redirect chains, missing titles, pages nothing links to. It is meant to be run by whoever owns or maintains the site, and if that is not you, the section above is the answer you came for.

There is no IP range, and that is not evasion

CrawlForge is not a service with a fleet of crawlers. It is a program that runs on the computer of whoever started it, so the requests come from their address, not from ours. There is no list of IPs to allow or block, and anyone claiming to publish one is not describing this tool.

What that also means: the User-Agent is the only thing identifying it, and anyone can put any string in a User-Agent. If what you are seeing is aggressive, ignores your robots.txt, or does not back off, it is worth considering that it may not be this.

What it does not do

  • It does not run JavaScript. It reads the HTML your server sends and nothing else.
  • It does not crawl other people's sites. Links pointing away from the audited site get a headers-only request to see whether they still resolve — no body is downloaded, nothing is parsed, nothing is stored beyond the status code.
  • It backs off. Three responses in a row that say you are overloaded — 429 or 503 — halve its concurrency for your host, and it recovers slowly on purpose. A crawler that takes down the server it was auditing is a useless crawler.
  • It does not collect personal data from your pages, and it has no interest in your content beyond the tags that decide how a search engine sees it.

If something is wrong

If this crawler behaved badly against your server — ignored a Disallow, hammered you, anything — we want to know, because that would be a bug and not a feature. Write to info@colorvivo.com with the date, your host and a few log lines, and it gets looked at.

The crawler is open source, so you do not have to take any of this on trust: the code that reads robots.txt and the code that decides when to slow down are both readable in the repository.

Run by [object Object], a company registered in Spain. Full details on theprivacy and cookies page.