We wrote a tool that competes with Screaming Frog, so any comparison we publish is suspect by definition. Fair enough. What you can do is check the numbers yourself, and for that we have to say how they were measured.
Let us start with the part that took us longest to accept.
Without a licence you can only compare 500 URLs
Screaming Frog caps free crawls at 500 URLs. The licence costs £199 a year. We had one and let it lapse, so every comparison in this article was run against the limited version.
That constrains what anyone can claim. If someone shows you a speed comparison over a 100,000-URL site against the free version, they are selling you something, because the free version does not crawl 100,000 URLs.
What you can compare without a licence, and what actually matters, is extraction.
The same 300 URLs to both
Both tools have a list mode. You hand it a file of URLs and it crawls exactly those, no more. That makes the test fair: if you compare by crawling, each one decides where to start and what to follow, and you end up comparing two journeys instead of two extractions.
We took 300 URLs from colorvivo.com, all returning 200 and all HTML:
sqlite3 crawl.sqlite "SELECT url FROM urls WHERE is_internal=1
AND status_code=200 AND content_type LIKE 'text/html%'
ORDER BY url LIMIT 300;" > list.txt
crawlforge list list.txt --out cf.sqlite
"/Applications/Screaming Frog SEO Spider.app/Contents/MacOS/ScreamingFrogSEOSpiderLauncher" \
--crawl-list "$PWD/list.txt" --headless \
--output-folder "$PWD/sf" --export-tabs "Internal:All" --overwriteScreaming Frog’s headless mode works with the app open, but it insists on absolute paths. That cost us ten minutes.
Then we compared six fields per URL: status code, title, meta description, H1, canonical and indexability. Eighteen hundred comparisons.
The result: one difference
OK status 0 of 300
OK title 0 of 300
OK description 0 of 300
≠ h1 1 of 300
OK canonical 0 of 300
OK indexability 0 of 300One. And it was ours.
The home page H1 was this:
<h1>Agencia Especializada en WordPress<br />con +25 años</h1>We returned WordPresscon. The <br> splits the content into two text nodes and we were joining
them with nothing in between. Screaming Frog got it right.
A one-line defect that had been there from the start, invisible to 810 tests, because a crawl of your own has nothing to check its answer against. We fixed it that afternoon and reran the test: zero differences across 1,800 comparisons.
That is the conclusion we care about, and it is not the one usually printed in a comparison: they extract the same thing. If you are thinking of switching tools, the first thing you need to know is that the data will match.
Speed, with every caveat
On those same 300 URLs, CrawlForge took 12.7 seconds and Screaming Frog 16.7. Part of that gap is the Java virtual machine starting up.
We are not turning that into a headline. It is 300 URLs of a site behind Cloudflare, measured once, on one particular MacBook. The network dominates the result, and a serious comparison would need many more runs and several subjects.
Where the difference does matter is on another metric.
Memory, which is the real argument
Screaming Frog has two storage modes. By default it keeps the crawl in memory, which is fast and has a clear ceiling: with the standard configuration, around 500,000 URLs if you give it enough RAM. Database mode raises that ceiling at the cost of speed.
We have one mode, and it is not configurable: every crawl is a SQLite file. The consequence is that memory does not grow with the size of the site.
On a full crawl of a news site with fifteen years of archive — 487,621 URLs, 4.4 million images, a 5.3 GB file — the process stayed at 202 MB. It rose to 259 MB at peak queue, with 155,000 URLs waiting, and dropped to 123 MB once the queue drained.
That one is reproducible: crawl a large site and watch your activity monitor.
What Screaming Frog does better
This is the part that makes a comparison worth reading.
The interface. Ours does not exist. CrawlForge is a command line today, and for many people that alone is reason enough not to look. Screaming Frog has spent fifteen years polishing an interface where you filter, sort, and click a URL to see its inbound links.
The catalogue. We have 59 rules. They check considerably more, and some of what we do not cover matters: JavaScript rendering, page speed analysis, custom extraction with XPath.
The integrations. Search Console, Analytics, PageSpeed Insights, Majestic, Ahrefs. Attaching data from those sources to a crawl changes what you can conclude. We have none.
The community. Any question you might have, someone has already had it and written the tutorial. With us you are on your own, or you ask.
Rendering. A site that builds its content with JavaScript looks empty in CrawlForge. It is planned and it does not exist.
What we do better
Comparing two crawls. It is the reason the tool exists. Screaming Frog has crawl comparison in its paid version, and it works, but it is built for looking at two audits rather than for becoming routine across a portfolio.
Fitting in a pipeline. crawlforge diff --fail-on high exits non-zero when a deploy introduces
something serious. Auditing the built folder before publishing touches no network and takes seconds.
The open file. A crawl is a plain SQLite database. Any question the report does not answer, SQL does. No exporting.
The price. The engine and the command line are Apache 2.0. Free, with the code visible so you can check what it does with the credentials you hand it.
Who each one is for
If you audit sites occasionally, want a graphical interface and value a wide catalogue with integrations, Screaming Frog is the better tool and £199 pays for itself quickly. We say that having paid it for years.
If you look after many sites, review them repeatedly, and what you need is to know what moved since last time, try ours. And if what you want is a technical SEO check inside your continuous integration, we think there is not much competition there today.
If you are in the second case, the sensible thing is to have both. We still have Screaming Frog installed.