Installing CrawlForge without Rust

Until 0.21.0, the only way to get CrawlForge was to clone the repository and compile it with Rust. That is a fine test of patience and a poor way to try a tool. 0.22.0 changes how you get it and leaves the engine alone: what it checks and what it writes are the same as in 0.21.0. What is new is that it can be installed by people who have never typed cargo.

brew install dcarrero/tap/crawlforge                    # macOS and Linux
curl --proto '=https' --tlsv1.2 -LsSf https://github.com/dcarrero/crawlforge/releases/latest/download/crawlforge-installer.sh | sh
powershell -ExecutionPolicy Bypass -c "irm https://github.com/dcarrero/crawlforge/releases/latest/download/crawlforge-installer.ps1 | iex"
cargo install crawlforge                                 # Rust 1.88 or newer

The latest release has binaries for macOS on Apple Silicon and Intel, Linux on x86_64 and ARM64, and Windows on x86_64, each with its checksum. On crates.io there are three packages: crawlforge, the command line, plus the engine and the rule catalogue as crawlforge-core and crawlforge-rules.

One caveat, said upfront. The macOS binaries are not notarized yet. Homebrew installs them without any warning, but a binary downloaded by hand needs xattr -d com.apple.quarantine crawlforge before Gatekeeper lets it run.

How a release gets built, for free

The releases are built by GitHub Actions, which costs nothing for a public repository, with dist 0.34 (it used to be called cargo-dist). Pushing a tag starts the whole thing. It builds for aarch64 and x86_64 macOS, x86_64 and aarch64 Linux (the ARM64 one on GitHub’s ARM runner) and x86_64 Windows, computes the checksums, writes the shell and PowerShell installers, creates the GitHub release and pushes the Homebrew formula to dcarrero/homebrew-tap. A tap can hold more than one tool, and this one already held another of my projects.

In the 0.22.0 run, Linux took between 2m36s and 2m50s, macOS on Apple Silicon 3m16s, Windows 5m11s and macOS on Intel 10m35s. The Intel time is mostly waiting: that runner queue is the slow one.

The binary is built with a dist profile that inherits the release profile unchanged. That matters because the performance regression is measured on the release build, so what ships is the build that gets measured.

The workflow writes to the tap with a GitHub token limited to the contents of that one repository, stored as a repository secret.

What went wrong, in order

Windows and line endings. 0.21.0’s CI failed only on Windows. A test compared the published JSON Schema with the generated one byte for byte, and a Windows checkout converts line endings to CRLF. The test now compares the content.

A timing test on a busy runner. One test checks that sitemap discovery keeps the queue topped up continuously instead of in batches. Locally it passed in 0.7 s. On GitHub’s macOS runner, which is shared and loaded, it took 2.1 s twice against a 1.5 s threshold. Raising the threshold by eye would have made the test pass without saying anything. I made the slow test sitemap slower instead, from 600 ms to 1.5 s, so that batching would take at least 4.5 s and continuous refilling about 1.5 s. The threshold is now 3.5 s, with a wide gap on each side.

A tag with no release. 0.21.1 was tagged and never released. Every build failed with profile 'dist' is not defined, because the profile was missing from Cargo.toml. The tag was already public, so I left it as it was rather than rewriting it. 0.21.2 added the profile and was the first release with binaries, still under the old crawlforge-cli-* names.

A rename that broke a link. 0.22.0 renamed the package from crawlforge-cli to crawlforge, so that it installs as cargo install crawlforge. That also renamed the release files, and the installer link published with 0.21.2, crawlforge-cli-installer.sh, stopped working with this release. This is why the website waited for 0.22.0 before documenting installation.

Packaging for crates.io. A crate published to crates.io contains only its own folder. The ten translations lived at the root of the repository, and the code read them with paths like ../../locales. Packaged, it would not have compiled. They moved inside crawlforge-rules, which now hands the command line its texts. Before publishing, I unpacked the three packaged crates in an empty folder and built them together, as if they had come from crates.io. The result printed crawlforge 0.22.0, and the Japanese and Polish output came out right.

Publishing without a stored token

The first publish to crates.io had to be manual, with an API token limited to these three crates and to publishing. The first token was rejected with a 403 because its permissions were wrong, and nothing was uploaded. The second one worked.

From then on, the release workflow uses crates.io Trusted Publishing. The workflow proves its identity to crates.io through GitHub’s OpenID Connect and gets a short-lived credential for that run. No long-lived token is stored anywhere, and the manual one was revoked and deleted.

The crates job also checks the crates.io API before publishing and skips versions that are already there, so a re-run doesn’t try to publish the same version twice. It did exactly that for 0.22.0, in 9 seconds.

Checks that now run on GitHub

cargo-deny, with the project’s deny.toml, runs on every pull request, on every push and every Monday. The Monday run is there because a new RustSec advisory can appear without anyone touching the code.

Every pull request that touches code also gets a performance A/B. The base and the branch are built on the same runner and run in six alternating rounds, and the job summary shows a table. It never blocks the pull request. In the first one the base median was 106,567 elements per second and the branch 106,046, a difference of −0.5 %, with rounds between 102,527 and 110,510 (about ±3 %). On my laptop the same benchmark had swung around 20 % whenever the machine was busy compiling other things, which is why it moved to a runner.

A test now fails if the version numbers that have to match drift apart: the workspace version, the two internal dependency versions, and the package.json this website installs by tag.

Trying it on a Mac

Before writing this I installed 0.22.0 all three ways on a Mac. Homebrew took a few seconds, 14.6 MB, with no Gatekeeper quarantine. The shell installer and cargo install crawlforge both reported crawlforge 0.22.0.

Next on this list are notarizing the macOS binaries and a winget package for Windows.