Scaling Collection Responsibly

Anyone can hammer a public endpoint. Scaling collection responsibly — growing coverage while remaining the kind of participant the open web tolerates, and ideally welcomes — is a genuine engineering discipline. This post collects the practices we've converged on after years of running collection infrastructure continuously.

Politeness is an engineering spec

"Be polite" is not a spec; the following is:

• Per-domain budgets — concurrency and request-rate ceilings enforced in code, not in team folklore.
• Identifiable agents — honest user-agent strings with contact information, so a source operator can reach a human instead of reaching for a blocklist.
• Directive honoring — robots and equivalent signals respected, with edge cases escalated to humans rather than parsed creatively.
• Off-peak scheduling — where a source has obvious load cycles, collection runs in the troughs.

These constraints are cheap. What they buy is durability: sources that don't feel abused rarely need to defend themselves, and a decade of stable sources is worth more than a quarter of aggressive coverage.

Governance scales people, not code

The hard ceilings on any collection program are governance ceilings: what may be collected, for which documented purposes, with what retention. We encode this as a program-level review — every new source gets an evaluation against the acceptable-use boundaries before a line of collection code exists. Feasibility isn't just "can we parse it" but "should we collect it, and can we defend that choice in a room of skeptical adults."

The economics of good citizenship

Responsible collection is often framed as a cost. The ledger reads differently over any horizon longer than a quarter: respectful programs avoid block-induced delivery gaps, build reputational capital with source operators, and accumulate longitudinal history that aggressive programs keep resetting through bans. History depth is a moat; bans are an anti-moat.

What we tell new engineers

Three sentences, usually in week one: you are a guest on other people's infrastructure; the long-term dataset matters more than this week's completeness; and when policy and pressure conflict, quality and ethics win, escalate, and sleep well. A collection program that internalizes those three sentences can scale almost indefinitely. One that doesn't will eventually be scaled *down* by someone else.