Public-web pipelines fail in boring ways: a layout changes, an encoding breaks, a rate limit appears. Engineering for those failures — detectably, recoverably — is what separates a dataset from a science project.
Every experienced data engineer has a story about a pipeline that worked beautifully until the day a marketing team redesigned a product page and three downstream dashboards went quietly wrong. Public-web pipelines live in an environment you don't control and can't negotiate with. Reliability here means something specific: failures are expected, so they must be detected fast and contained locally.
The unit of engineering is not "a scraper" but a source profile — a documented contract with each public source covering what is collected, at what cadence, under what politeness constraints, and with what structural assumptions. The profile is versioned like code, because every structural assumption it makes *will* eventually be violated.
source: example-retailer-listing
kind: public_catalog
cadence: daily
politeness:
max_concurrency: 2
rate_per_domain_per_min: 6
respect_robots: true
extraction:
schema: listing.v3
required: [title, price, currency, availability]
optional: [promo_label, rating, review_count]
drift:
fingerprint: dom_structure_v2
on_change: quarantine_and_alert
quality_gates:
completeness_min: 0.97
price_change_max_pct: 40 # flag for review, not discardStructural fingerprints — stable hashes over the DOM regions you actually parse — catch layout changes early. But the more insidious failures are *semantic*: the field is still there, and now means something slightly different. That's why every pipeline needs two layers of monitors:
The design rule we follow: no silent failure. Every degradation either passes a quality gate and ships, or is quarantined and alerts a human. "Shipped with a warning" is a state, visible in delivery metadata — never a default.
Raw extraction gives you strings; delivery requires meaning. Normalization maps those strings to canonical forms — currencies unified, units converted, entities resolved, vocabularies standardized. Do it in an explicit, versioned layer rather than scattering string munging across consumers. The quality framework only works if normalization is inspectable.
Median time-to-alert across monitored sources, trailing 30 days.
Respectful collection — conservative concurrency, identified user agents, honored directives — isn't just ethics; it's what keeps sources stable enough to build on. Aggressive collection gets you blocked, and blocks are the most expensive failure mode there is. The acceptable-use boundaries we publish are engineering constraints as much as policy ones.
A pipeline with real monitoring changes the conversation downstream. Analysts stop asking "is this data right?" and start asking "what is this data saying?" — which is, bluntly, the only question worth paying for. Everything in this post exists to protect that conversation.
The last mile of market intelligence is unglamorous and decisive: how data actually reaches the systems where decisions happen. Delivery design notes for API, warehouse and hybrid patterns.
“No silent failure” — simple to say, hard to enforce. The three-outcome gate design is the cleanest formulation I’ve seen.
Any chance of a follow-up on testing extraction logic? Fixture-heavy testing for parsers is where our coverage is weakest.
4 comments
The source-profile YAML is a great pattern. We’ve adopted something similar — including the quarantine-on-fingerprint-change rule — and on-call pages dropped noticeably.
The quarantine rule is controversial until the first time it saves a delivery, Clara. Thanks for the confirmation from the field.