Designing Market Monitoring That People Trust
Monitoring systems live or die on a human variable: whether the people they serve believe them. Notes on building that trust — provenance, humility, and the art of the visible caveat.
Data Quality 3 min read
Data Quality in Large-Scale Web Intelligence
Quality isn't a cleanup step — it's a designed property with measurable gates. The validation framework we apply before any dataset leaves the building.
"Is this data good?" is an unanswerable question, and the fact that stakeholders keep asking it means the provider has failed to give them something better: measured quality on stated dimensions. This post is the framework behind our answer — the same one summarized in every dataset's delivery metadata.
Four dimensions that matter for web data
| Dimension | Definition | How it's measured |
|---|---|---|
| Completeness | Required fields present and non-null | Per-source null-rate against schema requirements |
| Validity | Values conform to type, range and vocabulary | Schema validation + distribution bounds |
| Freshness | Age of observation relative to promise | Collection timestamp vs. delivery SLA per cadence tier |
| Consistency | Same entity means the same thing across sources and time | Cross-source entity resolution + historical coherence checks |
Accuracy — the dimension everyone asks about first — is deliberately delivered as the *outcome* of the other four plus sampling audits, not as a vague self-assessment. You can't certify accuracy on millions of records; you can measure it rigorously on audited samples and report confidence honestly.
Quality gates, not quality reviews
The framework is enforced as gates in the pipeline, not as post-hoc inspection. Each gate has three possible outcomes: pass (ship), fail (quarantine and alert), or degrade (ship with visible warning metadata). There is no fourth path where bad data ships silently, because "silent" is the only failure stakeholders never forgive.
{
"dataset": "competitor_prices_eu_v2",
"observation_window": "2026-09-01T00:00Z/2026-09-08T00:00Z",
"gates": {
"completeness": { "status": "pass", "rate": 0.994 },
"validity": { "status": "pass", "violations_ppm": 12 },
"freshness": { "status": "pass", "p95_lag_hours": 19 },
"consistency": { "status": "degraded",
"note": "source#41 quarantined for layout change" }
},
"audit": { "sample_size": 1200, "field_accuracy": 0.987 }
}Drift is the real adversary
In static databases, quality decays through process failure. In web data, it decays through reality changing — sources redesign, vocabularies shift, products vanish. That's why drift monitoring gets its own engineering, and why consistency is measured *across time*, not just across sources. A dataset that was 99% consistent last quarter and 92% this quarter isn't 92% good; it's a change detection problem wearing a quality costume.
3gate outcomes: pass, degrade, quarantine — nothing ships silently
1,200records in the standard per-delivery audit sample
0acceptable silent degradations (design tolerance, not aspiration)
Program defaults across current enterprise engagements; audit sample size is configurable by tier.
What to ask any data vendor
- Which quality dimensions do you measure, and where do the numbers live?
- What happens when a gate fails mid-delivery — show me the path, not the policy.
- How do I see quality *for the fields I actually use*, not dataset-wide averages?
- What did your last three degradations look like from the customer side?
Vendors who answer fluently have a framework. Vendors who answer with adjectives have a marketing deck. The security and acceptable-use posture of a provider is worth reading alongside quality — a program that's careless about boundaries is usually careless about validation too.
Related reading
Continue here
Tomás Oliveira
Delivery metadata with a quality block should be an industry standard. The ‘degraded’ example with the quarantined source note is exactly right.
Jordan Alvarez
Accuracy-as-an-outcome-of-the-other-dimensions is a genuinely useful reframe. Explains why ‘99% accurate!’ claims never came with methodology.
3 comments
The vendor questions list is worth the article alone. We used it in an RFP last month; two of four vendors couldn’t answer question two.