A flaw that breaks nothing — and that's exactly the problem
Silent duplication doesn't fail any load. The pipeline shows green, dashboards render, nobody is alerted. It's just that a satellite meant to record three changes a year records three hundred instead.
The consequences show up elsewhere: volumes explode, history queries slow down, and above all metrics like “number of address changes” or “time between status changes” become wrong without anyone knowing why.
Common causes
- A hash diff calculated on a set of columns including a technical timestamp, a batch number or a counter: each execution produces a different fingerprint.
- Unstable normalization of null values, trailing spaces, or case between two executions or two pipelines.
- Column order not fixed in the concatenation before hashing.
- Float types or time-zoned dates that serialize differently depending on the engine.
- A full reload of the source treated as a delta, without comparison to the last known state.
How to detect it in ten minutes
On each satellite, compare the row count to the number of distinct (key, content hash) pairs. A large gap signals rows that add no new information.
Complement this with a frequency measure: the average number of versions per key per month. When this value tracks the pipeline's run frequency rather than the actual business rhythm, the diagnosis is confirmed.
Fixing data without losing auditability
The temptation is to purge duplicates after the fact. That's risky: in a Raw Vault, history is authoritative, and any deletion must be traceable and justified.
The sound approach is to first fix the hash-calculation rule and normalization, start clean for subsequent loads, then treat existing history as a documented remediation operation, keeping a record of what was consolidated.
Preventing it by design
This flaw almost always comes from a rule implemented by hand, differently in each pipeline. When normalization and hash calculation are generated from the model, they're identical everywhere — including on sources added two years later.
beVault enforces these rules at generation time and natively embeds quality checks on volumes, freshness, and version consistency. Silent duplication becomes an alert, not a chance discovery during an audit.