The gap isn't an estimation error
When a warehouse project announced at nine months takes eighteen, the instinct is to blame the initial estimate. In most cases, the estimate was accurate on the visible scope: business rules, indicators, reports.
What was underestimated is the invisible part: loading code, controls, historical backfill, rejection handling, documentation, rework after source changes. This part doesn't show up in a functional spec, but it accounts for most of the real effort.
The Four Drift Hotbeds
- Manual pipeline writing: each source table produces repetitive code, written and tested by hand, with variations from one developer to another.
- Orchestration rebuilt separately: dependencies, incident recovery, scheduling — a project within the project, often handed to an external tool to administer.
- Quality treated at the end of the chain: controls arrive after production, when discrepancies are already arbitrated in meetings.
- Manually derived documentation: it's obsolete the day it's delivered, making each evolution slower than the last.
Adding engineers does not speed things up
Growing the team also increases the number of competing conventions, code reviews and sync points. On non-standardized repetitive work, the marginal output of an additional engineer quickly becomes low.
The useful question isn't “how many engineers do we need?” but “how much of what our engineers produce could be generated deterministically?”. On a Data Vault platform, the answer is: most of it.
What automation changes
When structures, loads, quality controls and orchestration are generated from the model, adding a source is no longer a project but an iteration. Teams spend their time on business rules — the only part that can't be automated.
Our clients measure delivery roughly 7 times faster on new sources, with a return on investment reached in six months on average. This isn't an individual productivity gain: it's the elimination of an entire category of work.
How to verify at your place in one hour
- Take the last three integrated sources and count the man-days actually consumed, excluding business rules.
- Identify the share of time spent writing or fixing loading code and orchestration dependencies.
- Relate this volume to the number of sources yet to be integrated on your roadmap.
- The resulting figure is your annual opportunity cost — and the basis for a business case you can defend internally.