Insights on data engineering, architecture, and the evolving analytics landscape.
Every pipeline that loads a source more than once has to work out what changed since the last run, and each way of doing that misses some changes while the run still reports success. Which of the four load patterns fits your pipeline, and do you know the consequences of the one you picked?
Read Article →
At a previous client every source loaded through a stored procedure that assembled the query, and the parallel loop had been set to thread = rand_int() % 1 for longer than anyone knew. It kept running, which is why nobody fixed it. This is the design we built instead, in four objects and one run lifecycle.
Read Article →
Every BI project ends up with the same request: can we export this to Excel? Conversational agents are the first thing I have seen that gives people a reason not to, and only if the semantic model underneath them is in a state where the answer can be trusted.
Read Article →
Hand-issued access charges you twice, in a grant web nobody can reconstruct and in datasets nobody knows exist. Both costs come from a human sitting in the grant path, and computing grants from identity attributes and data classification takes them out of it.
Read Article →
Meaning is committed once on the way into the warehouse and has to be resolved once on the way out. The exit edge is the one that never became authoritative, and it is why a certified Gold table still yields three numbers.
Read Article →
Self-service BI gets sold as a licensing decision and almost never delivered. Here's the usability bar a Gold dataset has to clear to earn it, and the one gap no data layer can close.
Read Article →
Without enforcement, contracts are just documentation. Here's how structural and value-level checks protect the ingestion boundary, and what the pipeline does when one fails.
Read Article →
Schema drift is inevitable, and the instinct to rewrite Bronze every time the source changes quietly breaks the captured history your recovery depends on. The durable fix is to absorb drift in the source-aligned Silver layer, version Bronze instead of rewriting it, and hold the producer to a real data contract.
Read Article →
Fourteen months after my Ben contract ended, the customer service system that held my passport number was breached. Five architectural decisions in that system shaped how the incident landed, and each is a pattern data professionals already follow.
Read Article →
How GDPR right to erasure interacts with the layered warehouse architecture, and why crypto-shredding with an anonymization tombstone resolves the tension between immutable Bronze and the legal obligation to forget.
Read Article →
GDPR personal data reaches well beyond names and emails, and ordinary fields can combine into Article 9 special-category data. Why that combination sensitivity has to be caught at ingestion design rather than in a later legal review.
Read Article →
Incremental pipelines built on updated_at watermarks cannot detect hard deletes. The result is a ghost row in your SCD Type 2 model and quietly wrong analytics downstream.
Read Article →
SAFe wraps agile vocabulary around a plan-driven framework and sells the result as scaled agility. What it actually scales is the appearance of agility, not the substance. For data teams, the damage is structural and compounding.
Read Article →
Governance only scales when metadata is produced by delivery workflows, not manual stewardship.
Read Article →
Principles, governance, ownership, and an implementation blueprint for operating the Silver semantic layer in practice.
Read Article →
The Silver-Gold boundary is where analytics platforms win or lose consistency. This part covers what the boundary is, why it fails, and how to design it explicitly with Conformed Entities.
Read Article →
Data professionals often default to normalized semantic models, and it tends to come back to bite them. Here is why the pattern keeps showing up, and what actually works better for analytics.
Read Article →
The biggest blocker: treating ingested data as software state instead of cargo moved by ingestion pipelines
Read Article →
What makes a 10x data engineer
Read Article →