You may not have guessed it, my name is Bram. One of the things I do is architect data platforms and share opinionated takes on how they should actually work. Here you'll find real insights for real problems, not the usual vendor narrative. Whether you're scaling platforms, making strategic decisions, or questioning your current approach, there's something here worth your time. Because data engineering is a craft.
Building scalable foundations using Cloud Native principles and modern data stacks.
Custom pipeline development and high-throughput stream processing systems.
Bridging the gap between raw data and business decisions through governance and AI readiness.
An opinionated blog designed to challenge decision makers, inspire data professionals, and spark meaningful discussions on the future of data engineering.
Every pipeline that loads a source more than once has to work out what changed since the last run, and each way of doing that misses some changes while the run still reports success. Which of the four load patterns fits your pipeline, and do you know the consequences of the one you picked?
At a previous client every source loaded through a stored procedure that assembled the query, and the parallel loop had been set to thread = rand_int() % 1 for longer than anyone knew. It kept running, which is why nobody fixed it. This is the design we built instead, in four objects and one run lifecycle.
Every BI project ends up with the same request: can we export this to Excel? Conversational agents are the first thing I have seen that gives people a reason not to, and only if the semantic model underneath them is in a state where the answer can be trusted.
For consulting inquiries or architectural engagements, please visit us at Xyntrel.