The Modern Data Engineer Part 12: Reaching Self-Service BI

Thirty customers, thirty semantic models

Early in my career I led the analytics team at a company that sold a multi-tenant SaaS analytics product. Every customer organisation got its own workspace and a shared Power BI semantic model, and their key people got accounts to connect to that model and build their own visuals. That was the self-service on offer, and for a handful of customers it held up fine. Somewhere around thirty customers it started to buckle.

The reports customers built off the semantic model did not agree with the reports we built off the same model. The definitions on our side had never been documented, and there was no manual worth the name, so when a customer’s number and our number disagreed, nobody could authoritatively say which one was right. Every disagreement became a support ticket. The analytics team and the customer success managers ended up spending hours every week crunching numbers by hand and answering the same question in a dozen forms: why is my report different from yours? We had handed customers query access and built ourselves a reconciliation help desk.

The raw model was open for a reason. Every customer had a different setup, which meant every customer wanted different visuals. Opening the semantic model felt like the only way to serve that variety without hand-building a bespoke report for each account. It was the pragmatic move at the time, and a modeled, certified layer would have made it unnecessary.

Onboarding showed what that pragmatism cost at scale. Bringing on a new customer meant cloning a semantic model, toggling the per-customer changes by hand, rebuilding a report to match the house style pixel by pixel, publishing it, granting access, and then troubleshooting why people still could not see the thing we had just published.

None of this was exotic. Disagreeing dashboards feeding an argument about whose number is correct is one of the most common ways a BI platform loses the trust of the people it serves, and here it was living in production, with support tickets attached.

The tool was fine. What was missing sat one layer below it, in what the customers were allowed to query and what those numbers were allowed to mean.

Self-service is earned at the Gold layer

Buying a self-service BI license does not create self-service, and neither does handing someone a semantic model to build on. The capability lives further down, in what you expose, at what grain, and with what guarantees behind it.

The war story sits between two failure poles. Expose raw or half-modeled tables and every user re-derives the business logic for themselves, then diverges from everyone else who did the same, which is exactly what happened when our customers built off the shared model. Expose only vendor-authored dashboards and there is no self-service to speak of, only a fixed report that someone else owns and the user has to queue behind. Gold is the layer between those poles, and reaching it is what everything the series has built so far was for. Bronze captured the source as it arrived. Silver conformed the entities. The contract kept the source honest and enforced. Gold is the layer where all of that discipline finally reaches a person who does not write SQL.

Zhamak Dehghani’s data-as-a-product characteristics name that standard, borrowed for a single dataset without taking on the decentralised organisation the rest of data mesh argues for [1]. A dataset earns the word self-service only when the consumer can find it without asking you, work out what its fields mean without a call, trust that its numbers carry one agreed definition, and reach it through whatever tool they already work in. The shared model we handed our customers failed most of that list at once. It was findable only in the sense that you could connect to it, none of it was documented, and its definitions were never conformed, so the moment two people queried it they got two answers.

Certified is what you call a dataset that clears that bar. It has an owner, a grain you can name, a definition written down, and a stability contract behind it. The owner and the layer it belongs to come straight from the per-layer ownership model in Part 4B and the governance argument in Part 5. The stability contract is the enforced contract from Part 10 and Part 11, which is what keeps certified from being a sticker anyone can print. The piece that matters most for the war story is the written-down definition. Its absence was the whole reason every disagreement turned into a ticket, because there was no authority to arbitrate against.

Certification needs raw material worth certifying, which is where the grain and the data-product framing come in, and where the fix the team eventually shipped actually landed.

The curated product that bought back trust

A Gold dataset is a business-facing product. It is modeled at the grain the question gets asked at, per customer, per order, per month, and named in the language the business uses, so the consumer never has to reassemble it from the normalized fact and dimension leftovers of the layer below. That is what the consumption end of the platform is supposed to hand over.

The fix we shipped was exactly that. Instead of thirty bespoke models exposed to thirty customers, the team built one simplified report covering the KPIs almost everyone actually asked for, backed by all the customer data together with row-level security so each customer saw only their own rows, and embedded it directly into the product’s admin page. It killed the reconciliation help desk. Everyone was finally reading one definition of each number, so there was nothing left to reconcile, and the trust came back.

That curated report was a fixed dashboard, and calling it self-service would be generous. It solved the trust problem and the bottleneck precisely because it took the query freedom away. Measured against the bar it cleared what it needed to, one agreed definition per number and delivery right where customers already worked, and it left discovery and independence unmet, because a fixed dashboard gives you nothing to explore and nothing to build on. It proved the value of a certified, conformed Gold product where each number meant one thing, and customers finally had a report they could rely on. The independence never came with it, though. They had swapped a freedom they could not use for a report that held together, which was the better deal at the time even if it was never where they needed to end up.

The discipline underneath that report matters more than the report itself. A Gold data product is a deliberate build, denormalised for the consumer, documented, owned, and versioned the way the contract in Part 10 is versioned. A view slapped over the Silver tables drifts the moment two people query it differently, because none of that discipline is built into it.

The team knew a fixed report was not the destination, so it tried to give the independence back properly.

The metric problem you cannot model your way out of

The next move was to raise a data API, so customers could build on the data themselves. The instinct was right, and it still did not reach self-service, because native access was the only part of the bar an API could clear. The datasets stayed hard to discover and undocumented, their definitions stayed unconformed, and the data itself was not yet mature enough to stand on its own.

Strip the specifics away and the failure is general. You can certify your datasets and expose a spotless API and still end up with revenue computed three different ways, because a metric is a definition that spans datasets. It lives across joins and business rules, so every place it gets redefined is a place it drifts.

This is the failure I have watched play out well beyond that one employer. Definitions end up living in a dozen places, the dashboards built on them disagree, trust erodes the same way it did over those support tickets, and eventually someone quietly exports to a spreadsheet and computes the number they trust by hand, while the BI tool takes the blame for a number that went wrong two layers below it.

The fix for the definition and documentation gap is a place to define each metric once, sitting above the datasets and below whatever consumes them, whether that is a BI tool or an API, so that every consumer resolves the same definition. That layer is the semantic layer. It is also where the documentation we never had would have lived, as the definitions the platform itself resolves against, kept current because the platform reads from them directly.

One gap stays open no matter how good that layer is, because no data layer can close it. Self-service quietly assumes a capable analyst on the other end, and the assumption stays invisible until you serve a real population with a real spread of skills, which on our customer side ran from very capable to barely present. Carrying that weight is the product’s responsibility. A semantic layer takes one bite out of it, because moving the definition of a metric out of the consumer’s head and into a governed layer means nobody has to be skilled enough to re-derive revenue the right way, they read the definition that is already there. It does not teach anyone to frame the question or read the answer.

That is the part AI is now reaching for. A data agent can take a question in plain language, turn it into SQL or DAX, run it, and hand the answer back as a sentence and a chart. That is a real attack on the framing-and-reading gap that stopped some of our customers cold. Point one of those agents at raw or unconformed data, though, and it industrialises the old problem, because now anyone can produce a confident wrong number in plain English, faster than the shared model ever allowed. It lowers the bar safely only when it resolves through conformed definitions. That is one more reason to build the layer that defines each metric once first. The agent layer itself is further out, in Part 14.

Self-service is earned at Gold, and it holds only when the dataset clears the whole bar, findable, documented, and resolving to one definition that everything reads from. Of those, the piece the series builds next is the layer that guarantees the single definition. Part 13 builds it.

References

[1] Z. Dehghani, Data Mesh: Delivering Data-Driven Value at Scale. Sebastopol, CA, USA: O’Reilly Media, 2022. [Online]. Available: https://www.oreilly.com/library/view/data-mesh/9781492092384/