The Modern Data Engineer Part 13: One Definition, Every Consumer

Three numbers and a clean investigation

A previous client’s leadership had three versions of the same number in front of them. The metric was active customers, and it led the board deck due at the end of the day. One dashboard showed one figure. The finance export that fed the deck showed another. A partner-facing API reported a third. The three were close enough that nobody had caught it for months and far enough apart that, side by side, no rounding reconciled them. The question landed at 4 PM, in a Slack message: where does this number come from?

I ran the investigation the governance playbook prescribes, expecting the usual break upstream, a glossary out of sync with a model. The glossary defined active customer precisely: at least one completed order in the last 90 days, excluding trial accounts. The Conformed Entity in Silver implemented exactly that, tested, versioned, owned. The Gold mart, certified the way Part 12 describes, aggregated it without reinterpreting it. Every link in the Part 5 chain held.

The numbers forked past Gold, inside the consumers. The dashboard computed active customers in a DAX measure its author had written by hand. The finance export ran its own SQL against the same certified table. The API resolved the metric in query code a backend team maintained. Three people had made three slightly different choices about the same definition. One measured 90 days from the report run date, another from the start of the month. One excluded trials by a status flag, another by an account-type join that missed a category.

Every control in that investigation governs up to the table. The glossary describes the table, the Conformed Entity builds it, the tests and the certification vouch for it. The metric is computed after the table, in whatever tool reads it, and no control reaches that far. The investigation came back clean because everything it could examine sat upstream of the fork.

The warehouse has two semantic edges

Meaning crosses the warehouse boundary twice. On the way in, source records become business entities. Part 4A designed that edge and Part 4B put it into operation: conformed entities in Silver, where what counts as a customer and a completed order is committed once, in code, and flows into Gold. That edge existed at this client, and it held: glossary, entity, and mart agreed.

On the way out, a consumer asks a question, and the answer is a computation. The Conformed Entity hands every consumer the same ingredients: which customer, which order, whether it completed, when. A Gold column is a value you can read off the table, while “active customers by region, monthly” exists only at query time, assembled from an aggregation, a window, and a set of filters. Whoever assembles it is defining the metric, whatever their job title says, and at this client that was a dashboard author, an export script, and a backend team, independently. Part 4B watched exactly this, one definition of active customer living in three places at once, drifting because lineage stopped at the warehouse boundary.

The industry calls this exit layer a semantic layer. This series has already used that term for the producer side, because the two layers perform the same job at opposite edges of the warehouse: one commits meaning, the other resolves it. Exit edges are decades old, but each was bound to its tool: Business Objects universes and SSAS Tabular were authoritative inside their own stacks and never for the consumers outside them, the three-places problem in institutional form. The entrance eventually got one shared layer; the exit never did, which is why a certified table still yields three numbers.

Why one more Gold table does not fix it

The obvious fix is one more Gold table. Materialize gold.active_customers_by_region, certify it, point the dashboard at it, and the dashboard is fixed. It stays fixed until the second cut arrives. Finance wants the figure monthly instead of as of today. Product wants a trailing 30-day version. Someone asks for it by signup cohort. Metrics multiply against dimensions, grains, and windows, and each combination becomes its own table with its own copy of the 90-day rule baked into its build logic. When the business moves the window from 90 days to 60, someone has to edit every one of those tables and hope they found them all, the war story’s drift rebuilt inside Gold.

Materializing a metric caches one resolution of it, the by-region cut as of the build schedule, and caching is often worth doing for performance. The definition still has no single home: the 90-day window lives in as many places as there are tables and measures that embed it. The fix has to be a place where the computation itself is written down once and every shape of the question resolves from it.

The resolver at the exit

At the exit, that written-down computation is a metric definition:

metric: active_customers
description: customers with at least one completed order in the last 90 days, excluding trials
aggregation: count(distinct customer_id)
source: gold.customer_orders
filters:
  - order_status = 'completed'
  - account_type <> 'trial'
  - order_date >= current_date - interval 90 day
dimensions: [region, signup_month]

This is the only place the 90-day window, the trial exclusion, and the completed-order rule are written down, and it mirrors what the Conformed Entity did at the entrance: the definition in code, owned, versioned, reviewed. A resolver combines that definition with a consumer’s requested dimensions, filters, and grain to produce the query. The dashboard asks for active customers by region and receives:

select
  region,
  count(distinct customer_id) as active_customers
from gold.customer_orders
where order_status = 'completed'
  and account_type <> 'trial'
  and order_date >= current_date - interval '90' day
group by region

The finance export asks for one monthly figure and the API asks for its own shape, and both inherit the same three filters from the same definition. Nobody types the window twice. Move it from 90 days to 60 in the definition and every consumer’s query changes on the next resolution, with no DAX measure or export script edited by hand.

For this to work, the resolver has to belong to no consumer. A definition trapped inside one BI tool is the Business Objects universe again, authoritative for that tool and invisible to the API, so the property that matters is headless, tool-independent resolution. dbt’s Semantic Layer and Cube are built as this kind of headless resolver, with Cube fully tool-independent. LookML aims at the same centralization but is headless to a lesser degree, because its definitions still resolve through a Looker instance even as it opens up to outside tools.

Who may define a number

The tools are not what has kept the exit edge open; headless resolvers have existed for years. Closing the edge is an authority decision. Making the definition above authoritative means telling the analyst closest to the question that they may not define a number locally, and they are exactly the person who can build this quarter’s slightly different cut in their own tool in five minutes. Organizations defer that decision because deferring is free today, and the cost surfaces later, at 4 PM, in a meeting where three numbers disagree. Buying the tool is often how the decision gets avoided: a semantic layer license on the invoice looks like the problem being handled while every consumer keeps its local measures. The layer holds only as the sanctioned path, the Part 5 discipline carried to the exit: definitions in code, changed through review, and every consumer required to resolve through them.

Some conflicts are real: Finance genuinely needs month-end active customers while Product needs a rolling 30-day version. Those are two metrics sharing a word, each deserving its own name and single definition in the layer. The layer forces one definition per name. The analyst’s quarterly cut is the same case: legitimate, and it belongs in the layer under its own name instead of in a local measure, which turns the authority decision from prohibition into routing.

One thing stays out of the definition file: who may see the result. A consumer that routes around a metric definition produces a wrong number you can catch and reconcile the way this client eventually did, while a consumer that routes around an access rule produces a breach nobody can take back, so access has to be enforced at a warehouse chokepoint no consumer can bypass. Deriving those grants from identity attributes and data classification is Part 14. And the least supervised consumer of all, the agent that turns a plain-language question into a query, resolves through whichever exit edge it finds, which is where Part 15 picks up.