The Modern Data Engineer Part 11: Enforcing the Contract

The survey that nobody could keep up with

At a previous client, one of the more important sources feeding the warehouse was Qualtrics, used to run employee onboarding surveys. The Qualtrics API was painful to work with, and the export structure was worse: a pivoted table with hundreds of columns, where the data had to be unpacked through a tangle of export logic, and the structure of that export dictated how you parsed it. Column question_35 held the answer to “how would you rate your first two weeks of onboarding.” It held that meaning only because the thirty-fifth slot in the survey happened to be that question. Nothing in the column itself recorded it.

That is the detail that mattered, and it is the one that does not show up in any schema. The meaning lived in the position. question_35 was a stable column name with a stable type, an integer rating on a one-to-five scale, and underneath that stable shape sat a question that could be reworded or reordered at will. Move the question, swap it for another, change its wording so the responses shift, and question_35 quietly became a different question while every structural check kept passing clean.

The API gave you no escape from this. Qualtrics did expose an endpoint meant to solve exactly the mapping problem: one that returned the questions a survey asked, so you could tie each column back to its real question instead of trusting the slot it landed in. It never worked. Every call came back with a completely empty dataset, so the mapping it promised never arrived and column position was all you had. And even a working endpoint would not have been enough, because nothing in the API carried a survey version. A batch of responses arrived with no marker for which iteration of the survey produced it, so matching answers to the questions actually asked meant guessing whether a result belonged to the current survey, the version before it, or some state in between.

The survey owners were also the dashboard stakeholders, so the contact was good and the communication ran both ways. What none of that surfaced was how much downstream work their small survey edits set off. They reworked the onboarding survey often, responsive to the business and good at their jobs, and nothing in that work made it obvious that reordering a question would quietly break a parse keyed to slot thirty-five. I flagged it as a maintenance trap early and pushed for data contracts. They never came, and the tweaking has not stopped to this day. So the team did the only thing left to do: it rewrote dbt models, again and again, chasing survey changes that looked like small edits to the people making them. To them the data team could always work some magic on the numbers; nobody saw that the magic was a standing queue of dbt rewrites.

This is the same shape of problem Part 10 opened on, the asset-management export reconfigured through a GUI by people who did not know the warehouse read from it. The setup rhymes: a silent source change and no executed check to catch it. The owner rhymes too, though with a twist: this one knew the warehouse read from the survey and only missed how much rework each small change set off downstream. What is different is where the cost landed. In Part 10 the structural breaks at least produced an error you could see. Here nothing structural ever broke, so there was nothing to catch. The damage surfaced weeks later as a dashboard whose onboarding scores had quietly stopped meaning what the label said, and the day-to-day cost was a team that kept rewriting transforms to chase changes that looked harmless to the people making them.

The survey changing was never the painful part. The painful part was that no check looked at the values arriving, and no one had decided what the pipeline should do when one of them came back wrong. That layer is exactly what Part 10 left for this post.

The bill Part 10 left open

Part 10 ended on one distinction: ODCS, the Open Data Contract Standard, is a specification, not an enforcement engine. It defines the contract and what a valid one looks like, and it says nothing on its own about whether your data obeys it. Something else has to run the rules. A data-contract CLI, a quality tool like Soda, the test framework inside your transformation tool, a schema registry at the ingestion boundary: the contract names the check, one of those executes it.

Part 10 also went further than that single line. It already named the detector, a schema-diff or ODCS breaking-change check sitting at the ingestion boundary, and it already named the high-level response, which was to push the change back to the producer and pause ingestion, or to cut a new Bronze version when you have to live with it. A reader arriving here knows a check belongs at the boundary and knows the two coarse options for handling a breaking change.

What that reader does not have yet is the layer underneath. How the checks actually run, what each kind of check can and cannot see, and the full operational playbook for the moment one of them trips. That playbook is the heart of this post. The leverage question, whether you can refuse a third-party API the way you can lean on an internal team, Part 10 settled.

The rest of the post turns on one distinction. A structural break and a value-level break are different failures, and they call for different responses; most of the work of enforcement is matching the failure to the response.

The cleaner half of that frame is the structural check at the boundary, the one Part 10 already pointed at, so that is where to start.

Running the structural check at the boundary

The structural check is the schema-diff that Part 10 named, now made to actually run. It sits at the ingestion boundary, in front of Bronze, before any of the incoming payload enters the warehouse, and it answers a single narrow question. Does the shape that arrived still match the shape the contract declares? It compares the incoming columns, their types, and their required-or-not status against the contract’s schema section, and it fires when they diverge.

What it catches is Part 10’s breaking drift, the renames and removals and type changes. A schema registry holding the expected shape at the boundary and rejecting on mismatch is one way to run this. Validating the contract’s schema section with a data-contract CLI in the load step is another. Both are options. Neither is the way, and the same tooling-neutrality discipline the series has held since Part 7 applies here.

Here is the limit that makes this only half a system. A structural check passes on question_35 every single time, on the morning the survey changes and on every morning after. The column name held, the integer type held, the one-to-five scale held. The position drifted, and position is not a structural property. Structural validation is blind to meaning by construction, which is the same lesson Part 6 reached from the other direction: a row can be perfectly well-formed and still be the wrong reality.

Catching the survey change needs a check that reads the values the source sent, which is the value-level rule, and the one piece of enforcement the team in the story never had.

Running the value-level rules

Value-level rules inspect the data itself against expectations the contract declares. Where the schema section governs the shape, the quality section governs the contents: enum membership, ranges, freshness, distribution. This is the axis that has any hope of seeing semantic drift, because semantic drift leaves the shape untouched and only the values move.

It is tempting to reach for an accepted-values check here, and for question_35 it would be a mistake worth understanding. Pin the column to its valid domain, the integers one through five, and the check passes the morning the question changes, because a reworded question still returns answers on the same one-to-five scale. The domain never moved. What moves is the shape of the answers. When the question underneath slot thirty-five changes, the distribution of responses changes with it: a question that used to sit around a mean of four starts clustering somewhere else, or the spread across the five buckets shifts. A check pinned to the expected answer pattern is the one that notices that.

The same intent can be written three ways, and the point of showing all three is that the intent lives in the contract while the execution lives in a tool. The ODCS quality section states it as a contract rule:

# a quality rule (value-level: a distribution check on the answer pattern)
# question_35 has historically averaged ~4.1 on a 1-5 scale.
# a reworded question keeps the 1-5 domain but moves the mean.
- type: sql
  query: SELECT avg(question_35) FROM {object}
  mustBeBetween: [3.9, 4.3]
  dimension: accuracy
  severity: error
  schedule: 0 20 * * *
  scheduler: cron

Soda expresses the same expectation as a check it runs against the table:

checks for stg_onboarding_survey:
  - avg(question_35) between 3.9 and 4.3:
      name: question_35 response pattern within expected baseline

And a dbt test states it as an assertion on the model, here as a singular test that fails when the batch’s mean leaves the expected band:

-- tests/assert_question_35_distribution.sql
-- fails (returns rows) when the mean drifts outside the agreed baseline
select avg(question_35) as observed_mean
from {{ ref('stg_onboarding_survey') }}
where loaded_at = '{{ var("batch_date") }}'   -- scope to the incoming batch, not history
having avg(question_35) not between 3.9 and 4.3

Each of these is one option. The transformation tool’s own test framework is another, a standalone data-contract CLI is another still. None is prescribed, and none of these blocks needs to grow past this size. The weight of the post is in what happens when one of them trips, and the syntax that defines the check is the easy part.

Two real-world caveats keep these honest. The first is the window each check runs over, and it cuts both ways. Too wide, and a single bad morning is averaged into months of clean rows and slips through. Too narrow, and you get noise: a delta load that brings in three changed responses gives you an average over three answers, which is evidence of nothing. The window has to be small enough that a real shift is not diluted and large enough that the average means something, set with a where on the load timestamp, a rolling window, or a staged population, and gated behind a minimum sample size before the check can fire. The second caveat is the band. A fixed [3.9, 4.3] is only illustrative; in production you baseline the expected pattern and usually watch more than the mean, since a hardcoded average false-positives on a quiet week and misses drift that holds the mean while shifting the spread.

Tie this back to the survey. A distribution rule pinned to the expected answer pattern of question_35 is the one thing that would have noticed the morning the question changed, because the responses stopped looking like the question the contract expected. The structural check could never have seen it, the obvious accepted-values check would have waved it through, and only the value-level rule reading the answer pattern catches it.

So now both halves exist, a structural check on the shape and a value-level check on the contents. Detecting the problem is only the start. What the pipeline does in the seconds after a check fires is the part most teams skip, and where the survey story turns from a warning into a worked example.

What to do when a check trips

Start with the morning it would have caught. Had question_35 been pinned to that distribution rule, here is how the day plays out. The overnight batch lands, the check runs, the mean of the responses comes back outside the agreed band because the question underneath the column changed, and the check trips. The batch is held before it propagates, an alert reaches the person who owns the rule, and the broken onboarding scores never reach the dashboard. Compare that to what actually happened: the change surfaced three weeks later as a report whose numbers no longer meant what the axis label claimed. Everything below is what that morning would have looked like, beat by beat.

Before the responses, the layering has to be exact, because three commitments the series has made all meet at this point. The check sits at the ingestion boundary, in front of Bronze, so enforcement decides what enters the warehouse in the first place. What passes the boundary lands in Bronze, the first warehouse layer, exactly as the source sent it, and holds there immutably, the property Part 9 made GDPR erasure lean on. What fails never enters the warehouse at all; it is written to a separate quarantine bucket that sits outside it, where it stays visible until the trip is resolved and the rows can be replayed. Circuit-breaking pauses ingestion at the boundary, so nothing enters Bronze until the batch is cleared. None of this edits a record Bronze already holds; a tripped check keeps bad data out of the warehouse, and never rewrites what already landed.

The response splits along the same axis the whole post turns on, and the split is the actual engineering decision.

A structural break should fail fast and circuit-break ingestion at the boundary. A rename, a removal, or a type change has corrupted the shape itself, so there are no good rows to rescue and propagating the batch poisons every model that reads it. Halt at the boundary and hand the problem back to the producer, which is the Part 10 stance. That fail-fast halt is the immediate reflex, the thing that stops the broken batch from moving. Resolution is a second step. When the producer fixes the export to match the agreement again, ingestion resumes. When the producer cannot or will not revert, you cut a new Bronze version, the mechanic Part 10 worked through, and reconcile the versions downstream. The reflex and the resolution sit in sequence, so a reader of both posts ends up with one coherent answer instead of two defaults that seem to compete.

A value-level break should be strict by default: quarantine the entire ingested slice and let nothing from that slice enter Bronze until someone has looked. That may sound heavy when only a few rows trip the check, but a row that breaks a value rule is rarely just one bad row. An impossible date or a malformed record is a signal that the application producing the data is itself behaving incorrectly, and an application that can emit one corrupt record can emit others that satisfy every check you wrote. That is why the whole ingested slice is suspect, down to the rows that passed. A pipeline that keeps the rows that happened to pass and drops only the ones that failed turns a clear signal of upstream corruption into a half-trusted dataset, which is often worse than no data at all.

This is especially dangerous with aggregate rules like the accepted average in the survey example. The rule measures one thing: the distribution of the whole slice. Treat it as a row filter, dropping whichever responses pull the average out of band until it lands back inside, and you throw away the evidence the rule was reading. What lands looks clean and means nothing, because you have reshaped the distribution by hand until the average passes.

So the extractor writes its whole output into a staged quarantine area that sits in front of Bronze, and nothing proceeds until someone resolves the held input. The owner the trip paged is the one who makes that call, and there are three ways it can go. The first is that the held slice is fine and the check was simply too strict, or the change behind it is legitimate, so the owner accepts it and lets it proceed into Bronze. That acceptance is a forward append of the held slice into Bronze, the same append every clean slice makes, so the records already in Bronze are never touched and the immutability the post leans on holds. The second is that the change is real and ought to become the agreed baseline, so the owner moves the contract to the new expectation and replays the held input through the check that now passes. The third is that the data is genuinely bad, or the producer broke the agreement, in which case the held slice is discarded and handed back to the producer and nothing lands at all.

The guiding question is no longer whether the bad rows can be separated from the good rows. At the ingestion boundary, the safer question is whether the slice can still be trusted as a unit. If it can, accept the whole slice and append it to Bronze. If it cannot, reject the whole slice and keep the warehouse clean. Get that call wrong in the permissive direction and you let a suspicious dataset bleed into everything downstream. Get it wrong in the strict direction and you delay data that might have been usable, which is painful, but at least it is visible and recoverable.

An alert only helps if it reaches someone who can act on it. Sent into a channel nobody owns, the check fired and nothing happened. So the alert routes to the owner of the layer where the check runs, which is the per-layer ownership model from Part 4B and the governance argument from Part 5. A structural break at the ingestion boundary pages the source-aligned owner, the person whose job is the conversation with the producer. A value-level trip pages whoever owns that rule and the model it guards. The routing is part of the design, decided when the check is written, the same way you decide what the check measures.

All of this is the automated replacement for what the team in the story actually did. Constantly rewriting dbt models to chase silent changes was the manual, after-the-fact, discovered-in-production version of this runbook. The same decisions got made, eventually, by hand, at the worst possible time, by an engineer staring at a report that looked wrong. The runbook moves those decisions to the boundary and to the morning the change arrives.

All of this has to run without anyone remembering to run it. A runbook that depends on an engineer following the steps by hand gets skipped the first busy week, which is usually when the bad batch arrives.

Enforcement the pipeline carries

This is the same argument Part 5 made, now standing at the producer edge. Governance scales only when the pipeline produces and enforces it, because anything a person has to administer by hand decays the moment that person is busy or gone. The structural check, the value-level rules, the quarantine and circuit-break logic, the paging: all of it lives in the pipeline and in CI, so it fires whether or not anyone is watching, whether or not the engineer who wrote it still works there, whether or not it is a busy week.

Go back to why the survey kept breaking. The relationship was good and the warehouse was no secret, yet no executed check made a change visible the morning it happened. Good contact tells you about the changes people know are changes; it does nothing for the one nobody flags, because reordering a question never felt like breaking anything. The fix was never going to be the survey owners internalizing how much downstream rework a small edit set off, when nothing in their tooling showed it. That was the part the team kept waiting on, and it was never coming. The contract writes that dependency down, and the executed checks carry the enforcement a good working relationship cannot. The warehouse stays protected whether or not anyone remembers the parse is keyed to position.

Part 2 posed the question this whole arc has been answering. When the payload does not match, reject it, flag it, transform it, your choice. This post turned that open choice into a concrete taxonomy keyed to what broke, which is the structural-versus-value-level split the trip-response section worked through. The choice now lives in the contract, and the pipeline enforces it, so it no longer falls to whoever happens to be on call the morning a survey changes.

That is the through-line of the whole post. A contract that nothing runs against is documentation, and documentation did not stop the survey from breaking for months. Detection only earns its keep when there is a defined response behind it, and that response depends on whether the shape broke or the meaning did. The survey kept breaking because it had none of this: no contract pinning down the position dependency, no check reading the values, and no runbook waiting for the morning a question moved.

That closes the ingestion and governance arc the series has run since Part 2. The contract is enforced, and trips are handled. Part 12 turns toward the other end of the platform, where the consumption arc begins with self-service BI.

Where to start

If you are looking at your own ingestion and wondering whether you have any of this, audit one source rather than all of them. Pick the one that would hurt the most if it drifted silently, usually the source feeding the most dashboards, or the one a GUI lets a business team reshape without telling you. For that source, walk four questions:

• Is there a contract at all, or just an assumption in someone’s head about what the source sends?
• Is there an executed check at the ingestion boundary, or is the contract a document that nothing runs against?
• Does the check read both the shape and the values, so a column that keeps its type but changes its meaning still gets caught?
• When a check trips, is there a defined response and a named owner who gets paged, or does the problem surface weeks later as a dashboard no one trusts?

If the honest answer to most of those is no, you are where most teams are, and the way out is smaller than it looks: one contract, one boundary check, and one owner, on the source that matters most. Then repeat for the next one.

I would genuinely like to hear how others handle this, especially the judgment call on when to circuit-break a whole slice and when to let it through. If you have a war story, or you think I drew the line in the wrong place, come argue with me on LinkedIn.