Data architecture patterns: choosing by fit, not by fashion
Pull-quote: “Most teams that reached for a data vault needed a Silver layer and two conformed dimensions.”
Data architecture patterns are not five competing options at the same level of abstraction, and treating them that way is how estates end up with machinery nobody can operate. Medallion describes how data moves. Dimensional modelling describes how a person queries it. A data vault describes how you absorb source change and reconstruct the past. Data mesh describes who is accountable. Lakehouse federation describes whether you copy the data at all.
Once the patterns are separated by the question each answers, the design conversation changes. You stop asking which one to adopt and start asking which constraints you actually have. This post is for the engineer or architect who has to defend a choice to people who read a different conference talk than you did.
By the end you will be able to describe each pattern in one sentence, say what it costs, and run a short procedure that points at a composition rather than a favourite.
What you will be able to do
- Define medallion, dimensional, data vault, data mesh, and lakehouse federation, and say what each is for.
- Name what each pattern costs you, so the tradeoff is explicit instead of discovered in year two.
- Recognise the signature failure of each pattern when it is applied to the wrong problem.
- Answer six constraint questions that point at a composition of patterns rather than one label.
- Explain why most mid-size estates need medallion plus dimensional marts and nothing more exotic.
Why do data architecture patterns get adopted for the wrong reasons?
Because a pattern arrives with a reputation attached, and reputation travels faster than the conditions that made it work. Someone reads that a large retailer runs a data mesh, or that a bank runs a data vault, and the pattern gets adopted without the constraint that justified it. The retailer had forty domain teams. The bank had two hundred source systems and a regulator who asks for numbers as of a past date.
The second reason is that patterns are usually presented as complete architectures, so choosing one feels like choosing everything. It is not. A working design normally composes two or three: a processing topology, a consumption model, and sometimes a historisation strategy. Asking which pattern to use is a category error, and it produces arguments that cannot be settled because the participants are answering different questions.
What does each pattern actually optimise for?
Each pattern buys one property and charges for it somewhere else. The charge is the part that gets left out of the conference talk.
| Pattern | What it optimises for | What it costs you | Signature failure when misapplied |
|---|---|---|---|
| Medallion | Replay from raw, isolated failures, one place to clean | Storage duplication, more pipelines, latency per hop | Bronze to Gold with no real Silver |
| Dimensional | Comprehension, one metric definition, BI performance | Modelling labour, surrogate keys, SCD machinery | Marts with no conformed dimensions |
| Data vault | Absorbing source change, as-of reconstruction, audit | Table count, join depth, a mandatory presentation layer | A vault nobody built marts on top of |
| Data mesh | Removing the central team as a delivery bottleneck | Platform investment first, engineering capacity per domain | Silos with a new vocabulary |
| Federation | Time to first query, no pipeline, one governance surface | Remote performance, load on the source, no history | A daily dashboard against production OLTP |
When is medallion right, and when is it just folder naming?
Medallion is right when you ingest from files or streams and need a boundary you can replay from. It becomes folder naming when the layers have no contract.
Medallion architecture, also called multi-hop, is a processing topology with three layers. Bronze holds raw data at source fidelity, append-only, never mutated. Silver holds validated data: deduplicated, typed, with a declared key and enforced expectations. Gold holds business-facing tables, which is where aggregates and marts live. The value is that a defect has one home. When a source sends malformed records, you fix Silver and reprocess from Bronze without asking the source system for a resend.
The contract is what makes it real. Bronze is immutable and never queried by consumers. Silver has a primary key, a documented grain, and quality expectations that fail the pipeline rather than passing bad rows downstream. Gold owns business definitions. Skip Silver and you get the failure mode from the Databricks cost optimization post: several Gold jobs each re-reading raw Bronze, each re-implementing the same cleaning, and definitions that quietly diverge.
Why do dimensional marts still matter on a lakehouse?
Because dimensional modelling is the only pattern here designed around how a person asks a question, and cheap columnar compute did not change that.
Dimensional modelling organises data into fact tables, which hold measurements at one declared grain, surrounded by conformed dimensions, which are the shared descriptive entities every fact refers to. Slowly changing dimension handling decides whether an attribute change overwrites history (Type 1) or creates a new version with validity dates (Type 2). Conformed means one customer dimension used by every fact table, not one per mart.
The performance argument for star schemas has largely gone: modern engines handle wide denormalised tables well. The comprehension argument has not. An analyst who can name the grain of a fact table and pick dimensions to slice by does not need to understand your ingestion. That is what self-service actually requires, and it is why Gold is usually dimensional even when nothing else in the stack is.
When do you need a data vault, and when is it overhead?
A data vault earns its cost when you have many source systems you do not control and a requirement to reconstruct the past exactly. Outside those conditions it is expensive machinery.
Data Vault 2.0 splits data into three structures, all insert-only. Hubs hold business keys. Links hold relationships between keys. Satellites hold descriptive attributes with load timestamps, so a change appends a row rather than updating one. Nothing is ever overwritten, which is what makes as-of reconstruction possible: you can rebuild what the data said on any past date because every version is still there.
The costs are structural. Table count multiplies, so a question that was one join becomes many. Nobody queries a vault directly, which means a presentation layer is mandatory rather than optional, and generation tooling is close to mandatory too. The signature failure is a vault on a small estate: the team spends its budget on hub, link, and satellite plumbing, never reaches the marts, and eighteen months in nobody can answer a business question.
Is data mesh an architecture or an operating model?
Data mesh is an operating model. It changes who is accountable for data, not how tables are shaped, which is why it can sit on top of any of the patterns above.
The four principles are domain ownership of data, data as a product with a contract and an SLA, a self-serve platform so domains do not each build plumbing, and federated computational governance, meaning rules enforced by the platform rather than by a review board. Unity Catalog can carry parts of the governance and discovery, but the ownership decision is organisational and no tool makes it for you.
Mesh addresses one bottleneck: a central data team that cannot keep up with many domains that have independent roadmaps. If your constraint is data volume, or pipeline reliability, or unclear semantics, mesh does not touch it. The signature failure is declaring a mesh without funding the platform or staffing the domains, which produces the silos you started with plus a new vocabulary. A mesh with one data team is one team with more meetings.
When should you federate instead of ingest?
Federate when the cost of copying exceeds the cost of querying remotely, which in practice means exploration, low query frequency, or a source you are not permitted to duplicate.
Lakehouse federation exposes external systems as foreign catalogs in Unity Catalog, so you query a PostgreSQL, SQL Server, or cloud warehouse table with the same SQL and the same permissions as a Delta table, without building a pipeline. It is excellent for the first look at a source, for a one-off join during an investigation, and as a bridge during a migration when both systems are live.
What you do not get is a pipeline’s guarantees. Performance depends on the remote system and on which predicates push down to it, so a query that scans a large remote table will be slow and will land that load on someone’s operational database. You get no historisation, so yesterday’s value is gone if the source overwrote it. Schema changes arrive without warning. The signature failure is federation as a permanent serving path for a dashboard that refreshes all day against production OLTP.
How do you actually decide?
Answer six questions about your constraints. The answers point at a composition, and in most cases the composition is smaller than the one you were considering.
- How many source systems feed this estate, and do you control their schemas? A handful you control means medallion is sufficient. Dozens you do not control is the condition where insert-only historisation starts to pay.
- How volatile are those sources? If schemas change several times a year and you cannot negotiate, a structure that absorbs change without downstream refactoring is worth its table count.
- How many independent teams own data, and can each staff engineering? One team means mesh is overhead. Several teams with capacity, blocked on a central queue, means mesh is addressing a real constraint.
- What is the audit requirement, precisely? “Show the number and where it came from” is satisfied by medallion plus lineage. “Reproduce this report exactly as it appeared last March” requires as-of history, which is vault territory or very carefully versioned Silver.
- How fast must a new attribute reach a consumer? Days is ordinary pipeline work. Hours, for exploration only, is what federation is for.
- Do you need history reconstruction or only current state? Current state means Type 1 dimensions and no vault. As-was reporting means Type 2 dimensions at minimum.
Written down, the answers usually point somewhere unglamorous. Most mid-size estates need medallion for processing, dimensional marts for consumption, and nothing more exotic. Federation covers exploration. A vault enters only when source volatility or as-of audit demands it, and mesh enters only when team count rather than data volume is the constraint.
Where this goes wrong
Bronze straight to Gold. The symptom is several Gold jobs reading raw tables and producing slightly different answers to the same question. The cause is that Silver was treated as optional. Build one Silver table with a declared key and expectations, and repoint Gold at it.
A vault with no presentation layer. The symptom is a technically correct vault that no analyst can query, so people extract to spreadsheets. The cause is budgeting the raw vault and not the marts. A vault is only half an architecture.
Mesh declared, platform unfunded. The symptom is domain teams each building their own ingestion, with no shared catalog or contract. The cause is adopting the ownership principle without the self-serve platform principle. Ship the platform first, then devolve ownership.
Federation on the critical path. The symptom is a dashboard that is slow at month end and a database administrator asking who is hammering production. The cause is treating a query bridge as a serving layer. Ingest the tables that get read repeatedly.
Marts with no conformed dimensions. The symptom is two reports disagreeing about customer count and both being defensible. The cause is each mart defining its own dimension. Conform the dimension once, in Gold, and make every fact table reference it.
Common questions
Can I use medallion and dimensional modelling together?
Yes, and that combination is the normal answer rather than a compromise. Medallion decides where cleaning happens and how you reprocess; dimensional modelling decides what the business-facing layer looks like. In practice Gold is where your fact tables and conformed dimensions live, fed by Silver.
Is data vault obsolete now that storage is cheap?
No, but its case is narrow. Cheap storage removed one objection to the table count without changing the two things a vault is genuinely good at: absorbing schema change from sources you do not control, and reconstructing what the data said on a past date. If neither is a real requirement, you are paying for insurance against a risk you do not carry.
How many medallion layers should I have?
Three, unless you can name the specific contract a fourth layer would enforce. A proposed platinum layer usually means Gold was never actually business-facing, in which case the fix is to define Gold properly rather than to add a hop.
Does data mesh require specific technology?
No. It requires domain ownership, product contracts, a self-serve platform, and governance enforced by the platform. Unity Catalog can carry the catalog, permissions, and lineage parts, but no product decides who owns a data product; that is an organisational commitment.
Where does a semantic layer fit in this?
On top of Gold, once more than one tool consumes your marts. A semantic layer gives a measure one definition that every consumer inherits, which matters most when dashboards, notebooks, and natural-language interfaces are all asking the same questions of the same tables.
Next steps
Write your six answers down before you design anything, in a document other people can argue with. Most architecture disagreements are really disagreements about constraints, and they resolve quickly once the constraints are written where both parties can see them.
Then read Genie and answers you can defend, because a natural-language interface is the strictest test of whether your Gold layer really has one definition per measure. If cost is the pressure behind your redesign, where Databricks spend actually goes explains why a missing Silver layer is the most expensive line on the bill. More engineering writing sits in the Signals index.
Zorost runs these compositions on client lakehouses as a trusted Databricks partner, where parity testing against the legacy report is part of the migration rather than an afterthought, because a new architecture that returns different numbers is not a migration.