Putting Genie inside your data engineering workflow

Pull-quote: “A question Genie cannot answer is a modelling ticket, not a Genie problem.”

Genie belongs in your data engineering workflow as an instrument on the pipeline, not as a feature you hand to executives once the pipeline is done. A Genie Agent is a domain-scoped natural language interface over a curated set of Unity Catalog tables, and what makes it useful to engineers is that it fails specifically. When Genie cannot answer a question the business actually asks, the fault is rarely the model. It is a column with no description, two dimensions that never got conformed, or a measure that exists only inside a BI tool.

That property turns Genie into a test for meaning. You already test row counts, nullability, and referential integrity. You probably do not test whether the model is intelligible to someone who did not build it, and that defect class survives every other check.

This post is for whoever owns a medallion pipeline and wants Genie in the delivery loop: during modelling, at the Silver to Gold gate, during on-call triage, and through a migration. If you have not stood one up yet, start with what a trustworthy Genie Agent actually needs. Databricks renamed the thing formerly called a Genie space to a Genie Agent in July 2026, with no change in capability. This post uses the current name, and notes space_id where the API still carries the old one.

What you will be able to do

• Read a Genie failure as a specific modelling defect and write the ticket it implies.
• Define Genie readiness as an acceptance gate on a Silver to Gold promotion, with checks a reviewer can verify in minutes.
• Run a benchmark question set as a regression test on semantics, and interpret its ratings.
• Triage a bad load in plain language, then promote the useful queries into monitors.
• Export a space configuration into version control and review it like code.

Why should Genie sit inside the pipeline instead of at the end of it?

Because a Genie Agent is a readable statement of what your model means, and reading that statement back is cheapest while the model is still soft. Once a Gold table has three downstream consumers, renaming amt_2 to net_revenue_usd stops being a naming decision and becomes a migration.

The loop is short. Model a subject area, register the tables in a space, ask the questions the business asks, and watch what breaks. Each break points at one of four causes: a name nobody outside the team understands, a missing description, a dimension that two facts join to differently, or a measure with no single definition in the lakehouse. All four are modelling defects, and all four are cheaper to fix now.

A space that answers business questions correctly is not a deliverable you build after the pipeline ships. It is evidence that the pipeline models the business. Databricks even caps the alternative: a space holds 100 instructions, where each example SQL query and each SQL function counts as one, and the entire general instructions block counts as one more. Spend them on reusable SQL rather than prose.

How does a Genie failure tell you your data model is ambiguous?

A Genie failure is a diagnostic because Genie has only what you gave it: table and column descriptions, example SQL, SQL functions, join specifications, and reusable filters, expressions, and measures. When it guesses wrong, read backwards from the bad SQL to the missing context. Genie is non-deterministic, so treat one wrong answer as a lead and a repeatable one as a defect.

What Genie does	What it usually means	The modelling fix
Picks the wrong column for a word like revenue	Two columns are plausible and neither is described	Describe both, exclude the one nobody should use
Joins on the wrong key, or invents a join	No join specification exists, or the grain is stated nowhere	Declare the join with its cardinality, conform the dimension
Cannot compute a metric the business names daily	The measure lives in a dashboard, not the lakehouse	Define it once as a metric view or SQL function
Answers correctly but filters differently each time	The business rule is implicit	Register it as a named filter with synonyms

The last two rows are the valuable ones. A measure that exists only in a BI tool is invisible to every other consumer, including the next tool you buy, so moving it into a metric view or a Unity Catalog SQL function is work you owed the platform anyway. Choosing that approach deliberately is the subject of data architecture as a fit decision.

What does Genie readiness look like as a definition of done?

Genie readiness is five checks a reviewer verifies before a Silver table is promoted to Gold, and it belongs in your promotion template beside the data quality expectations. The point is to refuse to promote a table whose meaning is undocumented.

1. The benchmark set passes. Questions covering what this table should answer, each with a SQL answer supplied, no unexplained failures in the latest run. A space holds up to 500 benchmark questions, and Databricks recommends two to four phrasings of each, because users do not phrase things the way you do.
2. Every exposed column has a description. What it measures, at what grain, in what unit, and what it excludes. Columns nobody should query are excluded from the space, not left to be guessed at.
3. Every measure is defined once. As a metric view or SQL function the space references. If the same SUM appears in three example queries, it is not a measure yet.
4. No answer requires tribal knowledge. Someone who did not build the table reads the generated SQL for three benchmark questions. If they cannot judge it without asking the author, the gate fails.
5. Joins are declared, not inferred. Every join Genie is expected to make has an explicit specification with a relationship type, because cardinality living only in the author’s head produces fan-out silently.

Reviewing that takes about ten minutes. A chat-mode benchmark run scores a question Good when the generated SQL matches your answer, when the result set matches, when the same rows return in a different order, or when numbers agree to four significant digits. It scores Bad on an error, an empty result, extra columns, or a different single-cell value. Everything else needs manual review. Capture the run summary at promotion, because response detail is retained for one week while the SQL persists.

How do you use Genie for pipeline triage and data quality investigation?

Point a space at your operational and quality tables, not just your business tables, and an on-call engineer can explore a bad night in plain language instead of recalling column names at 3am. This is the use that converts sceptical engineers fastest, because the questions are ones they already ask.

Build a separate space for it, curated over the tables that describe your pipeline rather than your business: the load audit table, data quality results, the pipeline event log published to Unity Catalog, and a partition-level row count history if you keep one. Describe those columns as carefully as you describe revenue, because an unreliable triage answer is worst exactly when someone is under pressure.

Then close the loop, because this is the part teams skip. Triage produces a query that found something real, and that query is a monitor you do not have yet. Review the SQL and promote it into a scheduled check with a threshold. A Genie response also offers an Add as instruction action that opens the example SQL editor prefilled with the question and generated query, so good queries improve the space too.

How do you check freshness and reconcile a migration with Genie?

Genie suits reconciliation work during a migration because parity checking is a long tail of one-off comparisons, and those are what nobody wants to write SQL for. Load the legacy report output as a table, register it alongside the new Gold table, and describe both, including which is authoritative and for what period.

Then ask directly: revenue by month from each source side by side, rows present in one and absent from the other, accounts disagreeing by more than a rounding difference. Freshness questions work the same way against the load audit table, asking the newest event date per source and which sources are behind contract. Two rules keep this honest. The parity check that matters becomes a persistent reconciliation query with a tolerance, and a result you cannot reproduce is not a result, so keep the generated SQL with the migration record.

What should you not use Genie for?

Do not use Genie where the same question must return an identical number every time and no governed measure sits behind it. Genie generates SQL, and generation varies. Example SQL and defined measures make it far more consistent, but consistency is not determinism. If a number appears in a filing, an invoice, a regulatory submission, or a contractual report, it comes from a parameterised query or a certified metric view, under change control.

The second exclusion is any question where an approximate answer is dangerous rather than merely wrong: a dosing question, a safety threshold, a payment amount, an eligibility determination. A plausible wrong answer in those domains costs more than no answer, and a chat interface gives you nowhere natural to put the refusal. Genie respects Unity Catalog permissions, so nobody reads what they were not granted, but permission is not the same as suitability.

Who owns the Genie Agent, and how do you review it like code?

The team that owns the Gold tables owns the space, and its configuration lives in the repository holding the pipeline that produces them. A space owned by nobody drifts within a quarter: instructions accumulate, example SQL contradicts itself, and accuracy nobody measures declines. In practice the pipeline team holds CAN MANAGE, instruction changes go through review, and the benchmark set is maintained as a test suite.

Version control is practical because the configuration serialises. Request a space with its serialised representation included and you get its instructions, joins, and benchmarks as one document. Create or update a space from that representation and you have a diff and a promotion path between workspaces.

# Export a space configuration for review and version control
curl -s -H "Authorization: Bearer $DATABRICKS_TOKEN" \
  "https://$DATABRICKS_HOST/api/2.0/genie/spaces/$SPACE_ID?include_serialized_space=true" \
  | jq -r '.serialized_space' | jq . > genie/sales_gold.space.json

Commit that file next to the pipeline and an instruction change arrives as a diff a reviewer can argue with. Three schema rules bite the first time: identifiers are 32-character lowercase hexadecimal strings, several collections must be pre-sorted by their documented sort key or the payload is rejected, and each join specification takes exactly two SQL elements, the condition and a relationship type annotation.

Then run a weekly review, because the space tells you what the business wants and your backlog usually does not. The Monitor tab lists every question and response, filterable by time, rating, user, and status, with a weekly digest of message volume, active users, and feedback. That feedback changes nothing on its own, which is the point: a human reads it and decides. Sort the failed and flagged questions into three buckets: fix the space, fix the model, or decline as out of scope. Bucket two is your backlog, and anything you fix earns a benchmark question.

Where this goes wrong

Symptom: the space passes its benchmarks and fails real questions. The benchmark set was written by the people who built the tables, in the vocabulary of the tables. Source questions from monitoring history instead.

Symptom: instructions grow to thousands of words and accuracy gets worse. Instructions and metadata consume the space’s token budget, and Genie filters context when it runs short, so the detail you needed drops out silently. A warning appears as you approach the limit, and past it the space stops accepting messages. That warning is your checkpoint.

Symptom: Genie returns a number that disagrees with the dashboard. Nine times in ten the dashboard has a filter the model does not, because someone excluded internal accounts or test orders inside a BI tool. Find it, register it in the lakehouse, and pin the number with a benchmark question. This is a governance finding, not a Genie bug.

Symptom: an answer was right last month and is wrong now. Something changed upstream and no test caught it: a column repurposed, a join gone many-to-many, a source sending a new enumerated value. The benchmark run is your detector, so schedule it and treat falling accuracy as a pipeline alert.

Common questions

Does Genie replace the semantic layer or the BI tool?

No. Genie is a natural language interface over governed data, and it depends on a semantic layer rather than substituting for one. Metric views and Unity Catalog SQL functions are where measures live, and Genie references them. Dashboards stay right for a fixed set of numbers watched repeatedly.

How many tables should a Genie Agent contain?

Fewer than you want, and no more than 25, the documented limit on tables a space keeps in context. A space works best over one subject area owned by one team, because that is the scope where you can honestly claim every column is described.

What permissions does this work need?

CAN MANAGE on the space to change configuration, read the full monitoring history, and delete conversations. CAN EDIT to run benchmarks and save corrected SQL as an example query. Every user also needs the Databricks SQL entitlement and at least CAN USE on a SQL pro or serverless warehouse, because the space runs its queries there.

Is asking Genie an audited action?

Yes. Databricks records Genie Agent activity in workspace audit logs, and the monitoring tab retains the questions and responses asked in the space. Unity Catalog lineage still applies to the underlying tables, so a question that reads a table is attributable to a user.

Next steps

Start with one subject area you already own. Stand up a space over its Gold tables, write ten benchmark questions from real requests you have received, run them, and fix what fails. Then add the gate: put the five readiness checks in your promotion template and export the space configuration into the pipeline repository.

When people ask the space things without checking with you first, it is ready to be a product surface, which is a different problem: authentication, cost, ambiguity, and low-confidence answers. Building applications and agents on Genie covers the integration surfaces and the confidence contract behind them.

Zorost runs this on the lakehouse as a trusted Databricks partner, where parity testing against the legacy report is part of the migration plan rather than an afterthought.