Technology / Databricks
A lakehouse designed for the full lifecycle.
Databricks is a unified lakehouse platform for large-scale data processing, streaming pipelines and machine learning, commonly organized into bronze, silver and gold data layers. Quorlytic defines the quality and ownership contract each layer actually protects rather than treating three layers as a checklist, and designs for replay and schema change from the start.
By QuorlyticUpdated
Connect ingestion, data processing and analytical consumption through a governed lakehouse architecture. Make the responsibilities of each layer explicit before adding workloads.
Architecture role
Where Databricks fits.
A Databricks lakehouse commonly spans ingestion, processing and model development, which makes layer contracts more important than the platform itself. The architecture decision is how much responsibility each layer carries and how replay and schema change are handled.
Position in the end-to-end path
- 01SourcesFiles, events, databases
- 02Ingest & streamingBatch, CDC and streaming
- 03DatabricksLakehouse processing and compute
- 04Models & productsCurated data and ML models
- 05Analytics & decisionsBI, applications and operations
Sources
- Batch files
- Streaming events
Lakehouse
- BronzeRaw, replayable
- SilverValidated, conformed
- GoldBusiness-level
- Delta tables open format · time travel
- Catalogue lineage · access · discovery
- Engineeringreads silver
- ML / AIreads silver
- BI & SQLreads gold
Illustrative architecture. It shows a typical position for discussion, not a mandatory stack or a client implementation.
Fit and capability
When it is relevant, and what the work involves.
Relevant workloads
- Large-scale processing
- Streaming pipelines
- ML training
- Batch engineering
Engineering capabilities
- Layer responsibilities and quality contracts
- Source replay and correction strategy
- Workload and access evaluation
- Deployment and operational handover
Integration points
What it connects to.
The platform is only as useful as the boundaries around it. These are the interfaces that usually need an explicit decision.
- 01Cloud object storage
- 02Streaming and ingestion
- 03Orchestration
- 04BI and ML tooling
Delivery and operating considerations
What to decide before you commit.
Each consideration changes something concrete in the architecture, the operating model or the cost profile.
- 01
Give each layer a contract
Databricks describes the medallion pattern as successive bronze, silver and gold layers. Use the pattern to distinguish retained source data, validated records and business-facing products. Naming three layers is not enough; define their quality and ownership boundaries.
- 02
Design for replay and schema change
Decide how source updates, late events and corrections move through the model. Make backfills repeatable and separate invalid records for review. Test business aggregates after replay, not merely whether the processing task completed.
- 03
Evaluate compute and governance together
Workload shape, permissions and team skills determine a suitable deployment. Compare processing and serving needs, inspect data access and track operating costs. Include test and development workloads in the estimate rather than pricing only the demonstration job.
- 04
Operationalize beyond the notebook
Version jobs and dependencies, use deployment reviews and provide monitoring and recovery instructions. Quorlytic does not claim official Databricks partner status. A scoped engineering assessment should establish fit before a wider migration.
- 05
Decide whether you need Spark at all
Databricks is a strong default when the work is genuinely large-scale, streaming or machine-learning heavy. When the workload is governed SQL reporting at modest volume, a warehouse is usually simpler and cheaper to operate. Keeping Spark for the jobs that need it, and moving the rest to SQL, is often the highest-value decision available.
- Workloads classified by processing need, not by preference
- Existing Spark code assessed before any rewrite
- Warehouse and lakehouse paths costed against each other
Typical delivery sequence
- 01Source / bronze
- 02Validated / silver
- 03Business / gold
- 04Analytics & models
Decision table
When Databricks fits, and when to compare.
| Dimension | Databricks is a strong fit | Compare alternatives when |
|---|---|---|
| Workload | Large-scale processing, streaming and machine learning on one copy of the data. | The work is mostly concurrent BI over modelled tables with light processing. |
| Data shapes | Semi-structured files, events and large volumes held in open table formats. | Data is structured, moderate in volume and already modelled. |
| Team | Engineers comfortable with Spark, Python, jobs and notebooks. | A SQL-led analytics team with little platform engineering capacity. |
| Governance | Unity Catalog can be the one place for permissions and lineage across workspaces. | Governance has to live inside a single warehouse with minimal assembly. |
| Operations | Someone owns file layout, compaction, cluster sizing and job tuning after launch. | Nobody will own that work once the build is finished. |
Choose a Hive metastore transition by constraint.
Hive metastore federation and table migration answer different transition needs. This hypothetical matrix starts with storage, identity and consumer dependencies before selecting an option. The AWS, Azure and GCP documentation is not identical; verify the workspace's cloud, region, table format, runtime and compute access mode before applying a procedure.
Scope and assumptions: Apply the matrix to a representative workspace and validate the selected runtime, storage and recovery path before making changes. It describes decision criteria, not a customer migration result.
| Dimension | Option and applicability | Evidence required before a transition |
|---|---|---|
| Federation | Consider a Unity Catalog catalog that mirrors the existing Hive metastore where supported. Treat this as a governance transition option, not proof that tables have moved into managed storage. | Confirm federation support and limitations for this metastore and cloud. Test allowed and denied access using the intended consumer identities. |
| External tables | The upgrade wizard and SYNC produce external Unity Catalog tables for eligible Hive tables. SYNC does not upgrade managed tables in DBFS root; inspect the actual storage location first. | Record format, path ownership, storage credentials, external locations and grants. Verify source/target metadata behavior and consumer reads before switching names. |
| Managed tables | Evaluate CLONE for eligible managed Hive Delta tables, or the documented alternative for the source format. Reserve the required Unity Catalog managed storage; do not treat an external-table sync as a managed-table migration. | Validate data and metadata differences, storage ownership and downstream dependencies. CLONE does not carry pre-migration table history into the new table. |
| Identities and jobs | Map workspace-local groups to account-level identities and review grants. Inventory qualified names, job principals, compute and file/script dependencies separately from tables. | Prove both permitted and denied operations and run representative jobs on supported compute. A successful table copy does not demonstrate a working workload. |
| UCX assessment | AWS and Azure guidance includes UCX for assessment and migration workflows. UCX is a Databricks Labs project, without formal Databricks support. The GCP options page differs; do not infer the same tool coverage. | Validate tool support, permissions and versions for the specific environment. The documented UCX code-migration workflow remains unavailable; dependent code still needs an explicit review plan. |
| Recovery boundary | Separate reverting a consumer reference from reversing writes or reconstructing table history. Keeping the original Hive table is not sufficient evidence of a complete rollback path. | Record the authoritative write path, replay/reconciliation method, recovery evidence and change approver before switching consumers or disabling legacy access. |
At narrow widths, scroll the worksheet to read every column. With the table focused, use the arrow keys to scroll.
Worked decision: suppose an inventory finds a managed Delta table in DBFS root and an external consumer that still depends on its current path. That rules out treating SYNC as a complete solution. The engineer must resolve the consumer and storage contract, compare eligible alternatives and rehearse recovery before choosing a migration path.
- Attach the documentation version checked, runtime version, compute access mode, cloud storage rules and required privileges to each object decision. This matrix deliberately supplies no universal minimum runtime or executable migration command.
- Keep source table history and audit evidence accessible for the approved retention period. Do not disable Hive access until dependency tests and the recovery decision are approved.
For the surrounding platform trade-offs, use the Snowflake and Databricks architecture comparison. The transition matrix remains specific to governance and workload compatibility.
Official references for this example:
Questions
Questions worth asking.
Do all workloads need three layers?
The medallion pattern is a useful organizational model, not a reason to create unnecessary copies. Decide what each stage protects or enables and simplify where requirements allow.
Is Databricks cheaper than a warehouse?
It depends on the workload and who operates it. For large-scale processing, streaming and machine learning that would otherwise need separate infrastructure, a lakehouse often costs less in total. For governed SQL reporting at modest volume, a warehouse is frequently simpler and cheaper. Compare the whole path, including engineering effort.
Integration ecosystem
What sits around Databricks.
Adjacent technologies grouped by the role they play next to this platform, and why that role usually needs an explicit decision.
Storage
- Cloud object storage
- Delta Lake
- Iceberg
- Parquet
The lakehouse is a table format over object storage, so file layout, compaction and retention are engineering work someone has to own.
Processing
- Spark
- SQL warehouses
- Structured Streaming
- Python
Batch, streaming and ML workloads run on the same data, which is the central argument for the architecture.
Orchestration
- Workflows
- Airflow
- dbt
- CI/CD
Layer promotion between raw, refined and curated needs declared dependencies rather than a schedule.
Machine learning
- MLflow
- Feature stores
- Model serving
- Notebooks
Training on the same governed tables analysts read removes the copy that usually separates analytics from ML.
Governance
- Unity Catalog-style catalogues
- Lineage
- Access control
- Audit
A lake without a catalogue becomes unsearchable quickly, and the catalogue is what makes access control possible at all.
Technology marks belong to their respective owners. No partnership, certification or endorsement is implied.
Migration and modernization
What moves, and what that asks of you.
Typical transitions. The lakehouse usually absorbs work from more than one existing system at once.
Data lake on object storageLakehouse tables
The most natural path: adding transactions, schema enforcement and time travel to files already in place. Compaction and small-file handling become explicit work.
Hadoop or on-premises SparkDatabricks
Job logic often transfers with moderate change; the harder part is re-establishing the catalogue, access model and cost attribution in a cloud consumption model.
Warehouse-only estateLakehouse alongside
Usually additive rather than replacing. Most estates end with both and a stated boundary about where business measures are defined.
Notebook-driven MLGoverned pipelines
Moving from a working notebook to a versioned, scheduled, monitored pipeline is the step where most ML work actually stalls.
Performance and economics
What drives the bill.
Cost is driven by cluster configuration and by how well the file layout supports the queries being run.
Attributing and reducing Databricks spend as an engagement, rather than a tuning exercise, is covered by data platform cost optimization.
- Cluster sizing
- Autoscaling bounds and termination timeouts are the most common source of avoidable spend in a lakehouse.
- Workload separation
- Job clusters for scheduled work and warehouses for interactive queries have different economics and should not be conflated.
- File layout
- Partitioning, compaction and file size decide how much data a query has to read, which is what actually drives compute cost.
- Storage
- Object storage is inexpensive; table history, orphaned files and uncompacted small files accumulate without a retention policy.
- Attribution
- Tagging clusters and jobs by workload is what makes cost a team-level decision rather than a platform-level total.
Governance and security
What has to be designed first.
A lakehouse gives more architectural freedom, which means more has to be decided explicitly.
- Catalogue
- A central catalogue with lineage is what keeps a lake discoverable. Without it, table sprawl arrives faster than anyone expects.
- Layer contracts
- Raw, refined and curated only mean something if promotion between them has agreed quality criteria and an owner.
- Schema evolution
- Enforced schemas with an explicit evolution policy, because permissive files and strict consumers is where silent breakage comes from.
- Access control
- Applied at the catalogue and table level rather than at the storage path, so a consumer cannot route around it by reading files.
- Retention and compaction
- Table history, vacuum and optimisation policies are both governance and cost, and they need a named owner per table.
Trade-offs
When it fits, and when it does not.
Where a lakehouse fits, and where it is more architecture than the problem requires.
Where Databricks tends to fit
- Large volumes of semi-structured or raw data that analytics and ML both need to read
- Estates where machine learning and data engineering share the same tables rather than copies
- Streaming and batch workloads that should run on one processing model
- Open table formats adopted as a deliberate portability position
Where another answer deserves consideration
- A warehouse such as SnowflakeThe workload is governed, concurrent SQL analytics with a SQL-first team. A lakehouse adds file-layout and compaction responsibilities that buy nothing in that case.
- A managed cloud warehouseThe team has no Spark or data engineering capacity. A lakehouse assumes engineering ownership; without it, the flexibility becomes unmanaged sprawl.
- A smaller, simpler platformData volumes are modest. Lakehouse architecture is a reasonable answer to scale and a heavy answer to a few hundred gigabytes.
- A dedicated streaming platformThe requirement is genuinely sub-second operational response rather than fast analytics.
Related architectures
Patterns this platform appears in.
References
Primary sources for this platform.
Vendor documentation, linked for verification. It is not an endorsement, partnership or certification.
Next step
Discuss your Databricks architecture.
Bring your layer boundaries and where schema change currently hurts. That is where lakehouse design decisions concentrate.