03 / Data modernization
Data Modernization & Cloud Data Migration
Data modernization is the process of moving a data warehouse or platform to new infrastructure without losing the business meaning of the data it holds. Quorlytic maps dependencies, proves the target platform on a representative workload, reconciles old and new outputs, and executes a phased, reversible cutover.
By QuorlyticUpdated
Modernize the architecture without losing the meaning of your data. A successful migration moves workloads, preserves business behavior and gives your team a platform it can operate.
Legacy
- Silos
- Manual ETL
- Legacy database
- Duplicate reporting
- Fragmented governance
Modernization
- Map
- Prove
- Reconcile
- Cut over
Modern
- Cloud platform
- Automated pipelines
- Governed data
- Shared semantic layer
- Analytics + AI
Parallel run until old and new outputs reconcile — every cutover reversible
The business problem
Inventory source systems, scheduled jobs, stored procedures, reports and downstream consumers. Find undocumented logic and business-critical reporting windows. The migration unit should be an end-to-end data product, not an arbitrary set of tables.
What we build
Select a representative workload and evaluate performance, access, cost and operability on the proposed platform. Snowflake, Databricks and BigQuery offer different implementation choices; existing cloud agreements and team skills belong in the evaluation.
The outcome
Dependency map and target architecture
When this is the right engagement
Symptoms that bring teams here.
Modernization is worth doing when the platform is the constraint. These symptoms distinguish a platform problem from a pipeline problem.
The nightly window no longer fits the night
Batch duration has grown past the time available, and adding hardware has stopped helping.
Cost rises without a corresponding rise in use
Workloads share capacity with no attribution, so nobody can see which one is expensive.
Business logic lives in stored procedures nobody will touch
Undocumented logic has become a dependency, which is the real migration risk.
A licence renewal or end-of-support date is forcing the question
The deadline is external, so the plan has to be reversible rather than optimistic.
New workloads cannot be added without affecting reporting
There is no workload isolation, so every new consumer is a negotiation.
Nobody knows the full set of downstream consumers
Discovery, not the move itself, is the first phase — and usually the longest.
Capability model
Current state to target state.
A migration is four decisions and one discipline. Select a stage to see what it involves and where migrations usually fail.
01 — Discover
Establish what actually depends on the current platform, including the parts nobody documented.
What it produces
A dependency map and a consumer inventory with owners.
What usually goes wrong
- Reports and extracts built outside the platform team
- Stored procedures holding business rules
- Scheduled jobs with undeclared dependencies
- Consumers who will only surface when something breaks
Technologies in this stage
- Dependency analysis
- Query log mining
- Lineage capture
02 — Prove
Test the target platform on a representative workload before committing the estate to it.
What it produces
A measured result on your workload: performance, access, cost and operability.
What usually goes wrong
- Benchmarks chosen to flatter the target
- Cost modelled on list price rather than real concurrency
- Operability judged by the people who chose the platform
Technologies in this stage
- Snowflake
- Databricks
- BigQuery
- Workload replay
03 — Translate
Move logic rather than transcribe it, taking the opportunity to remove what is no longer used.
What it produces
Translated, tested logic with deliberate exclusions recorded.
What usually goes wrong
- Dialect differences in null handling, precision and date arithmetic
- Procedural logic that has no direct equivalent
- The temptation to redesign everything mid-migration
Technologies in this stage
- dbt
- SQL translation
- Spark
- Automated conversion
04 — Reconcile
Run old and new in parallel and prove the business behaviour is unchanged before anyone is asked to trust it.
What it produces
A reconciliation report and an agreed release gate.
What usually goes wrong
- Comparing totals but not distributions
- Time zone and rounding differences dismissed as noise
- No agreed threshold for what blocks a cutover
- Late-arriving data making a fair comparison harder
Technologies in this stage
- Parallel run
- Reconciliation tests
- Sampling
05 — Cut over and retire
Move consumers in phases with a rollback path, then decommission deliberately rather than by neglect.
What it produces
Migrated consumers, a tested rollback and a dated retirement plan.
What usually goes wrong
- Rollback discussed but never tested
- Dual running extended indefinitely because nobody owns the decision
- Sources decommissioned because the job succeeded, not because the consumer signed off
Technologies in this stage
- Phased cutover
- Rollback plan
- Retention policy
Illustrative reference architecture
Reference architecture
The shape of a reversible migration. Parallel running is drawn as a stage rather than a footnote, because it is the only phase that proves the move is safe.
Legacy estate
The current platform, its jobs and its undocumented logic.
- Warehouse
- ETL jobs
- Stored procedures
- Extracts
Discovery
What depends on it, including consumers nobody listed.
- Dependency map
- Query logs
- Consumer inventory
Target platform
Proven on a representative workload before the estate follows.
- Snowflake
- Databricks
- BigQuery
Parallel run
Both platforms producing the same outputs for an agreed period.
- Counts
- Aggregates
- Record samples
- Key metrics
Cutover
Consumers moved in phases, each with a rollback path.
- Phased release
- Rollback triggers
- Consumer sign-off
Retirement
Deliberate decommission with retention satisfied.
- Retention
- Archive
- Licence exit
Across every stage
- Reconciliation evidence
- Rollback readiness
- Consumer communication
- Cost of dual running
- Sign-off record
The approach
- 01
Understand what depends on the current system
Inventory source systems, scheduled jobs, stored procedures, reports and downstream consumers. Find undocumented logic and business-critical reporting windows. The migration unit should be an end-to-end data product, not an arbitrary set of tables.
- 02
Prove the target before scaling the move
Select a representative workload and evaluate performance, access, cost and operability on the proposed platform. Snowflake, Databricks and BigQuery offer different implementation choices; existing cloud agreements and team skills belong in the evaluation.
- 03
Reconcile before cutover
Run old and new pipelines in parallel for an agreed period. Compare counts, aggregates, record samples and important business metrics. Test late-arriving data, time zones, null handling and decimal precision. Define which mismatches block release.
- Legacy warehouse and ETL modernization
- ERP and CRM integration with documented dependencies
- Phased migration, data validation and operational handover
- 04
Retire deliberately
A migration ends when the old platform can safely be retired. Agree rollback triggers, retention needs, consumer sign-off and the cost of dual running. Do not decommission a source merely because the target job completed successfully.
- 05
Warehouse, lakehouse or both
The decision belongs to the consumer, not the vendor. Governed concurrent reporting suits a warehouse; large volumes of semi-structured data, feature engineering and exploratory work suit a lakehouse; many estates run both with a stated boundary and one place where business measures are defined. Open table formats reduce lock-in, and they also create compaction and retention work.
- Consumers and concurrency profiled before the target is chosen
- One place named where business measures are defined
- Storage, compaction and retention ownership agreed
- 06
Migration economics include what you keep running
The cost of a migration is not only the target platform. Dual running, retained history, egress, duplicated licences, retraining and the operating effort of two estates belong in the business case. Comparing list prices per terabyte answers a different question from comparing the cost of the full data path.
- Dual-running window and its cost estimated
- Retention and archive decisions priced
- Operating effort counted alongside licences
Delivery model
How the work is sequenced.
Sequenced so that the reversible decisions happen first and the irreversible one happens last.
| Phase | Activities | Decisions made here | What you keep |
|---|---|---|---|
| AssessMap the estate and the constraint. |
|
| Dependency map and migration scope |
| ArchitectProve the target, then design the move. |
|
| Platform evaluation and migration design |
| BuildTranslate and reconcile in slices. |
|
| Translated pipelines and a reconciliation report |
| OperateCut over, then retire on purpose. |
|
| Cutover record, rollback evidence and retirement plan |
Technology fit
What each technology is here to do.
Migration technology choices are constrained by the source as much as the target. These are the roles involved.
Source estate
- Teradata
- SQL Server
- Netezza
- Oracle
- Redshift
The source dialect and its procedural logic determine most of the translation effort.
Target candidates
- Snowflake
- Databricks
- BigQuery
Chosen on measured workload evidence, existing cloud agreements and the skills of the team who will run it.
Platform detailTranslation
- dbt
- SQL conversion
- Spark
- Python
Moving logic into a tested, version-controlled form rather than transcribing it as-is.
Validation
- Reconciliation tests
- Parallel run
- Sampling
- Metric comparison
The evidence that business behaviour is unchanged. Without it, cutover is a matter of opinion.
Cutover control
- Phased release
- Rollback plan
- Access migration
- Retention policy
Making the move reversible until the point it deliberately is not.
Technology marks belong to their respective owners. No partnership, certification or endorsement is implied.
Engineering concerns
What has to be answered before it is trusted.
The concerns that decide whether a migration is a controlled change or an outage with a project plan.
- Reconciliation evidence
- Counts, aggregates, samples and key business metrics compared, with agreed thresholds for what blocks release.
- Rollback
- A tested path back, not a documented intention. Rollback that has never been exercised is not a rollback.
- Access migration
- Roles, row-level rules and service identities moved and tested before consumers, not after.
- Retention and archive
- Legal and operational retention satisfied on the legacy platform before decommission is scheduled.
- Cost of dual running
- Parallel running costs money every day it continues, so it needs a dated end and an owner.
- Consumer communication
- Every consumer knows what changes, when, and what to do if their number looks wrong.
Where it applies
Related work, and why it is related.
Where modernization is the right starting point, and what it connects to.
- Snowflake migrationWhen Snowflake is the chosen target, the source-specific translation and reconciliation work is covered in depth there.
- SnowflakePlatform architecture, workload isolation and cost model, for evaluating it as a target.
- DatabricksLakehouse architecture and layer contracts, for estates with large volumes or machine-learning workloads.
- Financial servicesReconciliation and explainability requirements make the parallel-run phase longer and non-negotiable.
- LogisticsOperational reporting windows constrain when a cutover can happen at all.
Choose a migration wave by dependency and consequence.
The smallest dataset is not always the safest first move. In this hypothetical planning exercise, a reporting estate contains a product dashboard, a shared customer dimension and a finance close extract. The worksheet separates a candidate that can be isolated from one that merely looks easy to copy.
Scope and assumptions: These are proposed planning decisions, not a committed migration schedule. Confirm dependencies with system owners and execution evidence; a diagram alone cannot establish that two consumers can be separated safely.
| Dimension | Decision to record | Evidence required before scheduling |
|---|---|---|
| Product dashboard | Consider a bounded pilot if its source snapshot and consumer can be isolated. Keep its existing reporting path available during evaluation. | Dashboard owner identifies the accepted measures, refresh dependency and a test consumer. Confirm that the pilot does not change a shared production dataset. |
| Shared customer dimension | Treat dependent marts as a group until an interface permits separate moves. Do not schedule each table independently because it has a different owner. | Map joins, identity rules and update order. Test how old and new consumers interpret the same customer changes, including deleted or merged identities. |
| Finance close extract | Place the business calendar ahead of dataset size. Hold the cutover proposal until the close owner agrees to the validation and recovery window. | Record blackout dates, accepted business-day rules, downstream deliveries and the authority that can stop or reverse the transition. |
| Temporary coexistence | Define which system supplies each consumer while both platforms run. Avoid an undocumented mix of old and new metric definitions. | List synchronization direction, reconciliation owner, stale-data detection and the condition for returning each consumer to its original source. |
| Wave closure | Separate consumer acceptance from permission to retire the old path. An apparently successful dashboard does not establish that every dependency has moved. | Attach signed consumer checks, unresolved exceptions and an inventory of remaining readers. Require separate retirement approval and an agreed retention decision. |
At narrow widths, scroll the worksheet to read every column. With the table focused, use the arrow keys to scroll.
Worked decision: the dashboard is the proposed first candidate only if dependency checks confirm isolation. If it writes back to the shared customer dimension, that assumption fails: regroup the affected consumers or establish a tested interface before scheduling. The finance extract remains outside that wave until its business-calendar constraints are resolved.
- Record why each candidate was included, held or grouped. Unknown dependencies are an open decision, not evidence of independence.
- Revisit the wave record after rehearsal. A discovered consumer, unavailable recovery path or changed business date can invalidate the original sequence.
Once the group is bounded, use the warehouse migration validation and cutover playbook to define acceptance evidence for its consumers.
Official references for this example:
Questions
Questions worth asking.
Is a full platform replacement necessary?
Not always. A phased improvement to ingestion, modeling or compute may solve the constraint. Compare the cost of remaining, incrementally improving and replacing the platform before committing.
Can you guarantee zero downtime?
Downtime requirements must be designed and tested for the specific system. The scope should state allowed interruption, recovery objectives and rollback criteria; no universal guarantee is appropriate.
How do we decide between a warehouse and a lakehouse?
Start with the consumers: which queries run concurrently, over what data shapes, with which refresh requirement, and who defines the business measures. A warehouse is usually the simpler answer for governed reporting; a lakehouse earns its place when the same data also feeds machine learning or large semi-structured volumes. If both are needed, state the boundary and keep one definition layer.
Next step
Review your modernization roadmap.
Bring the current platform, its consumers and the constraint you are trying to remove.