What does the source estate actually contain?
Inventory the source distribution and versions, storage paths, Hive metadata, Spark jobs, libraries, schedulers, security policies and consuming systems. Group these by the decision or data product they serve. The word Hadoop does not identify one migration unit or establish that every component has the same target.
For each workload, record its entry point, execution identity, runtime, file/table formats, input/output locations, schedule and external dependencies. Identify jobs that rely on a particular local file layout, native library, network path or service outside Spark. Those details often decide whether code is portable or needs a different implementation.
Use an AWS target as the bounded reference here: object storage, a selected Databricks runtime, governed tables and named job/consumer identities. Other clouds have their own identity and storage controls. The Databricks platform page gives destination context without turning this example into a whole-estate product compatibility claim.
- 01
Adapt a represented job
Spark code and dependencies fit the selected target execution model.
Compare output under named runtimes, libraries and identities.Example: a scheduled Spark transformation with explicit file/table inputs.
- 02
Move a data reference
Storage/table ownership and consumers need a governed target path.
Specify location, format, metadata and effective-access checks.Example: a Hive-backed report whose target reference is a governed table.
- 03
Retain or replace separately
A non-Spark service or external dependency has another responsibility.
Name its retained owner or selected replacement and evidence.Example: an HBase serving dependency outside the Spark migration scope.
Illustrative AWS/Databricks decisions. Matching platform names or a successful notebook does not disposition every Hadoop service.
How can a working job still expose the wrong data?
Test effective access separately from job execution. The pipeline identity may need to write a restricted table while an analyst may read only an authorised result. A successful job run proves neither that the analyst's permitted query works nor that the analyst's restricted path is denied.
The illustrative matrix below uses two invented identities and two data products. A pipeline principal can publish both products; an analyst can read the aggregate and must be denied the restricted detail. Run the checks using the identities and paths that the application will actually use, including any direct storage path relevant to the architecture.
Databricks' SDK reference distinguishes direct grants from effective permissions that include inherited privileges. Capture that context when assessing the target. Recreating a familiar role name does not establish policy equivalence, and a table-level denial is incomplete evidence if another permitted path exposes the same restricted records.
Illustrative identities: pipeline principal publishes data; analyst reads the permitted aggregate and is denied restricted detail.
- 01
Pipeline principal - 02
Analyst - 03
Aggregate product - 04
Restricted detail
| Dimension | Aggregate product | Restricted detail |
|---|---|---|
| Pipeline principal | Publish allowed | Publish allowed |
| Analyst | Read allowed | Read denied |
Check the actual principal and access path. Include inherited privileges and any direct storage route relevant to the architecture.
Which responsibilities should be retained, adapted or replaced?
Give every workload responsibility a disposition and a reason. Retain a component when it continues to have a defined role; adapt a workload when its behavior can be preserved with a bounded change; replace a responsibility with an explicitly selected component; exclude it when it is outside the migration scope.
The modernization service can frame the dependency and operating-model assessment. Keep the disposition precise enough that another engineer can identify what remains on the source after a given consumer moves.
Keep the responsibilities that a tool does not represent visible in the disposition record. Databricks Labs' UCX describes assistance for Unity Catalog table/view migration and downstream references inside Databricks; that is a narrower scope than moving a complete Cloudera estate. Its stated support boundary also matters. Assess any assistant against the source distribution, target cloud/runtime and represented components rather than extending a catalogue tool into an unsupported whole-estate promise.
| Dimension | Decision boundary | Evidence to retain |
|---|---|---|
| Spark code and runtime | Adapt only the code, dependencies and execution model represented by the workload. | Source/target runtime versions, libraries, configuration and equivalent output fixtures. |
| Hive and storage | Choose table ownership, location, format and the target catalogue path explicitly. | Metadata, file/table assumptions, update behavior and tested consumer references. |
| Scheduling and publication | Assign dependencies, retries, secrets and consumer publication to actual components. | A complete run/failure path rather than a notebook-only demonstration. |
| Ranger/Sentry policy intent | Translate the intended access boundary into target identities and controls. | Allowed and denied cases, inherited privileges and any alternate storage route. |
| Other Hadoop services | Give HBase, specialised serving and other non-Spark dependencies their own disposition. | An explicit retained/replacement/excluded scope; no inferred automatic substitute. |
What should runtime and data compatibility checks cover?
Compare a representative workload under named source and target conditions. Check schemas, key relationships, decimal/time handling, file/table assumptions and the behavior of updates or late data. Keep the code revision and dependency environment with the observed output.
A Spark application can rely on libraries, configuration, paths or external systems that are absent from a new job environment. Include those dependencies in the fixture and record which are replaced or retained. A notebook that processes a small sample is useful development evidence, but it does not establish the scheduled workload's publication window or production access path.
Select batch or change-driven movement from the consumer's actual freshness requirement. The CDC, batch and streaming guide helps separate those choices. If the source workload has no stable row key or change history, make that constraint visible before promising a continuous incremental path.
The handoff map follows storage/catalogue, runtime/schedule and identity/access as separate lanes. Each needs a named target and an accepted consumer boundary. Include direct storage routes only where they exist in the selected architecture, and retain the effective identities used to observe them. Use the data contracts guide to connect the governed dataset with its intended consumers, freshness and operating responsibilities.
- Source workloadPaths and owners recorded
Retain source table/file formats and metadata references.
- Candidate targetTarget ownership specified
Choose storage, table/catalogue path and lifecycle responsibility.
- Consumer handoffReferences accounted for
Identify readers, writers and retained source consumers.
- Source workloadJob environment fixed
Keep libraries, entry point, identity and external dependencies.
- Candidate targetScheduled result compared
Check data, refresh and failure behavior under the selected runtime.
- Consumer handoffOperating owner named
Keep the schedule, dependencies and restore boundary with the product.
- Source workloadPolicy intent recorded
Specify allowed and denied populations for actual identities.
- Candidate targetEffective paths checked
Include inherited privileges and applicable direct storage routes.
- Consumer handoffConsumer reference moved
Accept the permitted result and the denied alternate path together.
Illustrative handoff. Catalogue access and storage access are architecture-specific; this map does not assert automatic policy translation or tested enforcement.
What defines a safe consumer handoff?
Carry the data product, its allowed consumers and its publication boundary together. Compare accepted source and target results, verify the target's effective permissions and define the operating owner before moving the consumer reference. Shared storage or metadata dependencies can bind several jobs into one wave.
The workflow below pairs workload disposition with data and permission evidence. Retain the actual identities used for each allowed and denied case. If the target catalogue, inherited policy or source correction changes, revisit the affected cases rather than treating the first successful read as permanent access acceptance.
Record the last recoverable data publication, the source history needed to resume, and any consumer action a restore cannot undo. The pipeline reliability guide provides the publication and restored-state frame. Keep managed table ownership, storage lifecycle and external consumers in that recovery scope.
- 01Inventory
Identify storage, metadata, code, scheduling and policy dependencies.
Keep: distribution/runtime versions and workload ownership.
- 02Disposition
Give every responsibility a retain, adapt, replace or exclude decision.
Keep: rationale and any component left on the source.
- 03Compare
Check equivalent data and the permitted/denied consumer paths.
Keep: runtime context and effective access observations.
- 04Handoff
Move the consumer reference with an operating and recovery owner.
Keep: publication, acknowledgement and restore boundary.
An unresolved non-Spark dependency or restricted access path stays attached to its workload. Do not infer its acceptance from another job's successful run.
How should capacity, cost and run ownership be assessed?
Measure the complete workload path: data movement, storage, runtime, schedule, concurrency and the people or functions that operate it. Compare equivalent accepted outputs and operating conditions. A platform price or a single Spark job duration cannot establish the cost of a full Cloudera exit.
Include source/target overlap, retained services, network constraints, dependency management and incident handling in the assessment. Test whether the selected target job and access model can be operated with the actual run capacity available. An architecture that silently requires an additional operating team is missing a material part of the comparison.
Use the managed data and AI service for responsibilities and exclusions, and the platform cost assessment for bounded cost evidence. Neither a synthetic fixture nor a source inventory establishes a savings result.
Estimate the source responsibilities that remain after the first wave as well as the target workload. Retained serving services, storage, security operations and network dependencies can change the full-exit economics. Keep those residual responsibilities attached to an owner. The boundary pack below makes inherited access, alternate storage and a runtime dependency concrete without implying that this guide executed those checks or validated a selected cluster version.
Illustrative acceptance pack for the bounded AWS design. The expected outcomes are specifications to test with the actual environment and identities.
- 01
Inherited privilege
Input / conditionAnalyst has no direct table grant but inherits a parent privilege
Expected contractEvaluate effective access, not the direct-grant list alone
Evidence to retainKeep parent context and an actual allowed/denied query observation.
- 02
Alternate storage path
Input / conditionRestricted detail is reachable outside the intended table path
Expected contractThe declared restricted population stays denied on applicable paths
Evidence to retainInclude storage identity/routes in the architecture's permission evidence.
- 03
Missing runtime dependency
Input / conditionNotebook sample succeeds; scheduled job needs a native library
Expected contractThe job environment explicitly supplies or replaces that dependency
Evidence to retainRetain versioned libraries, job configuration and scheduled-output comparisons.
No live policy or cluster was tested. A catalogue permission result does not by itself describe every storage or execution path.
What should the first Cloudera migration brief include?
Provide a versioned workload inventory, a named target cloud, representative code and table references, the source policy intent, accepted consumer outputs and the freshness requirement. Include at least one non-Spark dependency and one denied-access case so the assessment covers more than code portability.
Share synthetic identities and appropriately permitted samples. Mark each dependency with its proposed disposition and identify the evidence still needed to confirm it. A concrete consumer boundary makes the initial assessment useful even when the wider estate contains components outside the selected target design.
If workflow conversion is also part of the programme, read the PowerCenter to dbt guide. For a warehouse consumer, use the warehouse migration playbook. Keep those responsibilities connected without presenting them as one universal conversion mechanism.
Questions and trade-offs
Does a working Spark job establish a complete Cloudera migration?
It establishes only the behavior represented by that job and its tested conditions. Storage, Hive metadata, scheduling, identity, non-Spark services and downstream consumers need explicit dispositions and evidence.
Include the scheduled job and its real dependencies, not only an interactive development environment. Retain the distribution/runtime/library versions and accepted data product. The wider estate may still need retained serving, storage or security responsibilities after that one job moves.
Can source policy names simply be recreated in Unity Catalog?
The intended allow/deny boundary must be expressed and tested with the actual target identities and paths. Record effective inherited privileges and alternate storage routes. Matching role names alone does not establish equivalent enforcement.
A direct grant list can omit inherited privileges. Use the target's effective permission context and the applicable storage/execution routes with actual consumer identities. The intended restricted population must remain bounded wherever the architecture permits a consumer to reach it.
Does this guide cover every Hadoop service?
It provides a workload-disposition method and a bounded AWS/Databricks reference. HBase, specialised serving and other components need their own retain, replace or exclude decision. No automatic replacement is inferred from Spark compatibility.
Keep a separate disposition for services outside the represented runtime and catalogue scope. A platform feature or migration assistant should be assessed against the actual component and versions selected, not treated as an automatic replacement for the estate's entire service inventory.
What is the most useful first assessment sample?
Choose a real consumer boundary with representative code, data, scheduling and access requirements. Include a meaningful edge case and an explicit denied path. That sample reveals the dependencies a successful notebook-only demonstration can miss.
Retain the permitted result, a denied access case, the scheduled run environment and the source dependency disposition together. That sample can support a bounded implementation decision without implying that every service or policy in the estate has been migrated.
What is the difference between a Cloudera migration and a Unity Catalog upgrade?
A Cloudera estate can include storage, Hive, Spark, schedulers, policy systems and non-Spark serving. A Unity Catalog transition concerns a selected Databricks data-governance and reference boundary. Databricks Labs' UCX describes assistance for tables/views and downstream Databricks references; it does not establish a whole Cloudera exit.
Keep tool scope and support terms with the workload disposition. Choose evidence for the actual source components, target cloud/runtime and consumers instead of extending one catalogue demonstration into a full-estate claim.
Can HDFS files and Hive tables simply be copied?
Data movement is one responsibility. A usable target also needs explicit storage ownership, table/file format assumptions, catalogue references, update behavior, consumers and access controls. Capture the source metadata and the actual paths each reader and writer uses.
Compare data results and governed access through the selected target reference. A copied directory or recreated table name alone does not establish equivalent lifecycle, permissions, refresh behavior or downstream compatibility.
What determines the migration's cost and sequence?
Group jobs by data products and shared storage, catalogue, runtime, schedule and access dependencies. Estimate adaptation, data movement, retained services, target operation, comparison evidence and recovery separately. Source/target overlap and a service left behind can materially affect the exit boundary.
Use representative observed workloads and record exclusions. A notebook benchmark, cluster count or platform price cannot establish full-estate savings or a fixed project duration.
When can the source cluster be retired?
Retirement needs an accounted-for disposition for every in-scope reader, writer, service and operating responsibility. Confirm that consumer references, retained history, recovery needs, access boundaries and non-Spark dependencies no longer rely on the retiring component.
Keep the agreed retention and recovery evidence before changing that dependency. The first accepted Spark job or a catalogue migration can be a useful wave milestone; it is not by itself evidence that the whole source estate is ready to retire.
