Engineering perspectives
Insights are original engineering perspectives on data platforms, analytics and enterprise AI — written as decision guides rather than announcements. Each article states its assumptions, trade-offs and technical references, and links to the capability that delivers the work, so a reader can evaluate the approach before committing to it.
Original engineering perspectives on data platforms, analytics and enterprise AI — with explicit assumptions, trade-offs and technical references.
Task contract · Source contract · Permission test · Release & rollback
Featured perspective
A practical framework for AI-ready enterprise data: source ownership, permissions, freshness, retrieval evaluation and safe operational handover.
The library
37 published guides
Architecture guideAI Systems
AI-Ready Enterprise Data Architecture
A practical framework for AI-ready enterprise data: source ownership, permissions, freshness, retrieval evaluation and safe operational handover.
14 min read
Engineering guidePlatform Engineering
Plan warehouse migration waves with dependency mapping, business reconciliation, dual running, rollback and retirement acceptance criteria.
12 min read
ComparisonPlatform Engineering
Compare Snowflake and Databricks on workload fit, architecture, operating model, skills, governance, cost and migration implications.
10 min read
ComparisonPlatform Engineering
Data Lake vs Warehouse vs Lakehouse
Compare data lake, data warehouse and lakehouse on workload fit, architecture, governance, cost, skills and migration implications.
9 min read
ComparisonData Architecture
Compare ETL and ELT on where transformation runs, latency, cost model, skills, governance, schema-change handling and replay.
8 min read
ComparisonPlatform Engineering
Compare Databricks and BigQuery on processing model, concurrency, governance, cost model, skills and migration implications.
8 min read
ComparisonAnalytics & BI
Compare Power BI, Tableau and Looker on modelling approach, governance, licensing, skills, delivery model and migration implications.
10 min read
ComparisonAnalytics & BI
Data Analytics vs Data Science
Compare data analytics and data science on the questions they answer, skills, tooling, governance and where each creates business value.
8 min read
Engineering guideData Architecture
What a data pipeline is, how batch, streaming and change data capture differ, and what production pipelines must guarantee to be trusted.
8 min read
Engineering guideData Architecture
Data Orchestration: Airflow and dbt
How orchestration differs from scheduling, where Airflow and dbt fit, and who owns dependencies, backfills, alerts and recovery.
8 min read
Engineering guidePlatform Engineering
Warehouse layers, grain, marts and semantic models: how to design a warehouse that answers the same question the same way every time.
9 min read
Analytics guideAnalytics & BI
What Is Business Intelligence?
What business intelligence is, how it differs from reporting and data science, and what a governed decision layer actually requires.
8 min read
Analytics guideOperations
Semantic Models and Metric Governance
How semantic models and metric governance stop teams reporting different numbers, and what a metric definition must state to be usable.
8 min read
Analytics guideAnalytics & BI
Predictive Analytics: From Signal to Action
How predictive analytics becomes a decision: model families, label quality, evaluation, intervention design and monitoring in production.
9 min read
AI guideAI Systems
What retrieval-augmented generation is: the retrieval pipeline, chunking and indexing choices, permissions, citations, evaluation and failure modes.
9 min read
AI guideAI Systems
Generative AI in the Enterprise
How enterprises choose generative AI tasks: workflow fit, data readiness, evaluation, permissions, cost, failure modes and human review.
9 min read
AI guideAI Systems
AI Agents: When Autonomy Works
Where AI agents help, how they differ from workflows and RAG, and how to bound tool permissions, approvals, audit and rollback.
8 min read
Operations guideAI Systems
MLOps: Operating Models in Production
MLOps operating models: model release paths, data and feature validation, evaluation gates, monitoring, drift, rollback and incident ownership.
9 min read
ComparisonAI Systems
AI vs Machine Learning vs Deep Learning
How artificial intelligence, machine learning and deep learning relate, what each needs in practice, and which questions belong to which layer.
8 min read
Decision guideAI Systems
RAG vs fine-tuning: which problem each one actually solves, what they cost to run and keep current, and why most enterprise systems need both.
12 min read
ComparisonData Architecture
Compare batch extraction, log-based change data capture and event streaming on freshness, source load, deletes, replay and schema change.
13 min read
Engineering guideOperations
How to make a data pipeline dependable: freshness objectives, the checks that detect failure, containment that protects readers, and safe replay.
12 min read
Architecture guidePlatform Engineering
What bronze, silver and gold layers each owe, when the medallion pattern earns three copies of the data, and the failure modes that make it expensive instead.
12 min read
Architecture guidePlatform Engineering
Where Snowflake credits actually go — warehouse runtime, serverless features, storage and time travel — and the architectural changes that reduce each one.
12 min read
ComparisonOperations
Data Observability vs Data Quality
How data quality and data observability differ: stated rules versus learned behaviour, what each catches, and why neither replaces the other.
11 min read
Engineering guideOperations
How to design and enforce a data contract: what belongs in one, which compatibility rule to choose, and how to deprecate a field safely.
12 min read
Operations guideOperations
Data Governance Operating Model
How to assign data governance decision rights across a central function, domain owners and the platform, and why federated models work.
12 min read
Governance guideAI Systems
How to govern the corpus a retrieval system reads: approved sources, entitlement at query time, classification, deletion, index freshness and audit.
13 min read
Evaluation guideAI Systems
How to measure a retrieval system: separating retrieval from answer quality, testing refusal on unanswerable questions, and the metrics that mislead.
15 min read
Analytics guideData Architecture
A diagnostic sequence for reconciling two dashboards that disagree: grain, filters, time basis, identity, refresh state and definition drift.
13 min read
ComparisonData Architecture
Which forecast accuracy metric to use and when: how MAPE, WAPE, RMSE and MAE behave on low-volume items, outliers and portfolio-level reporting.
12 min read
ComparisonData Architecture
How attribution and incrementality differ, what each can and cannot establish, and when a holdout or geo experiment is the only credible answer.
13 min read
Engineering guidePlatform Engineering
Teradata to Snowflake Migration Guide
Scope a Teradata to Snowflake migration with duplicate handling, session semantics, batch dependencies, reconciliation and consumer cutover checks.
15 min read
Engineering guidePlatform Engineering
Oracle to Snowflake Migration Guide
Plan an Oracle to Snowflake warehouse migration around PL/SQL dependencies, null handling, extraction consistency, Exadata context and consumer parity.
15 min read
Engineering guidePlatform Engineering
Informatica PowerCenter to dbt Migration Guide
Plan an Informatica PowerCenter to dbt migration with mapping decomposition, incremental state, deletes, reject handling and orchestration boundaries.
15 min read
Engineering guidePlatform Engineering
Cloudera to Databricks Migration Guide
Assess a Cloudera to Databricks migration across Spark, Hive, storage, scheduling and access, with workload dispositions and a permission matrix.
15 min read
Engineering guidePlatform Engineering
Netezza to Snowflake Migration Guide
Plan a Netezza to Snowflake migration with SQL and extension inventory, numeric transport, rounding fixtures, workload comparison and consumer cutover.
15 min read
Dependency inventory · Reconciliation matrix · Cutover criteria · Retired deliberately
Featured perspective
Engineering guide · Platform Engineering
Plan warehouse migration waves with dependency mapping, business reconciliation, dual running, rollback and retirement acceptance criteria.
Read the guideKnowledge domains
Data architecture, platform engineering, analytics, AI systems and operations — each connected to the capability that delivers it.
From perspective to implementation
Questions
Each guide is written and reviewed by Deepesh Patel, Founder & Data Engineer. Guides state their assumptions, trade-offs and technical references, and vendor facts link to the vendor's own documentation.
No. Worked examples are hypothetical and labelled as such. Client work is published separately as case studies, and only with the client's written approval.
Each guide shows the date it was last revised, and that date moves only when the guidance itself changes, not for wording edits.
Yes. Send a short brief or a WhatsApp message describing your situation and the guide you read; the reply comes from the people who do the engineering.
Your next data decision
Bring the business question.
We'll help shape the engineering brief.
