Technology / Google Cloud
Google Cloud data engineering, arranged around the workload.
Google Cloud data engineering arranges BigQuery, Cloud Storage, Dataflow, Dataproc and Pub/Sub so each service owns one job in a GCP estate. Quorlytic designs the pipelines and access boundaries between them, treats BigQuery as the governed analytical layer rather than a landing zone, and makes slot, byte and storage cost observable per workload before the first migration wave.
By QuorlyticUpdated
Decide what BigQuery should own before migrating workloads into it. Quorlytic scopes the processing model, the access boundary and the cost model together, then proves them on a representative workload.
Architecture role
Where Google Cloud fits.
Google Cloud commonly carries both the storage foundation and the analytical warehouse: Cloud Storage holds landed and archived data, BigQuery serves governed analytics, and Dataflow, Dataproc or Pub/Sub move and process it. The architecture decision is which service owns each job and how byte, slot and storage cost stay visible.
Position in the end-to-end path
- 01SourcesApplications, databases, events
- 02LandingCloud Storage and Pub/Sub
- 03Google CloudBigQuery, Dataflow and Dataproc
- 04Governed modelsDatasets, policies and tags
- 05Analytics & MLLooker, notebooks and applications
In
- Pub/SubEvents
- Cloud StorageLanded files
- Dataflow · DataprocWhen it must run outside
BigQuery
Storage and compute, separated
- Raw datasets
- SQL · dbt
- Modeled datasets
Policy tags · IAM
Out
- Looker
- Connected Sheets
- ML features
- Applications
Cost follows bytes scanned or slots reserved — design for the query, not the table
Illustrative architecture. It shows a typical position for discussion, not a mandatory stack or a client implementation.
Fit and capability
When it is relevant, and what the work involves.
Relevant workloads
- Warehouse analytics
- Batch and streaming pipelines
- Lake on object storage
- Machine learning features
Engineering capabilities
- Workload, cost and access baseline
- Target architecture with service responsibilities
- Pipeline and migration validation plan
- Operating and cost-observability checklist
Integration points
What it connects to.
The platform is only as useful as the boundaries around it. These are the interfaces that usually need an explicit decision.
- 01IAM and policy tags
- 02Orchestration and scheduling
- 03BI and semantic layer
- 04Slot, byte and cost controls
Delivery and operating considerations
What to decide before you commit.
Each consideration changes something concrete in the architecture, the operating model or the cost profile.
- 01
Give BigQuery one clear responsibility
BigQuery works well as the governed analytical layer, and poorly as an ingestion buffer or an unmanaged landing zone. Name the layers first: object storage lands raw and archived data, a processing service moves and validates it, and BigQuery serves governed models that reporting and models read from. That boundary decides partitioning, retention and who can change a table.
- Landing, governed and serving layers named explicitly
- Datasets scoped to a domain or data product, not to a team
- External tables considered where data already sits in Cloud Storage
- 02
Choose the processing model before the service
BigQuery handles set-based SQL transformation well. Dataflow suits pipelines that need the same logic in batch and streaming, and Dataproc suits existing Spark code, notebooks or heavy feature engineering. Pub/Sub transports events; it does not decide how they are replayed. Each option carries a different operating burden, and the choice should follow the decision the data supports, not the service list.
- Batch or streaming decided by the decision’s freshness requirement
- Existing Spark jobs evaluated before they are rewritten
- Late data, replay and schema change designed in from the start
- 03
Make cost visible in bytes, slots and storage
BigQuery cost has three separate drivers: bytes scanned by queries, compute capacity consumed, and stored data. On-demand pricing and capacity pricing behave differently under concurrency, and partitioning or clustering decisions change how much each query reads. Establish a baseline per workload before optimising, then revisit the pricing model rather than assuming one is cheaper.
- Bytes and slots attributed to a named workload
- Partitioning and clustering aligned to real query filters
- Storage billing model and retention reviewed explicitly
- 04
Decide the access boundary in the platform
Access control belongs in the data platform rather than in a dashboard. Service accounts per pipeline, dataset-level IAM, column policies and policy tags, plus audit logging, make it possible to answer who could see which record and why. Where regulatory or residency requirements apply, network and key boundaries need an explicit decision rather than a default.
- A service identity per pipeline, not a shared human account
- Row and column policies tested with real user identities
- Audit logging and key management agreed before go-live
- 05
Know when another platform is the better answer
Google Cloud is not automatically the cheapest or simplest option for an estate already running elsewhere. Egress, duplicate tooling, team skills and an existing committed platform all change the arithmetic. Compare the cost of operating one more platform against the benefit of the workload it would carry, and prove it with a representative workload rather than a benchmark.
- Existing platform cost compared, not assumed
- Skills and operating model counted, not just licence
- Migration justified by a measured workload test
Typical delivery sequence
- 01Baseline & workload map
- 02Architecture & access design
- 03Prove, migrate & validate
- 04Operate & observe cost
Decision table
When Google Cloud fits, and when to compare.
| Dimension | Google Cloud is a strong fit | Compare alternatives when |
|---|---|---|
| Analytics | Serverless SQL analytics in BigQuery, with capacity managed for you. | The team needs full control of clusters and processing engines. |
| Existing estate | Operational systems, identity and data already live in Google Cloud. | Most systems and skills sit in AWS or Azure. |
| Pipelines | Pub/Sub for events and Dataflow for streaming and batch processing. | Pipelines are Spark-first and a lakehouse is already in production. |
| Machine learning | BigQuery ML and Vertex AI kept close to the data for first models. | Model work is already standardised on another platform's tooling. |
| Cost control | Partitioned, clustered tables and a cost model the team reviews. | Ad-hoc scans run unbounded and nobody reviews bytes processed. |
Keep a BigQuery estimate separate from the executed job.
An estimated scan size helps assess a query, but it is not the executed workload's bill or a complete spending control. In this hypothetical BigQuery reporting change, the assessment keeps the query, dry-run evidence, execution settings and observed job statistics together. It also distinguishes on-demand byte-based analysis from capacity-based economics before interpreting a smaller scan as a cost reduction.
Scope and assumptions: This is a read-only evaluation design, not a runnable query or an account configuration change. The example supplies no prices, quotas or savings estimate. The applicable billing model, reservation settings, permissions and workload constraints must be established for the actual project.
| Dimension | Evidence boundary | Record and interpretation |
|---|---|---|
| Exact query candidate | Keep the SQL revision, parameters, referenced objects, location and expected reporting result. Compare the current and proposed forms under the same business filters rather than measuring two different questions. | Retain representative result checks and the relevant schema state. Removing a join can lower estimated bytes while also dropping required records, which would invalidate the comparison before any cost interpretation. |
| Dry-run estimate | Capture the dry run with its timestamp and settings. Google's Python sample demonstrates a dry run with query cache disabled and reports estimated bytes processed without executing the query. | Keep this value labelled as an estimate for that query state. It does not demonstrate actual runtime, concurrency behavior, an executed job's billed bytes or the workload's full operating cost. |
| Execution boundary | For an applicable on-demand query, distinguish the configured maximum_bytes_billed job setting from a budget alert or the dry-run observation. Record whether the setting is actually present on the execution path being evaluated. | The Python client documents maximum_bytes_billed as a per-job maximum. It is not a platform-wide budget, and a byte limit is not a universal cost control for capacity-based pricing or other services in the pipeline. |
| Observed job evidence | For a separately permitted execution, retain the job identity, cache status, bytes processed, bytes billed, elapsed time and available slot statistics. Compare equivalent workloads and explain whether cached results influenced the observation. | Report estimates and observations in different fields. A cache hit or changed source data can explain a difference without proving that the proposed SQL is cheaper under the intended production workload. |
| Whole workload economics | Include refresh frequency, concurrent consumers, storage and upstream processing in the decision. Under capacity pricing, reduced scan estimates alone do not show how reservation utilisation or the bill changes. | Tie the conclusion to the billing model and representative workload window. Preserve correct report results and freshness while stating which cost components the comparison does and does not cover. |
At narrow widths, scroll the worksheet to read every column. With the table focused, use the arrow keys to scroll.
Worked decision: the proposed query has a smaller dry-run estimate, but the observed comparison uses a cache hit for one candidate and a full execution for the other. Retain both observations, align the comparison conditions and verify result equivalence before interpreting performance or cost. No savings percentage or benchmark result follows from this hypothetical example.
- Verify controls at the caller that submits the actual job. A documented limit in a notebook does not prove that a dashboard, scheduled query or another service sends the same job configuration.
- Keep query-level evidence distinct from account-level billing reconciliation. Attributing one workload requires an explicit window and coverage boundary, especially when resources or reservations are shared.
For a wider assessment across workloads and billing components, see data platform cost optimisation.
Official references for this example:
Questions
Questions worth asking.
Is BigQuery cheaper than the warehouse we already operate?
There is no general answer. It depends on query shape and concurrency, how the data is laid out, storage billing, existing commitments and the cost of running a second platform. The first deliverable is a measured comparison on representative queries, including storage and egress.
Should we use Dataflow or Dataproc?
It depends on the code you already have and how you want to operate. Dataflow runs Apache Beam pipelines that share logic between batch and streaming; Dataproc suits existing Spark workloads, notebooks and heavier feature engineering. Both need the same replay, late-data and schema-change discipline.
Integration ecosystem
What sits around Google Cloud.
Adjacent technologies grouped by the role they play next to this platform, and why that role usually needs an explicit decision.
Storage
- Cloud Storage
- BigQuery storage
- Iceberg
- BigLake
Managed table storage and object storage have different governance and cost properties, and most estates end up with both.
Processing
- Dataflow
- Dataproc
- BigQuery SQL
- dbt
Much transformation can run in BigQuery itself, which removes a processing tier and concentrates cost in query.
Streaming
- Pub/Sub
- Dataflow
- BigQuery streaming inserts
The streaming path is unusually well integrated, which makes near-real-time analytics comparatively straightforward here.
Consumption
- Looker
- Power BI
- Connected Sheets
- Applications
Looker integration is close, but BigQuery is tool-agnostic enough that consumption is a separate decision.
Governance and cost
- IAM
- Policy tags
- Slot reservations
- Query cost controls
Column-level policy tags and per-query cost controls are two of the platform's more distinctive governance features.
Technology marks belong to their respective owners. No partnership, certification or endorsement is implied.
Migration and modernization
What moves, and what that asks of you.
Typical transitions into and around BigQuery.
On-premises warehouseBigQuery
Serverless operation removes cluster sizing entirely, which is a genuine simplification. The work moves to partitioning, clustering and query cost control.
On-demand pricingSlot reservations
Worth considering once query volume is predictable enough to commit, and it needs measured job history rather than an estimate.
Dataproc or Spark jobsBigQuery SQL
Much Spark transformation can be expressed in SQL, removing a processing tier. Not all of it should be, and the exceptions are usually ML feature work.
BigQueryA lakehouse
Worth considering when ML workloads need direct file access at scale, or when open formats are a deliberate portability position.
Performance and economics
What drives the bill.
The pricing model is the most distinctive thing about the platform, and it changes how queries should be written.
Attributing and reducing Google Cloud spend as an engagement, rather than a tuning exercise, is covered by data platform cost optimization.
- Bytes scanned
- On-demand querying bills on data scanned, so partitioning and clustering are direct cost controls rather than tuning niceties.
- Slot reservations
- Committed capacity makes cost predictable and rewards steady utilisation, which needs measured job history to size.
- Storage tiers
- Long-term storage pricing applies automatically to untouched partitions, which is one of the few savings that needs no action.
- Streaming inserts
- Priced separately from batch loading, and worth comparing against micro-batching for the same freshness.
- Cost controls
- Per-query and per-project limits stop a single unbounded query becoming a monthly surprise.
Governance and security
What has to be designed first.
Column-level controls and project structure carry most of the governance weight here.
- Project and dataset structure
- Projects are the natural cost and access boundary, so the structure is a governance decision as much as an organisational one.
- Policy tags
- Column-level access control through a taxonomy, which handles sensitive fields without splitting tables.
- Authorized views
- Sharing derived results without granting access to the underlying tables, which is a clean pattern where it fits.
- Residency
- Dataset location is fixed at creation and cannot be changed in place, so it has to be right the first time.
- Audit
- Cloud Audit Logs record access and query, and they need retention and an accountable reviewer to be worth anything.
Trade-offs
When it fits, and when it does not.
Where BigQuery fits, and where another platform deserves consideration.
Where Google Cloud tends to fit
- Estates already on Google Cloud, or teams that want no cluster sizing to manage
- Analytical workloads with variable volume where serverless pricing suits the pattern
- Streaming analytics where the ingestion path is a first-class requirement
- Governance needs met well by column-level policy tags and authorized views
Where another answer deserves consideration
- SnowflakeWorkload isolation between teams and multi-cloud portability matter more than serverless simplicity.
- DatabricksMachine learning and large-scale file processing need direct access to the same data rather than an analytical copy.
- Staying on the current cloudThe estate is not on Google Cloud. Cross-cloud data movement and egress usually outweigh the platform difference.
Related architectures
Patterns this platform appears in.
References
Primary sources for this platform.
Vendor documentation, linked for verification. It is not an endorsement, partnership or certification.
Next step
Review your BigQuery cost and architecture.
Bring your INFORMATION_SCHEMA job history. Slot and byte questions are answerable directly from it.