Capability / Cost optimization
Data Platform Cost Optimization Services
Data platform cost optimization is the work of making cloud data spend attributable, then reducing it without reducing what the business gets. Quorlytic attributes spend to workloads and teams, finds the queries, pipelines and storage actually responsible, changes the expensive ones, and puts the governance in place that stops the cost returning within two quarters.
By QuorlyticUpdated
Most platform bills are dominated by a small number of workloads nobody has looked at. The hard part is not finding savings once, it is making them visible enough that they hold.
- 01Attribute
- 02Rank
- 03Remediate
- 04Govern
The business problem
A bill that cannot be broken down by workload, team and data product can only be absorbed, not managed. Tagging, warehouse or cluster separation, and query-level attribution turn an invoice into a list of decisions. Without that, cost work becomes a series of guesses followed by an argument about which one helped.
What we build
Cost tends to concentrate: a handful of pipelines, a few dashboards refreshing far more often than anyone reads them, one unpartitioned table scanned repeatedly. Ranking spend by workload before changing anything is what keeps the effort proportionate, and it usually makes the first few decisions obvious.
The outcome
An attributed cost baseline, a ranked set of changes with their measured effect, and the governance that keeps the result in place
When this is the right engagement
Symptoms that bring teams here.
Cost work is worth starting when spend has become unattributable rather than merely large.
The bill rose and nobody can explain which workload caused it
There is no attribution, so cost can be absorbed but not managed.
Finance is asking the platform team to explain a number
Spend is not reported to the teams that generate it, so nobody adjusts behaviour.
Development environments are sized like production
Standing compute is being paid for capacity nothing needs.
Dashboards refresh far more often than anyone opens them
Refresh cadence was set as a default rather than against real consumption.
A previous cost exercise saved money that came back
The changes were made without governance, so new workloads reintroduced the pattern.
Capability model
Where platform spend goes.
Four places account for most recoverable cost. Select one to see what drives it and what usually fixes it.
01 — Compute
Right-size and isolate compute so a workload pays for what it uses and cannot inflate another.
What it produces
Compute sized from measured query patterns, isolated per workload.
What usually goes wrong
- Warehouses or clusters that never suspend
- Sizing inherited from a contention problem that no longer exists
- Heavy transformations contending with interactive reporting
Technologies in this stage
- Warehouse sizing
- Auto-suspend
- Cluster policies
- Workload isolation
02 — Storage
Put a policy and an owner on retention, time travel, clones and file layout, so storage stops growing by default.
What it produces
A storage lifecycle policy with a named owner per data product.
What usually goes wrong
- Time travel and failsafe windows left at maximum
- Abandoned clones and intermediate tables
- Small-file accumulation with no compaction owner
Technologies in this stage
- Retention policy
- Lifecycle rules
- Compaction
- Clone management
03 — Frequency
Match refresh cadence to how the output is actually consumed, which is a conversation with the consumer.
What it produces
Cadence agreed against real consumption, with unused pipelines retired.
What usually goes wrong
- Fifteen-minute refresh on a daily-read report
- Full refresh where incremental would serve
- Pipelines still running for retired consumers
Technologies in this stage
- Incremental models
- Schedule review
- Usage analytics
04 — Governance
Keep the result by making spend visible to the teams that generate it and setting a standard for new workloads.
What it produces
A recurring review and reporting line that keeps cost visible at the moment it is incurred.
What usually goes wrong
- Attribution visible only to the platform team
- No sizing standard, so each new workload starts oversized
- Cost reviewed annually, by which time it has drifted
Technologies in this stage
- Tagging standards
- Budget alerts
- Recurring cost review
- Sizing defaults
Illustrative reference architecture
Reference architecture
Cost optimisation has no system to draw, so this is the control loop instead. The governance step is what distinguishes it from a one-off exercise that erodes within two quarters.
Telemetry
Query history, job metadata, storage metrics and billing exports.
- Query history
- Job metadata
- Storage metrics
- Billing
Attribution
Spend mapped to workload, team and data product.
- Tagging
- Compute separation
- Query attribution
Ranking
Cost concentrated, so effort goes where the money is.
- Top workloads
- Idle compute
- Storage growth
Change
Sizing, cadence, storage lifecycle and query efficiency.
- Sizing
- Cadence
- Lifecycle
- Query fixes
Measurement
Effect measured against the baseline, including performance.
- Before and after
- Performance check
- Verified saving
Governance
Reporting, alerts and a standard for new workloads.
- Recurring review
- Budget alerts
- Sizing standard
Constant throughout
- Performance protected
- Baseline recorded
- Changes attributable
- Consumer agreement
- Reporting to spending teams
The approach
- 01
Attribution comes before optimisation
A bill that cannot be broken down by workload, team and data product can only be absorbed, not managed. Tagging, warehouse or cluster separation, and query-level attribution turn an invoice into a list of decisions. Without that, cost work becomes a series of guesses followed by an argument about which one helped.
- Spend attributed to workload, team and data product
- Compute separated so contention and cost are both traceable
- A baseline recorded before any change is made
- 02
A small number of workloads usually dominate
Cost tends to concentrate: a handful of pipelines, a few dashboards refreshing far more often than anyone reads them, one unpartitioned table scanned repeatedly. Ranking spend by workload before changing anything is what keeps the effort proportionate, and it usually makes the first few decisions obvious.
- 03
Compute: sizing, isolation and idle time
Oversized warehouses, clusters that never suspend, and development environments running at production scale are common and straightforward to correct. Isolating workloads so a heavy transformation cannot contend with reporting usually improves both cost and reliability at once.
- Auto-suspend and sizing set from measured query patterns
- Workloads isolated so one does not inflate another
- Non-production environments sized for their actual purpose
- 04
Storage: lifecycle, retention and file layout
Storage is rarely the largest line, but it grows without anyone deciding it should. Time travel windows, failsafe retention, unmanaged clones, abandoned intermediate tables and small-file accumulation all accrue quietly. Each needs an owner and a policy rather than periodic manual cleanup.
- 05
Query and pipeline efficiency, where it pays
Pruning that never triggers, joins that fan out, full refreshes where incremental would serve, and orchestration that reruns unchanged work are all fixable. This work is only worth doing on the workloads attribution has already identified as expensive; optimising a cheap query is a hobby.
- 06
Frequency is a business decision, not a technical default
A dashboard refreshing every fifteen minutes that is read once a day costs roughly ninety-six times what it needs to. Refresh frequency should be set against how the output is actually consumed, and that is a conversation with the consumer rather than a setting an engineer changes unilaterally.
- Refresh cadence matched to real consumption patterns
- Unused reports and pipelines retired on evidence
- Frequency changes agreed with the consumer, not imposed
- 07
Governance is what stops the cost returning
Cost work without governance is a one-off saving that erodes over two quarters. A recurring cost review, attribution reported to the teams that generate the spend, budget alerts, and a sizing standard for new workloads are what keep the result. The aim is that an expensive decision stays visible at the moment it is made.
Delivery model
How the work is sequenced.
A bounded engagement that produces evidence first and changes second. The first phase alone is often enough to redirect the conversation.
| Phase | Activities | Decisions made here | What you keep |
|---|---|---|---|
| AttributeTurn an invoice into a list of decisions. |
|
| An attributed cost baseline by workload and team |
| RankFind where the money actually goes. |
|
| A ranked, quantified list of candidate changes |
| RemediateMake the changes and measure them. |
|
| Implemented changes with before-and-after measurement |
| GovernMake the result hold. |
|
| Cost reporting, alerting and a recurring review |
Technology fit
What each technology is here to do.
The levers differ per platform; the failure modes do not. These are the categories the work operates on.
Attribution sources
- Query history
- Job metadata
- Storage metrics
- Billing exports
Everything else depends on these. Without them, cost work is a sequence of guesses.
Compute levers
- Warehouse sizing
- Auto-suspend
- Cluster policies
- Job compute
Idle and oversized compute is usually the largest single recoverable line.
Platform detailStorage levers
- Retention
- Time travel
- Compaction
- Clone lifecycle
Rarely the biggest line, but it grows without anyone deciding it should.
Pipeline levers
- Incremental models
- Schedule tuning
- Partition pruning
- Orchestration
Worth applying only to workloads attribution has already shown to be expensive.
Governance controls
- Tagging standards
- Budget alerts
- Cost review
- Sizing defaults
What separates a lasting result from a saving that erodes over two quarters.
Technology marks belong to their respective owners. No partnership, certification or endorsement is implied.
Engineering concerns
What has to be answered before it is trusted.
What keeps a cost engagement honest.
- Performance protection
- Query and refresh performance measured before and after every change. A change that degrades a user-visible path is a defect, not a saving.
- Measured baseline
- A recorded baseline period, so the effect of each change is demonstrated rather than claimed.
- Consumer agreement
- Cadence and retirement decisions made with the consumer, because a unilateral change to a refresh schedule is an outage from their side.
- Attribution visibility
- Spend reported to the teams generating it. Attribution seen only by the platform team changes nobody's behaviour.
- No claimed percentages
- We do not quote savings before seeing the platform, and results are reported as measured figures against your baseline.
- Durability
- A sizing standard and a recurring review, so new workloads do not reintroduce the pattern that was just removed.
Where it applies
Related work, and why it is related.
Where this fits, and what it hands off to.
- Managed data and AIThe parent service. Ongoing cost review is usually folded into a managed arrangement, because governance is what makes the saving durable.
- SnowflakeWarehouse sizing, auto-suspend and workload isolation are the platform-specific levers behind most Snowflake cost work.
- Data modernizationOptimise before migrating for cost reasons — a migration undertaken to reduce spend tends to carry the same inefficiencies onto a new platform.
- SaaSPer-tenant cost attribution is both a margin question and an engineering one, which makes the attribution phase directly commercial.
Define the evidence a cost assessment must produce.
A useful diagnosis connects consumption to a workload and a business constraint. This hypothetical worksheet scopes a read-only assessment of a Snowflake reporting workload. It separates evidence collection from a later, explicitly approved experiment; a recommendation does not authorize a production change.
Scope and assumptions: Agree access boundaries, evidence retention and deliverable acceptance before collecting data. This worksheet does not establish pricing, delivery dates or a savings estimate.
| Dimension | Input or boundary to agree | What an acceptable deliverable records |
|---|---|---|
| Read-only scope | Approve a least-privilege role or sanitized exports for the agreed usage and workload records. Exclude permission changes and production configuration writes. | Evidence provenance, collection permissions, redaction, retention and any unavailable records. No credentials in the report. |
| Observation window | Select dates covering ordinary reporting and the relevant peak or close cycle. Record time zone, missing periods and concurrent changes. | A dated baseline, its coverage gaps and why the selected interval represents the workload. An incomplete cycle remains a limitation. |
| Workload attribution | Agree the mapping from warehouses, query history, schedules and available tags to teams or products; separate shared consumption. | A reproducible attribution method with an explicitly unallocated category. Reconcile compute, storage and data-transfer scope rather than equating query time with the whole bill. |
| Business constraint | The workload owner supplies the completion deadline, freshness requirement, concurrency pattern and permissible change window. | The named constraint and its owner beside each recommendation; the assessor does not invent an acceptable performance trade-off. |
| Proposed experiment | For example, evaluate a warehouse configuration change only after the owner approves a representative test and recovery path. | An experiment plan with comparable inputs, recorded configuration, consumption, elapsed time and queueing evidence. Implementation requires separate approval. |
| Acceptance and handoff | Name the cost owner and technical reviewer. Agree whether the engagement ends at diagnosis or includes separately scoped follow-up. | Accept an evidence-backed recommendation, assumptions and rejected options. Leave savings unestimated where the data cannot support them. |
At narrow widths, scroll the worksheet to read every column. With the table focused, use the arrow keys to scroll.
Worked decision: a lower-consumption run that misses the reporting deadline fails the agreed constraint. Keep it as a rejected experiment with its evidence, rather than reporting it as an accepted saving. Changes in workload volume or background activity also need to be disclosed before comparing runs.
- Snowflake documents warehouse, serverless and cloud-services compute as distinct categories, alongside storage and transfer costs. State which categories the evidence covers and which it cannot attribute.
- An assessment is a bounded diagnostic. Ongoing observation, change execution and operational ownership require their own agreed responsibilities; they are not implicit in a cost recommendation.
For the technical mechanisms behind the worksheet, read the Snowflake cost investigation guide. This service worksheet concerns assessment inputs and acceptance.
Official references for this example:
Questions
Questions worth asking.
How much can we expect to save?
We do not quote a figure before seeing the platform, and any firm that does is guessing at your workload. Savings depend entirely on how much of the current spend is idle compute, over-frequent refresh and unmanaged storage. The attribution phase produces a ranked, quantified list, and the decision to proceed is made against that rather than against a promise.
Will this slow anything down?
It should not, and any change that does is a defect. Performance is measured before and after each change, and workload isolation frequently improves reporting performance because a heavy transformation is no longer contending with it. Changes that trade user-visible latency for cost are presented as a decision for you rather than made on your behalf.
Is this a one-off engagement or ongoing?
It can be either, and the distinction matters. A bounded engagement produces an attributed baseline, a set of changes and the governance to maintain them. Without the governance the saving typically erodes within two quarters as new workloads arrive, which is why ongoing cost review is often folded into a managed operations arrangement instead.
Which platforms does this cover?
The approach is platform-agnostic because the failure modes are: unattributed spend, idle compute, over-frequent refresh and unmanaged storage appear on Snowflake, Databricks, BigQuery and Fabric alike. The specific levers differ — warehouse sizing and auto-suspend, cluster policies and job compute, slot reservations and partition pruning — and we work with what you run.
Do we need to change platform to reduce cost?
Rarely, and it is usually the most expensive way to attempt it. Most platform bills have significant recoverable waste before any migration is considered, and a migration undertaken for cost reasons tends to carry the same inefficiencies onto a new platform. Optimise first, then decide whether the platform is genuinely the constraint.
What access do you need?
Query history, job and pipeline metadata, storage metrics and billing or usage reports. Production data itself is not required for the attribution and analysis work, and access to it is not part of scoping.
How is the result reported?
As an attributed baseline, a ranked list of changes with their measured effect, and a recurring report showing spend by workload and team. The point of the reporting is that the teams generating spend can see it, because attribution visible only to the platform team changes nobody's behaviour.
Next step
Get your platform spend attributed.
Query history and a usage export are enough to produce a ranked list. No production data is needed.