Technology / AWS
An AWS data platform built around the workload.
An AWS data platform is assembled from services: S3 stores the data, Glue or EMR processes it, Redshift serves concurrent analytical queries, Kinesis carries streaming events and Lake Formation governs access. Quorlytic gives each service one job, models Redshift around query patterns rather than table counts, and keeps cost attributable to a named workload.
By QuorlyticUpdated
Assign one responsibility to each AWS service before building pipelines across them. Quorlytic designs the storage, processing, serving and access boundaries together, then proves the resulting platform on a workload you already run.
Architecture role
Where AWS fits.
An AWS data platform is assembled rather than bought: S3 holds the data, Glue or EMR processes it, Redshift serves concurrent analytical queries and Kinesis carries events. Because no single service owns the whole path, the architecture decision is the responsibility boundary between services, not the choice of one product.
Position in the end-to-end path
- 01SourcesApplications, databases, files
- 02IngestBatch, CDC and streaming
- 03AWSS3 storage, Glue / EMR processing, Redshift serving
- 04Governed modelsCatalogue, models and access
- 05Analytics & decisionsBI, models and applications
Sources
- Apps
- Databases
- Files
- Events
Governance IAM · Lake Formation · Glue Catalog · cost tags
Ingest
- Kinesis
- MSK
- DMS
Process
- Glue
- EMR
- Lambda
- Step Functions
Serve
- Redshift
- Athena
- OpenSearch
S3 Shared storage every service reads and writes · Iceberg tables
- BI
- APIs
- Applications
- ML
Assembled, not bought — the boundaries between services are the design
Illustrative architecture. It shows a typical position for discussion, not a mandatory stack or a client implementation.
Fit and capability
When it is relevant, and what the work involves.
Relevant workloads
- Warehouse analytics
- Lake storage and processing
- Streaming ingestion
- Legacy warehouse migration
Engineering capabilities
- Workload and cost baseline across services
- Service responsibility and access design
- Redshift or Athena modelling recommendation
- Migration validation and operating checklist
Integration points
What it connects to.
The platform is only as useful as the boundaries around it. These are the interfaces that usually need an explicit decision.
- 01IAM roles and policies
- 02Catalogue and lineage
- 03Orchestration and CDC
- 04Cost and usage controls
Delivery and operating considerations
What to decide before you commit.
Each consideration changes something concrete in the architecture, the operating model or the cost profile.
- 01
Assign each AWS service one job
Most AWS data estates drift because several services do the same thing. Decide explicitly what S3 owns (durable storage and the conversation that follows), what Glue or EMR owns (processing and catalogue), what Redshift owns (concurrent analytical workloads) and what Kinesis owns (event transport). A service that owns nothing in particular usually becomes the one nobody can operate.
- One storage convention, versioned and lifecycle-aware
- Processing separated from serving
- Catalogue updated by the pipeline, not by hand
- 02
Model Redshift around queries, not tables
A Redshift model is judged by the queries it serves. Distribution and sort keys, materialised views, workload management and Spectrum for colder data all change how the platform behaves under concurrency. Query patterns and refresh windows matter more than the number of tables, and Spectrum extends reach without requiring every byte to live in the warehouse.
- Query patterns and concurrency profiled before modelling
- Hot analytical data separated from archive access
- Refresh and vacuum expectations documented
- 03
Make consumption visible before it surprises you
AWS cost is spread across services and load patterns, which makes an unattributed bill hard to defend. Redshift compute, Athena bytes scanned, Glue job hours, S3 storage classes and Kinesis shards each respond to a different design decision. Attribute usage to a workload so a cost conversation can happen before the invoice does.
- Cost attributed per workload and per environment
- Storage lifecycle and file layout reviewed together
- Query patterns reviewed against bytes scanned
- 04
Control the account and access boundary
Access on AWS is a combination of IAM roles, bucket policies and catalogue permissions, and it is easy to end up with one over-permissive role that everything uses. Give each pipeline its own identity, scope permissions to the data it actually needs, and test with the roles that real consumers hold rather than with an administrator account.
- A service role per pipeline, scoped to its data
- Catalogue and storage permissions tested together
- Cross-account and audit requirements decided explicitly
- 05
Know when AWS is the wrong place for a workload
If the analytics centre of gravity already runs on another cloud, adding AWS means a second operating model, duplicated tooling and egress costs. The comparison is rarely service-to-service: it is the cost of running the whole path, including the people who operate it. Prove the case with a workload before committing an estate to it.
- Total operating path compared, not one service price
- Existing commitments and skills included
- A representative workload used as the test
Typical delivery sequence
- 01Baseline & service map
- 02Storage & access design
- 03Pipeline & warehouse build
- 04Operate & control cost
Decision table
When AWS fits, and when to compare.
| Dimension | AWS is a strong fit | Compare alternatives when |
|---|---|---|
| Existing estate | Applications, identity and data already run on AWS. | The organisation is standardised on another cloud. |
| Warehouse | Redshift for governed SQL analytics, with S3 as the lake underneath. | A multi-cloud warehouse or a lakehouse platform is the stated target. |
| Pipelines | Glue and Lambda for managed ingestion and ETL, orchestrated with Step Functions or Airflow. | Pipelines depend on a Spark platform the team already operates elsewhere. |
| Governance | IAM and Lake Formation control access across the lake and the warehouse. | Access policy must be centralised in a platform outside AWS. |
| Operations | A team comfortable composing several managed services. | The team wants one integrated platform with fewer moving parts. |
Define what an S3 recovery can actually restore.
A versioned landing bucket is useful evidence of recoverability, but the setting alone does not prove that a data product can be restored. Object versions, deletion type, encryption access and downstream publications each affect the result. This hypothetical AWS assessment follows one general-purpose S3 landing object through a recovery decision without treating storage versioning as a complete analytics backup strategy.
Scope and assumptions: This is a read-only assessment worksheet for general-purpose S3 buckets. It supplies no delete, lifecycle or recovery commands and authorises no changes to a live bucket. Directory buckets have different versioning support; versioning-suspended buckets also need separate treatment. Retention and deletion obligations remain applicable during recovery.
| Dimension | State to distinguish | Evidence needed for a credible recovery claim |
|---|---|---|
| Bucket and version state | Record the bucket type, current versioning state and the versions relevant to the consumer's accepted dataset. Enabling versioning today does not reconstruct an object that was already permanently removed. | Keep object keys and version references in a bounded inventory with suitable access controls. Do not infer the state of every object from a screenshot showing only the bucket's current versioning setting. |
| Delete marker or version removal | AWS documents that an ordinary delete in a versioning-enabled bucket inserts a delete marker. A request naming a specific version can permanently delete that version. These are different recovery situations even if the current object is absent in both. | Inspect which versions actually remain and which deletion occurred. A delete marker may hide an older retained version; it is not evidence that every historical version is still recoverable. |
| Retention and dependencies | Include the retention policy, lifecycle treatment of noncurrent versions and any object-lock or deletion constraints relevant to the dataset. Identify the encryption key and permissions needed to read the retained content. | Confirm availability of both the object version and its required access path. A retained encrypted object is not a demonstrated usable recovery if the required key or permitted role is unavailable. |
| Dataset consistency | A published dataset may consist of many objects and a manifest. Restoring whichever version looks newest for each key can combine files from incompatible publications. | Tie the candidate recovery to an accepted dataset manifest or equivalent consistent reference. Check schema, business date and expected consumer totals as well as the presence of individual files. |
| Downstream and permitted use | List curated tables, extracts and reports created after the landing publication. A restored object does not automatically restore those stores or reverse a business action that already consumed them. | Define which consumers must be rebuilt or reconciled and which deleted or withdrawn data must remain excluded. Document what the recovery covers before accepting a platform-wide recovery claim. |
At narrow widths, scroll the worksheet to read every column. With the table focused, use the arrow keys to scroll.
Worked decision: an older object version remains, but the publication manifest refers to other files whose required versions are gone. Treat the retained object as partial evidence, not an accepted dataset recovery. Identify another permitted reconstruction source or record the unrecoverable boundary. This hypothetical finding illustrates the assessment deliverable and does not describe an incident or an achieved recovery time.
- Assess version-specific deletion permissions separately from ordinary object deletion. A broad statement that versioning prevents deletion overlooks the API's explicit permanent-version removal path.
- Use synthetic or approved non-production data for any later recovery exercise. Record actual restored-state comparisons and timings only after a separately scoped test; the inventory alone establishes neither.
To connect storage recovery to consumer publication and reliability measures, read data pipeline reliability.
Official references for this example:
Questions
Questions worth asking.
Is Redshift or Athena the right choice?
Redshift suits repeated, concurrent analytical queries with predictable workloads and modelling. Athena suits occasional querying directly over S3 where scanning cost and latency are acceptable. Many estates use both, with the boundary decided by concurrency, refresh cadence and how the results are consumed.
Do we need Lake Formation if S3 policies already exist?
Bucket policies control access to objects; a catalogue layer controls access to tables and columns and records how data is governed. If analysts query through a catalogue, permissions there decide what they can see. Choose one place to answer that question rather than two that disagree.
Integration ecosystem
What sits around AWS.
Adjacent technologies grouped by the role they play next to this platform, and why that role usually needs an explicit decision.
Storage
- S3
- Lake Formation
- Glue Catalog
- Iceberg
S3 is the foundation nearly every AWS data architecture is built on, which makes the catalogue and permission model the deciding design.
Processing
- Glue
- EMR
- Lambda
- Step Functions
Several services can do the same job with different operational profiles, so the choice is about who operates it.
Serving
- Redshift
- Athena
- OpenSearch
- RDS
Concurrency and query pattern decide between a provisioned warehouse and serverless querying over the lake.
Streaming
- Kinesis
- MSK
- DMS
Managed Kafka and Kinesis differ mainly in operational burden and ecosystem compatibility rather than in capability.
Governance and cost
- IAM
- Lake Formation permissions
- Cost allocation tags
- CloudWatch
Because no single service owns the path, identity and cost attribution have to be designed across services rather than inherited from one.
Technology marks belong to their respective owners. No partnership, certification or endorsement is implied.
Migration and modernization
What moves, and what that asks of you.
Typical transitions on AWS. Most are consolidations of an assembled estate rather than platform swaps.
On-premises warehouseRedshift or a lake-first architecture
The decision is whether to recreate the warehouse or to land in S3 first and serve selectively. The second is usually more flexible and more work.
Scripts on EC2Managed processing and orchestration
Glue, Step Functions or a managed Airflow removes undifferentiated operational work, and requires accepting each service's constraints.
Redshift-onlyLake plus warehouse
Moving cold and semi-structured data to S3 with Athena or Redshift Spectrum over it, keeping the warehouse for concurrent governed analytics.
AWSA single managed platform
Worth considering when the assembly cost across services exceeds the flexibility it buys, which is a real and legitimate outcome.
Performance and economics
What drives the bill.
Cost is spread across many services, which makes attribution the first problem rather than the last.
Attributing and reducing AWS spend as an engagement, rather than a tuning exercise, is covered by data platform cost optimization.
- Storage tiers
- S3 lifecycle policies move cold data down tiers, and it is among the most reliable savings available on the platform.
- Compute model
- Provisioned Redshift, serverless Redshift and Athena price the same query very differently depending on how often it runs.
- Data scanned
- Athena and Spectrum bill on bytes scanned, so partitioning and file format are cost controls, not just performance ones.
- Commitments
- Savings plans and reserved capacity reward predictable workloads and penalise misjudged forecasts.
- Attribution
- Cost allocation tags applied consistently across services, without which the bill cannot be broken down by workload at all.
Governance and security
What has to be designed first.
The governance work on AWS is cross-service by nature, which is its main architectural cost.
- Identity
- IAM roles, service roles and Lake Formation permissions have to compose into one coherent access model rather than three overlapping ones.
- Catalogue
- A shared catalogue is what lets several query engines see the same tables with the same permissions.
- Network boundaries
- VPC design, endpoints and cross-account access are part of the data architecture on AWS, not a separate infrastructure concern.
- Encryption and keys
- Key management and rotation policy per data classification, decided once rather than per service.
- Audit
- CloudTrail and access logging give the evidence trail, and it is only useful with retention and someone accountable for reviewing it.
Trade-offs
When it fits, and when it does not.
Where an assembled AWS data platform fits, and where a single managed platform is the better answer.
Where AWS tends to fit
- Estates already operating substantially on AWS, where identity and networking are established
- Architectures needing several processing models over the same object storage
- Teams with the engineering capacity to own boundaries between services
- Workloads where fine-grained control over storage tiering and compute is worth the assembly cost
Where another answer deserves consideration
- SnowflakeYou want one platform to own storage, compute, governance and sharing rather than composing them across services.
- DatabricksLarge-scale processing and machine learning need one governed lakehouse rather than several AWS services stitched together.
- Staying multi-serviceThe flexibility is genuinely used. Consolidating a working estate has a real migration cost and should clear a real constraint.
Related architectures
Patterns this platform appears in.
References
Primary sources for this platform.
Vendor documentation, linked for verification. It is not an endorsement, partnership or certification.
Next step
Review your AWS data architecture.
Bring the service boundaries you have today. On AWS the boundary decisions matter more than the service list.