Capability / Real-time streaming
Real-Time Data Streaming Services
Real-time data streaming moves events from source systems to a consumer within seconds rather than on a schedule. Quorlytic designs the parts batch pipelines never have to answer: event time versus arrival time, delivery guarantees, how processing state is held and recovered, and the serving layer an application or dashboard actually reads from.
By QuorlyticUpdated
Streaming is worth its operating cost when a decision loses value while it waits. That test comes before any discussion of Kafka, Flink or a streaming table.
- 01Latency case
- 02Stream design
- 03Build & reconcile
- 04Operate
The business problem
Streaming is frequently requested where hourly batch would serve the decision equally well, and it carries a permanent operating cost that batch does not. Start by naming the decision, how quickly it degrades, and what the business does differently at one second, one minute and one hour of delay. If the answer is the same at all three, the architecture should be batch.
What we build
An event carries the time it happened; a pipeline sees the time it arrived. Mobile clients, retries, network partitions and replays mean these diverge routinely, and a window that closes on arrival time silently misattributes late events. Deciding which clock a calculation uses, and how long a window waits for stragglers, is the first real design decision in a stream.
The outcome
A streaming pipeline with stated delivery semantics, recoverable state and a reconciliation check against the system of record
When this is the right engagement
Symptoms that bring teams here.
Streaming carries a standing operating cost, so it should be introduced against a specific symptom rather than as a modernisation goal.
A decision is made on data that is already hours old
The batch interval sits inside the window where the decision still had value.
An application polls the warehouse every minute
A streaming requirement is being met with a batch tool, expensively and unreliably.
Operational alerts fire long after the condition passed
Detection latency exceeds the time the response needed to matter.
A real-time dashboard disagrees with the daily report
Two paths compute the same measure with no reconciliation and no stated boundary.
A stream exists but nobody can replay it
Retention and offset handling were not designed, so recovery means data loss.
Capability model
What a stream has to answer.
Four design questions separate a working stream from a demonstration. Select one to see what it decides and what goes wrong when it is left to a default.
01 — Time
Decide which clock a calculation uses — when the event happened, or when the pipeline saw it — and how long a window waits for late arrivals.
What it produces
A stated windowing and lateness policy the consumer has agreed to.
What usually goes wrong
- Windows closing on arrival time and misattributing late events
- No watermark, so stragglers are silently dropped
- Client clock skew treated as noise rather than as a design input
Technologies in this stage
- Event-time windows
- Watermarks
- Allowed lateness
02 — Delivery
Choose the delivery guarantee and make the consumer able to survive it, rather than assuming the transport will be perfect.
What it produces
A delivery guarantee with matching consumer behaviour, tested under replay.
What usually goes wrong
- Exactly-once assumed across systems that cannot provide it
- Consumers that are not idempotent, so replay duplicates records
- No agreed idempotency key with the producing system
Technologies in this stage
- At-least-once delivery
- Idempotent consumers
- Deduplication windows
03 — State
Decide what the stream remembers, where that memory lives, and how long recovery takes when the process restarts.
What it produces
Bounded, recoverable state with a measured restart time.
What usually goes wrong
- State store growing without a retention policy
- Recovery time never measured until an incident
- Version upgrades that invalidate existing state
Technologies in this stage
- Checkpointing
- State stores
- Rebalance handling
04 — Serving
Decide what the consumer reads from, and how the low-latency path relates to the analytical system of record.
What it produces
A serving store matched to the read pattern, reconciled on a schedule.
What usually goes wrong
- A stream treated as a query interface
- Two definitions of the same measure on two paths
- No reconciliation, so drift goes unnoticed
Technologies in this stage
- Streaming tables
- Operational stores
- Caches
- Reconciliation jobs
Illustrative reference architecture
Reference architecture
The low-latency path, drawn alongside the reconciliation that keeps it honest. The comparison back to the system of record is part of the architecture, not an afterthought.
Event producers
Applications, devices and change data capture from operational stores.
- Applications
- Devices
- CDC
Event transport
Durable, replayable, with retention set deliberately.
- Kafka
- Managed topics
- Retention policy
Stateful processing
Windows on event time, checkpointed state, idempotent output.
- Windowing
- Checkpoints
- Deduplication
Serving store
Matched to how the consumer actually reads.
- Streaming tables
- Operational store
- Cache
Reconciliation
Scheduled comparison against the system of record.
- Counts
- Aggregates
- Drift alert
Across the whole path
- Schema registry
- Consumer lag monitoring
- Replay procedure
- Ownership
- Standing compute cost
The approach
- 01
Establish the latency the decision actually requires
Streaming is frequently requested where hourly batch would serve the decision equally well, and it carries a permanent operating cost that batch does not. Start by naming the decision, how quickly it degrades, and what the business does differently at one second, one minute and one hour of delay. If the answer is the same at all three, the architecture should be batch.
In financial-services data foundations, a transaction update needs its posting time, value date and source identity retained so a low-latency consumer can still be reconciled with governed reporting.
- Freshness target stated per decision, not per platform
- Cost of the latency requirement made visible before committing
- Batch or change data capture chosen where they meet the target
- 02
Event time and processing time are different clocks
An event carries the time it happened; a pipeline sees the time it arrived. Mobile clients, retries, network partitions and replays mean these diverge routinely, and a window that closes on arrival time silently misattributes late events. Deciding which clock a calculation uses, and how long a window waits for stragglers, is the first real design decision in a stream.
- 03
Delivery semantics are a design choice, not a default
At-most-once loses records under failure, at-least-once duplicates them, and exactly-once is achieved only within specific boundaries and at a cost. Most estates are best served by at-least-once delivery with idempotent consumers, because that combination survives replay without requiring the whole path to participate in a transaction.
- Duplicate handling designed into the consumer, not assumed away
- Idempotency keys agreed with the producing system
- Replay treated as a normal operation rather than an incident
- 04
State is the part that makes streaming operationally different
Aggregations, joins and deduplication windows hold state, and that state has to survive restarts, rebalances and version upgrades. Checkpoint frequency, state store size and recovery time are the operational properties that decide whether a stream can be maintained. A stateless stream is a straightforward service; a stateful one is a database with a different name.
- 05
Design the serving layer the consumer reads from
A stream is not itself a query interface. Dashboards, applications and APIs need a store with the right read pattern, and the choice between a streaming table in the warehouse, an operational store and a cache changes both latency and cost.
- Read pattern and concurrency profiled before the store is chosen
- Boundary agreed between the streaming path and the analytical platform
- Backfill of the serving layer from history designed up front
- 06
Reconcile the stream against the batch truth
Streams drift. Late events, schema changes and partial failures all cause divergence from the system of record, and it usually goes unnoticed because nothing compares them. A scheduled reconciliation against the batch or source-of-record figures, with a threshold that raises an alert, is what keeps a real-time number trustworthy.
- 07
Operating a stream is a standing commitment
Consumer lag, partition skew, rebalances, broker capacity and schema evolution are continuous concerns rather than launch-day ones. Someone has to own lag thresholds, the on-call path and the schema registry, and that ownership should be agreed before the first topic is created rather than discovered at the first incident.
Delivery model
How the work is sequenced.
The first phase can end in a recommendation not to build a stream. That is a legitimate and comparatively cheap outcome.
| Phase | Activities | Decisions made here | What you keep |
|---|---|---|---|
| Latency caseEstablish whether streaming is warranted. |
|
| A latency requirement and a recorded architecture decision |
| Stream designDecide time, delivery, state and serving. |
|
| A stream design with stated semantics and a serving choice |
| Build and reconcileImplement, then prove it agrees with the record. |
|
| A tested pipeline with reconciliation evidence |
| OperateHand over something that can be run at 3am. |
|
| Monitoring, runbooks and a named operational owner |
Technology fit
What each technology is here to do.
Streaming technology by role. The transport is the most visible choice and rarely the most consequential one.
Transport
- Kafka
- Managed event services
- Change data capture
Durability, replay and the number of independent consumers decide this. Managed services are often sufficient and much cheaper to operate.
Processing
- Flink
- Spark Structured Streaming
- Platform-native streaming
Where windowing, joins and state live. Whether the team can operate stateful processing matters more than throughput benchmarks.
Platform detailServing
- Streaming tables
- Operational stores
- Caches
- APIs
Chosen from the consumer read pattern, because a stream is not itself a query interface.
Contracts
- Schema registry
- Data contracts
- Versioning
A stream has producers you do not control. A schema change without a contract is a silent outage downstream.
Operations
- Lag monitoring
- Replay procedures
- Reconciliation jobs
Streaming failure modes are continuous rather than scheduled, so monitoring is part of the build.
Platform detail
Technology marks belong to their respective owners. No partnership, certification or endorsement is implied.
Engineering concerns
What has to be answered before it is trusted.
What a streaming pipeline has to answer before an application is allowed to depend on it.
- Ordering
- Whether order matters per key, and what the partitioning strategy guarantees when it does.
- Replay
- A rehearsed procedure to reprocess from a chosen point, with retention long enough to make it possible.
- Schema evolution
- A registry and a compatibility policy, so a producer change does not break consumers silently.
- Backpressure
- Defined behaviour when a consumer falls behind: buffer, shed or slow, chosen rather than discovered.
- Reconciliation
- Scheduled comparison against the system of record, with a threshold that raises an alert.
- Standing cost
- Continuous compute has a cost floor. It belongs in the comparison against batch before the design is fixed.
Where it applies
Related work, and why it is related.
Where a latency requirement is real, and what this connects to.
- Data engineeringThe parent service. Ingestion, modelling and quality work the same way whether the path is batch or streaming.
- LogisticsDispatch and exception handling are decisions that lose value in minutes, which is the clearest case for a stream.
- Retail and ecommerceInventory availability during a promotion is one of the few retail measures where batch latency is genuinely too slow.
- Managed data and AIConsumer lag and replay are continuous operational concerns, so streams are often run under a managed arrangement.
Agree what happens after a late event arrives.
A low-latency display can still show a provisional answer. In this hypothetical fulfilment flow, a warehouse scanner buffers shipment events while disconnected. When it reconnects, the events belong to an earlier business period even though the stream processes them now. The design needs a correction path as well as a fast first result.
Scope and assumptions: Set the event timestamp, clock assumptions, window rules and consumer behavior for the actual system. A watermark describes event-time progress; it is not proof that no older event can ever arrive. Use sanitized fixtures to check the selected engine's configured behavior.
| Dimension | Decision to establish | Acceptance evidence |
|---|---|---|
| Time basis | Separate when the shipment occurred, when the source recorded it and when processing received it. Define which timestamp assigns the reporting period. | Include an offline-device fixture and a business-day boundary case. Retain the input timestamps and the expected period, including timezone assumptions. |
| First result | Decide whether a consumer sees a provisional total, a delayed total or a last-complete value. Explain that state through the serving interface. | Observe what the actual consumer reads before the buffered event arrives. Check that an apparently final label does not conceal an incomplete period. |
| Late arrival | Choose how the processing design handles an event beyond its configured lateness boundary: update retained state, route it for correction or handle it through reconciliation. | Exercise the chosen route with a known late event. Capture the updated result or the retained exception and its accountable owner; do not silently discard it. |
| Consumer change | Define whether a revised total replaces a prior value or creates a separately identified correction. Consider consumers that already acted on the first result. | Check the reporting consumer and any downstream action separately. Replacing an analytical total must not accidentally repeat a fulfilment instruction. |
| Final comparison | Compare the corrected period with an agreed complete source extract. Preserve both the correction history and the rule that closes the period. | Attach the source cutoff, unmatched-event disposition and consumer acceptance. Reconciliation completeness needs evidence independent of a healthy stream processor. |
At narrow widths, scroll the worksheet to read every column. With the table focused, use the arrow keys to scroll.
Worked decision: the buffered shipment belongs to yesterday's reporting period. Updating today's total merely because it arrived today would preserve transport throughput while changing the business meaning. Route the event according to the agreed event-time and correction rules, then verify the analytical total and any triggered action independently. This is an illustrative acceptance decision, not a measured deployment result.
- Keep the state-retention boundary and the correction route together. If the required period is no longer retained, a restart alone cannot establish that the historical total has been repaired.
- Specify how operators identify unresolved late events and how consumers distinguish provisional, corrected and complete results. Product labels should describe the data state rather than imply a latency guarantee.
Before selecting the transport, compare the source and consumer needs in the batch, CDC and event-streaming decision guide. A faster delivery mechanism cannot supply a missing business timestamp.
Official references for this example:
Questions
Questions worth asking.
How do we know streaming is justified?
Name the decision the data supports and describe what changes at one second, one minute and one hour of delay. If the business action is identical across all three, batch or change data capture will serve it at materially lower operating cost. Streaming is justified when the value of the decision measurably decays inside the batch interval.
Is Kafka always the answer?
No. Kafka is a strong fit for durable, replayable, multi-consumer event streams, and it brings operational responsibility with it. Managed cloud event services, change data capture into a warehouse, or a streaming table in the platform you already run are often sufficient and much cheaper to operate. The volume, the number of independent consumers and the replay requirement decide it.
Can we get exactly-once delivery?
Within specific boundaries, yes, and it is usually the wrong thing to optimise for. Exactly-once guarantees hold inside a processing framework and its supported sinks, not across an arbitrary chain of systems. At-least-once delivery with idempotent consumers gives the same practical outcome, survives replay, and does not require every component in the path to participate.
What happens to events that arrive late?
That is a decision you make rather than a property you inherit. A window can wait a defined period for stragglers, emit an early result and update it, or route late events to a separate correction path. Each option changes what a consumer sees, so it is agreed with the consumer before the pipeline is built.
How does a stream fit with our existing warehouse?
Usually as a parallel path with an explicit boundary: the stream serves the low-latency consumer, the warehouse remains the system of record for analysis, and a scheduled reconciliation compares them. Problems arise when the boundary is left implicit and two numbers with different definitions both get called real time.
What does it cost to run once it is built?
A stream consumes compute continuously rather than in scheduled bursts, so its cost profile is a floor rather than a curve. Brokers or managed topics, always-on processing, the state store and the serving layer all carry standing cost, and that total is worth comparing against the batch alternative before the design is fixed.
Who operates it after handover?
Consumer lag, partition skew, schema evolution and broker capacity need a named owner and an on-call path. We build the monitoring and runbooks as part of delivery and rehearse a failure before handover; where run capacity is the constraint, operations can be scoped as a managed arrangement instead.
Next step
Test the case for streaming.
Bring the decision and how fast it decays. That conversation decides whether streaming is warranted at all.