Late-arriving data is a data engineering issue where records reach a software system after the event occurred or after a pipeline processed that time period. It can make dashboards, reports, and automated decisions temporarily incomplete or incorrect. The right response depends on the data's business importance, expected delay, and whether historical results can be revised.
Late-arriving data
What is late-arriving data?
Late-arriving data reaches a system after the event happened or after that period was processed. For example, a mobile app may record an event at 10:00 but upload it at 10:20 after reconnecting. The record is valid, but a dashboard, warehouse, or workflow may already have produced an incomplete result.
Event time vs. processing time
These timestamps help identify the delay. Late does not automatically mean wrong.
| Timestamp or term | Meaning |
|---|---|
| Event time | When the action actually occurred. |
| Ingestion time | When the system first received the record. |
| Processing time | When a pipeline transformed or stored the record. |
| Latency or lag | The delay between stages, such as an API response or delayed ingestion. |
| Late-arriving data | A valid record that arrives after an expected cutoff or closed processing window. |
Why is data delayed?
If your data is delayed, investigate the source and transport path first.
- Offline mobile devices may buffer events until a connection returns.
- Network retries and API latency can delay delivery.
- Batch exports may send records only on a schedule.
- Upstream outages, clock skew, and out-of-order messages can obscure the true sequence.
How to handle late-arriving data in a pipeline
A resilient pipeline preserves original records and applies corrections safely.
- Store raw records before transforming them.
- Keep event, ingestion, and processing timestamps.
- Set an allowed-lateness policy based on source delays.
- Deduplicate records using a stable event identifier.
- Update affected totals or derived results when late records arrive.
- Reconcile records beyond the cutoff through scheduled reprocessing.
A watermark is a pipeline's point of confidence that most earlier events have arrived. It is a practical cutoff, not proof that no older record will appear.
Late-arriving facts and dimensions
Warehouses handle delayed transactions and delayed reference information differently.
| Type | Handling approach |
|---|---|
| Late-arriving fact | Load the delayed transaction and recalculate affected aggregates. |
| Late-arriving dimension | Create a placeholder reference record, then reconcile it when full details arrive. |
| Historical attributes | Match the transaction to the attribute values valid at its event time, not merely current values. |
Practical limits and trade-offs
Teams choose between fast provisional output and slower, more complete output.
- Longer lateness windows improve completeness but delay results.
- Reprocessing historical partitions increases compute cost.
- Corrected records can change past dashboard totals.
- Duplicate events and unclear freshness labels can reduce trust.
Monitor lateness and design for correction
Measure record arrival time, track events beyond the threshold, and alert on unusual changes. Label dashboards with refresh time or revision status. Set the lateness window from source behavior and the cost of being wrong.
Late-arriving data in apps built with Emergent
With Emergent, teams can specify a data pipeline that preserves raw events, updates a database, and runs reconciliation tasks. They can build data monitoring dashboards that show refresh status and flag delayed records. Build resilient data apps with Emergent.
Start Building
on Emergent today
on Emergent today
Start Building
