Change data capture is a data integration pattern that detects inserts, updates, and deletes in a source database and sends those changes to other software systems. Rather than repeatedly copying an entire table, CDC moves only the changed records, often with low delay. Its design must account for ordering, duplicate delivery, deleted records, schema changes, and recovery after interruptions.
Change Data Capture
What Is Change Data Capture?
Change data capture, or CDC, detects inserts, updates, and deletes in a source database and sends those changes to other systems. Instead of repeatedly copying an entire table, CDC delivers changed records, often with low delay.
A change event usually includes a record key, operation type, changed values, and source position. Analytics stores, search indexes, caches, and application services use these events to update their data safely.
How Change Data Capture Works
CDC follows a committed database transaction to a destination system.
- A source database commits an insert, update, or delete.
- A CDC connector detects it through logs, triggers, queries, or snapshots.
- The connector creates an event with the record key, operation, and source position.
- A pipeline may transform the event using data mapping before delivery.
- The destination applies the event to a current-state table, history store, cache, or index.
- The consumer saves a checkpoint, or offset, to resume after an interruption.
Common CDC Methods
Methods differ in freshness, operational complexity, and production database load.
| Method | How it works | Best fit and tradeoff |
|---|---|---|
| Log-based | Reads transaction logs. | Low table-query load and detailed changes, but needs log access and retention. |
| Trigger-based | Database triggers write changes to audit tables. | Useful where logs are unavailable, but adds write-path overhead. |
| Timestamp or query-based | Polls rows changed since a saved time or ID. | Simple for basic updates, but can miss deletes or tied timestamps. |
| Snapshot comparison | Compares table copies over time. | Works for limited sources, but costs more for large tables. |
Benefits of Change Data Capture
CDC keeps data fresher without reloading everything.
- Synchronizes changed records incrementally across systems.
- Reduces work compared with frequent full-table reloads.
- Keeps analytics, caches, and search indexes closer to source data.
- Enables event-driven workflows and data sync automations.
- Can preserve an audit trail when events are retained.
Practical Limits of CDC
CDC improves delivery speed, but does not remove consistency engineering.
- Events can arrive twice or out of order during retries.
- Deletes, multi-row transactions, and schema changes need explicit handling.
- Short log retention can prevent recovery after a long outage.
- Change streams may expose sensitive fields to consumers.
- CDC alone does not guarantee an exactly-once business outcome.
Designing a Reliable CDC Pipeline
Reliable CDC needs clear producer and consumer rules.
- Use a stable primary key and retain the source position and operation type.
- Make destination writes idempotent so replaying an event has the same result.
- Store checkpoints durably and test replay and backfill procedures.
- Monitor delivery lag, errors, schema changes, and log-retention risk.
- Decide whether each destination needs current state, complete history, or both.
Teams can use an AI app builder such as Emergent to create interfaces and automations that use mapped, synchronized data responsibly.
Start Building
on Emergent today
on Emergent today
Start Building
