Build a change data capture pipeline

---
name: build-a-change-data-capture-pipeline
category: data
description: Build a change-data-capture pipeline with source position, event identity, schema history, snapshots, ordering, tombstones, idempotency, reconciliation, and recovery. Use when downstream state must follow database changes.
---
# build-a-change-data-capture-pipeline
## When to use
- Use for replication, search, analytics, integration, or cache updates.
- Do not infer business intent from low-level changes without a contract.
## Procedure
1. Define source tables, keys, transactions, consumers, positions, and retention.
2. Capture snapshot and change stream under one consistent boundary.
3. Assign event identity, source position, transaction, schema version, and operation.
4. Preserve ordering, tombstones, before and after meaning, and large-transaction behavior.
5. Make sinks idempotent and record accepted source watermarks.
6. Handle schema change, backpressure, restart, gap, poison, and replay.
7. Reconcile source, stream, sink, and business invariants continuously.
## Failure plan
- Stop publication and recover from the last proven position when a gap or ordering loss appears.
## Worked example
A snapshot overlaps live changes but the recorded high-watermark prevents missing or duplicating customer rows.
## Done
- A CDC implementation records source boundary, identity, positions, schema, ordering, tombstones, sink outcomes, and recovery
- Snapshot, live overlap, transaction, schema, tombstone, duplicate, gap, restart, and reconciliation tests verify correctness