Designing Durable Salesforce CDC Processing with MuleSoft

Salesforce Change Data Capture (CDC) is an effective way to react to record changes without repeatedly polling Salesforce. But consuming CDC directly into business processing creates an architectural risk that becomes more important as event volume grows.

Salesforce retains Change Data Capture events on the event bus for 72 hours. That retention window is valuable for reconnecting a subscriber and replaying events after a short outage, but it should not be treated as the long-term recovery store for an enterprise integration platform.

If a consumer is unavailable for too long, downstream processing becomes slower than event production, or a large burst creates days of backlog, the integration can eventually reach the edge of the replay window.

A more resilient design separates two responsibilities:

1. Receive Salesforce events quickly and durably capture them.
2. Process the captured events at the speed the downstream landscape can safely sustain.

That separation changes the CDC subscriber from a full business-processing pipeline into a durable ingestion boundary.

This article develops that pattern, including replay checkpointing, sequencing, parallelism, backpressure, idempotency and error recovery with MuleSoft.

The Core Problem

Consider a Salesforce org producing a continuous stream of Account, Opportunity and Order changes.

A simple consumer might look like this:

Salesforce CDC
     ↓
MuleSoft subscriber
     ↓
Transform
     ↓
Call downstream APIs
     ↓
Update database

This looks straightforward, but the subscriber's ability to keep up is now coupled to every downstream dependency.

If a downstream API slows from 100 ms to 5 seconds, the CDC consumer slows with it. If the API is unavailable for several hours, the subscriber may accumulate a large replay backlog. If failures continue long enough, events can eventually age beyond Salesforce's guaranteed retention period.

The architecture is using the Salesforce event bus as if it were the integration platform's durable work queue.

It is better to treat the event bus as the source event stream, not as the only durable recovery mechanism.

The Design Principle: Durable Handoff Before Business Processing

A stronger design introduces a durable boundary immediately after event consumption.

                      ┌──────────────────────────┐
Salesforce CDC ──────►│ MuleSoft CDC Ingestion   │
                      │                          │
                      │ Decode + validate        │
                      │ Persist durable envelope │
                      │ Advance replay checkpoint│
                      └────────────┬─────────────┘
                                   │
                                   ▼
                         Durable Event Journal
                         / Queue / Log / Store
                                   │
                    ┌──────────────┼──────────────┐
                    ▼              ▼              ▼
              Worker 1        Worker 2        Worker N
                    │              │              │
                    └──────────────┼──────────────┘
                                   ▼
                         Downstream Systems
                                   │
                                   ▼
                           Error Hospital

The ingestion flow should do as little work as practical before the durable handoff.

Its responsibility is primarily:

receive → validate envelope → persist → checkpoint

The worker layer performs the slower work:

read durable event → enrich/transform → invoke target → record outcome

The important architectural consequence is that a temporary downstream outage no longer prevents the integration from continuing to consume Salesforce events, provided the durable layer has capacity.

Why the 72-Hour Window Still Matters

Durable ingestion does not make Salesforce replay unnecessary.

The replay mechanism still protects the gap between Salesforce and the ingestion service.

For example:

Salesforce
   │
   │ events not yet durably captured
   ▼
CDC subscriber
   │
   │ events durably captured
   ▼
Integration-owned durable store

There are now two recovery domains:

Failure location	Recovery mechanism
Before durable capture	Salesforce replay within the retention window
After durable capture	Integration-owned durable store and retry/reprocessing

That is a much stronger model than asking Salesforce replay to recover every failure in the entire downstream pipeline.

Where Should the Replay Checkpoint Move?

Checkpoint timing is one of the most important decisions in the design.

The checkpoint should represent:

The event position that has been safely handed off to durable integration-owned storage.

It should not necessarily represent completion of all downstream business processing.

Consider event E100.

E100 received
   ↓
E100 persisted durably
   ↓
checkpoint E100
   ↓
E100 processed downstream later

If MuleSoft crashes after the durable write but before downstream processing, the event remains available from the durable store.

If the checkpoint were advanced before the durable write, a crash could create data loss.

E100 received
   ↓
checkpoint E100       ← dangerous
   ↓
application crashes
   ↓
E100 was never persisted

The replay position has moved beyond an event that the integration does not actually own.

The safe rule is:

Advance replay state only after durable handoff succeeds.

Salesforce replay IDs should also be treated as opaque stream positions rather than numbers to increment or derive.

What Should Be Persisted?

It is tempting to store the complete CDC payload indefinitely, but that is not always the correct design.

Payload retention can create concerns around:

• sensitive or regulated fields;
• data residency;
• encryption and access controls;
• storage growth;
• retention requirements;
• stale data being replayed much later.

A durable event record can instead contain an intentionally designed envelope such as:

{
  "eventKey": "generated-integration-id",
  "topic": "/data/AccountChangeEvent",
  "replayId": "opaque-value",
  "recordIds": ["001..."],
  "changeType": "UPDATE",
  "commitTimestamp": "...",
  "sequenceKey": "001...",
  "receivedAt": "...",
  "status": "READY",
  "attemptCount": 0
}

Then choose one of three payload strategies.

Strategy 1: Persist the Event Payload

Use this when exact event content must be reproducible and policy allows it.

CDC event → durable event + payload → later processing

The advantage is deterministic reprocessing from the captured event.

The cost is greater storage and governance responsibility.

Strategy 2: Persist Identifiers and Re-Fetch Salesforce State

Store enough metadata to identify the changed record, then retrieve current source state when processing or retrying.

CDC event
   ↓
record identifier + metadata
   ↓
worker fetches Salesforce record
   ↓
process current state

This can reduce payload-retention concerns and can be useful when the downstream integration needs the latest state rather than the historical event snapshot.

But it changes semantics: the state fetched during retry may be newer than the state that originally generated the event.

Strategy 3: Hybrid

Persist selected fields plus identifiers.

This is often a practical compromise.

The correct option is a design decision, not a universal rule.

Sequencing: What Order Do You Actually Need?

Salesforce's event bus is time ordered, but an integration can easily destroy that ordering after receiving the events.

For example:

Account A update #1
Account B update #1
Account A update #2

If three Mule workers process these concurrently, completion might occur as:

Account A update #2
Account B update #1
Account A update #1

That may be harmless for some targets and incorrect for others.

Before designing concurrency, define the required ordering scope.

Global Ordering

Every event must be processed in exactly the same order in which it was consumed.

E1 → E2 → E3 → E4

This is the most restrictive model and dramatically limits parallelism.

Very few integrations actually require global ordering.

Per-Record Ordering

Changes for the same Salesforce record must remain ordered, while unrelated records may process concurrently.

Account A: A1 → A2 → A3
Account B: B1 → B2

A and B may run in parallel.

This is much more scalable.

The Salesforce record ID can often serve as the sequencing key.

Per-Business-Aggregate Ordering

Sometimes ordering must be broader than a single record.

For example, Opportunity, OpportunityLineItem and Quote events might all affect one downstream aggregate.

Then the sequencing key may be:

OpportunityId
OrderId
AccountId
ContractId

rather than the changed record's own ID.

The key architectural question is:

What is the smallest scope within which order must be preserved?

Smaller sequencing domains create more safe parallelism.

Partition for Ordering, Parallelize Across Partitions

Once the sequencing key is known, the durable layer can partition work by that key.

Conceptually:

hash(sequenceKey) % N

produces a worker partition.

Account A ──► Partition 2 ──► sequential processing
Account B ──► Partition 7 ──► sequential processing
Account C ──► Partition 2 ──► queued behind earlier partition work
Account D ──► Partition 4 ──► sequential processing

Across partitions:

parallel

Within a sequencing partition:

ordered

This gives the system bounded concurrency without allowing unlimited parallel processing to reorder related changes.

A Kafka-style partitioned log is one implementation of this pattern, but the principle is technology independent. A database-backed work table, durable queueing platform or other message infrastructure can implement equivalent ownership and sequencing semantics if designed carefully.

MuleSoft Should Separate Ingestion from Workers

A useful MuleSoft implementation separates applications or flows by responsibility.

CDC Ingestion Flow

Salesforce Pub/Sub Connector
        ↓
Decode CDC event
        ↓
Extract event metadata
        ↓
Determine sequencing key
        ↓
Write durable record/message
        ↓
Persist replay checkpoint

Avoid expensive enrichment and downstream calls here.

The ingestion path should be optimized for capture throughput.

Processing Worker

Durable store / queue
        ↓
Acquire event
        ↓
Idempotency check
        ↓
Optional source-system re-fetch
        ↓
DataWeave transformation
        ↓
Target API/database
        ↓
Mark success

Now the number of workers can be tuned independently from the Salesforce subscription.

Use Salesforce Pub/Sub Flow Control Intentionally

Pub/Sub API uses a pull-based model. The subscriber controls how many events it requests according to the capacity it has to receive them.

That should become part of the backpressure strategy rather than simply requesting the maximum possible amount all the time.

Conceptually:

Durable queue healthy
      ↓
request normal/high event batch

Durable queue approaching threshold
      ↓
request smaller batches

Durable persistence unhealthy
      ↓
stop requesting additional work

The current Pub/Sub API allows up to 100 requested events per Subscribe/ManagedSubscribe call across requests. MuleSoft's Salesforce Pub/Sub Connector also exposes event batch sizing so the subscriber can balance memory usage and server-call frequency.

The correct batch size should be tested under actual event size and persistence latency rather than chosen only for maximum throughput.

How to Increase Processing Throughput Safely

Once events are durable, there are several independent levers.

1. Increase Worker Parallelism

If downstream systems can handle it, run multiple processing workers.

But concurrency should remain bounded.

more concurrency ≠ automatically more throughput

Excessive concurrency can create connection-pool exhaustion, API throttling, database contention and retry storms.

2. Parallelize Across Independent Sequencing Keys

This is usually safer than parallelizing every event indiscriminately.

Account A events → ordered worker lane A
Account B events → ordered worker lane B
Account C events → ordered worker lane C

3. Batch Compatible Downstream Operations

If the target supports bulk operations, several independent events can sometimes be converted into one request.

For Salesforce-originated events targeting another bulk-capable system:

100 individual API calls

may become:

1 bulk request containing 100 records

Do this only when batching does not violate ordering, transaction or error-isolation requirements.

4. Reduce Repeated Enrichment

If several events require the same reference information, cache stable reference data where appropriate rather than repeatedly calling the same system.

5. Keep the Ingestion Tier Fast

The easiest way to survive event bursts is to ensure that business-processing latency does not slow event capture.

Idempotency Is Mandatory

Replay and distributed processing imply duplicate delivery is possible.

For example:

persist event
↓
worker updates target
↓
worker crashes before marking success
↓
worker receives event again

Without idempotency, the same logical change may be applied twice.

An idempotency key might derive from a combination such as:

topic + replay/event metadata + record ID + change identity

or from a durable integration-generated event key established during ingestion.

The exact key depends on the downstream operation.

For an upsert of current state, natural idempotency may already exist through an external ID. For a non-idempotent command such as "create payment," stronger deduplication is necessary.

Integrating the Error Hospital Pattern

The durable CDC architecture works particularly well with an Error Hospital because a failed downstream event no longer has to block ingestion of new Salesforce events.

The processing flow becomes:

Durable Event
     ↓
Processing
  ┌──┴───────────────┐
  │                  │
Success            Failure
  │                  │
Mark complete      Error Hospital
                     │
                     ├─ retry same durable event
                     ├─ re-fetch source record and retry
                     ├─ manual repair and replay
                     └─ terminal disposition

The Error Hospital record should reference the durable event identity rather than depending only on the transient Mule message.

Useful metadata includes:

event key
Salesforce record ID
CDC topic
change type
replay ID
sequencing key
processing stage
target system
attempt count
first failure time
last failure time
error classification

Whether the original event payload is retained in the Error Hospital should remain a deliberate policy decision.

In some environments, storing payloads is useful for exact reproduction. In others, retaining source identifiers and re-fetching data from Salesforce during retry is safer from a privacy, storage or freshness perspective.

Do Not Let One Poison Event Stop a Whole Partition Forever

Strict sequencing creates a difficult failure case.

Suppose a partition contains:

A1 → A2 → A3 → A4

and A2 repeatedly fails.

If order is mandatory, processing A3 and A4 might be invalid.

There are several possible policies.

Block the Key

Pause only the sequencing key associated with the failure.

Account A blocked
Account B continues
Account C continues

This preserves per-key order without freezing the entire pipeline.

Move the Failed Event Aside and Continue

Use this only when later events can safely supersede the failed event or order is not semantically required.

Collapse to Current State

For some synchronization workloads, multiple historical updates can be replaced with a fresh read of the current Salesforce record.

For example:

A1 failed
A2 arrived
A3 arrived

Instead of replaying all three mutations, the recovery process may fetch the current Account state and upsert the target once.

That is not equivalent to event replay and is inappropriate when each intermediate transition has business meaning. But for state synchronization it can dramatically reduce recovery work.

Status Lifecycle in the Durable Store

A simple durable event lifecycle might be:

RECEIVED
   ↓
READY
   ↓
PROCESSING
   ↓
SUCCEEDED

with failures moving to:

RETRYABLE_ERROR
PERMANENT_ERROR
MANUAL_REVIEW

After successful processing, another design choice appears: should the durable record remain?

Retain Completed Records

Useful for auditability and operational investigation.

Retention can be time bounded:

30 days
90 days
1 year

based on requirements.

Clear Completed Records

Useful when the store is intended only as transient recovery infrastructure and another audit system already records outcomes.

Archive Then Delete

Move compact completion metadata to cheaper storage and delete operational payloads.

The correct choice depends on audit, privacy, storage and troubleshooting requirements.

Success should not automatically imply indefinite payload retention.

Recovery Beyond Salesforce's Retention Window

This architecture changes what happens during a long outage.

Without durable handoff:

Salesforce CDC retained 72 hours
         ↓
outage > retention
         ↓
possible unrecoverable event gap

With durable handoff:

Salesforce CDC
      ↓
Mule ingestion remains healthy
      ↓
Durable integration backlog
      ↓
downstream unavailable for days
      ↓
workers resume later from durable backlog

Salesforce's 72-hour retention is still the protection window for the ingestion subscriber itself, but downstream recovery is no longer limited to that same window.

For catastrophic ingestion outages that exceed the retention window, a separate reconciliation strategy is still required—for example querying source records using a trusted modification watermark and reconciling target state.

A resilient architecture should explicitly document that fallback rather than assume replay will always be available.

Monitor Lag in Two Places

Once the pipeline is split, there are two forms of lag.

Salesforce Ingestion Lag

latest Salesforce event position
        -
latest durably captured position

This tells you whether the subscriber is falling behind the event bus.

Processing Lag

latest durable event
        -
latest successfully processed event

This tells you whether downstream processing is keeping up.

Operational dashboards should expose both.

A system can have excellent ingestion health while processing is hours behind. That is preferable to losing events, but it still requires intervention before storage capacity or business SLAs are affected.

Useful metrics include:

CDC events received per minute
CDC events durably persisted per minute
oldest unprocessed event age
ready backlog count
processing throughput
retry count
Error Hospital count
per-target latency
per-partition backlog
subscriber reconnect count
checkpoint age

End-to-End Reference Architecture

Putting the pieces together:

┌─────────────────────┐
│ Salesforce          │
│ Change Data Capture │
└──────────┬──────────┘
           │ Pub/Sub API
           ▼
┌──────────────────────────────┐
│ MuleSoft CDC Ingestion       │
│                              │
│ • flow control               │
│ • decode / validate          │
│ • sequencing-key derivation  │
│ • durable write              │
│ • replay checkpoint          │
└──────────────┬───────────────┘
               ▼
┌──────────────────────────────┐
│ Durable Event Layer          │
│                              │
│ • event identity             │
│ • replay metadata            │
│ • sequencing key             │
│ • payload or source IDs      │
│ • processing status          │
└──────────────┬───────────────┘
               │
       partition / route
      ┌────────┼─────────┐
      ▼        ▼         ▼
 Worker A   Worker B   Worker C
      │        │         │
      └────────┼─────────┘
               ▼
┌──────────────────────────────┐
│ Target Systems               │
│ APIs · Databases · Platforms │
└──────────────┬───────────────┘
               │
        failure│
               ▼
┌──────────────────────────────┐
│ Error Hospital               │
│                              │
│ • classify                   │
│ • retry                      │
│ • re-fetch source            │
│ • manual repair              │
│ • audit/disposition          │
└──────────────────────────────┘

Design Decisions to Document Explicitly

A production implementation should answer these questions before deployment:

Decision	Question
Durable boundary	What exactly must succeed before the replay checkpoint advances?
Payload retention	Full event, selected fields, identifiers only, or hybrid?
Sequencing scope	Global, record, or business aggregate?
Partition key	How are related events guaranteed to enter the same ordered lane?
Concurrency	How many independent partitions/workers may process simultaneously?
Backpressure	When should the subscriber reduce or pause event requests?
Idempotency	How is duplicate processing detected or made harmless?
Retry	Which failures are retryable and for how long?
Poison event	Does one failure block a key, get skipped, or trigger state reconciliation?
Success retention	Keep, archive, compact, or delete completed event records?
Replay-window breach	How is source-to-target reconciliation performed after more than 72 hours?

Final Principle

The most important shift is conceptual.

Do not design the CDC pipeline as:

Salesforce owns the event until every downstream action succeeds.

Design it as:

Salesforce owns the event
        ↓
MuleSoft durably accepts responsibility
        ↓
Integration platform owns recovery
        ↓
Downstream processing proceeds independently

That durable handoff gives the integration platform control over backlog, sequencing, retries, parallelism, error recovery and retention rather than forcing all recovery to remain inside Salesforce's 72-hour event window.

For high-volume CDC integrations, that boundary is often the difference between a subscriber that merely works during normal traffic and a platform that can recover predictably from real production failures.