Designing Safe Replay and Reprocessing in Integration Platforms

When an integration fails, teams often say: retry it. That phrase hides several different operations with very different risks.

A resilient platform should distinguish retry, replay and reprocessing.

Three Different Recovery Operations

Retry

Retry repeats a failed technical operation, usually within the same processing context.

Call target -> timeout -> wait -> call target again

Retry is appropriate when a failure is likely transient and repeating the operation is safe.

Replay

Replay consumes an event or durable message again from an earlier position.

Event stream
  101
  102
  103 <- resume here
  104

Replay is transport-oriented. It restores delivery, but it does not guarantee that repeating business effects is safe.

Reprocessing

Reprocessing deliberately starts business processing again for a failed item, often after the original execution context is gone.

Failure record -> operator/system selects -> fetch source -> process again

This may happen minutes or days later and therefore needs stronger controls.

The Central Risk: Duplicate Effects

Suppose an integration creates an invoice and times out before receiving the response.

Integration -> Billing: create invoice
Billing creates invoice
Response is lost
Integration sees timeout

A blind retry may create a second invoice.

The recovery design therefore begins with idempotency, not with the retry count.

Use Stable Business Identities

Where possible, downstream writes should use a stable external identity.

sourceOrderId = ORD-48392

Instead of treating every request as a new create, the target can recognize repeated processing for the same business operation.

Different operations may require different idempotency scopes. ORD-48392 might identify an order, while ORD-48392:SHIPMENT:1 identifies a specific shipment action.

Decide What a Failure Record Stores

One option is to retain the failed payload.

Advantages:

• exact failed input is available;
• replay can be independent of the source;
• useful for forensic analysis.

Risks:

• sensitive data duplication;
• stale data during later recovery;
• storage growth;
• retention and access-control requirements.

Another option is to store only identifiers and processing context:

{
  "recordId": "001xx000003ABC",
  "operation": "customer-sync",
  "failedStage": "target-upsert",
  "correlationId": "abc-123"
}

During recovery, the integration fetches current state from the source system again.

This reduces payload retention but changes semantics: reprocessing now uses current source state, not necessarily the exact state that originally failed.

Neither approach is universally correct.

Model Recovery as State

A recoverable item should have an explicit lifecycle rather than a boolean failed flag.

FAILED
  -> RETRYING
  -> RECOVERED

FAILED
  -> RETRYING
  -> FAILED

FAILED
  -> IGNORED

Additional states may include MANUAL_REVIEW, EXPIRED or NON_RECOVERABLE.

State transitions help prevent two workers or operators from reprocessing the same item simultaneously.

Automatic vs Manual Recovery

Classify errors.

Transient technical errors may be retried automatically:

timeout
connection reset
temporary service unavailable
rate limit

Data or business errors often require correction first:

invalid country code
missing required relationship
business rule rejection

Repeatedly retrying invalid data only creates load and noise.

Protect the Platform During Bulk Replay

Recovery traffic can become an incident of its own. If 500,000 records accumulated while a dependency was unavailable, releasing all of them at once may overwhelm the recovered dependency.

Use controlled recovery:

recovery backlog
      |
      v
bounded workers -> rate control -> downstream

Measure normal traffic and recovery traffic separately. In some systems, live business traffic should have priority over backlog recovery.

Ordering Changes the Problem

If events for the same entity must be processed in order, replaying one failed event independently may be incorrect.

Imagine:

1. Account created
2. Account renamed
3. Account closed

Reprocessing event 1 after event 3 can restore an obsolete state if the consumer applies events literally.

Options include entity-level ordering, version checks, current-state re-fetch, or state-based idempotent updates.

Recovery Beyond Event Retention

Event-stream replay windows are finite. A recovery architecture should not assume the source broker will retain events forever.

For critical integrations, consider a durable handoff or an independent recovery mechanism that can reconstruct work from source identifiers. The correct design depends on whether the integration is event-state based, command based or snapshot based.

Auditing vs Cleanup

After successful recovery, should the failure entry remain?

Two valid approaches are:

Delete/clear after success when the store is purely operational and audit history exists elsewhere.

Retain terminal status when recovery history is important for audit, analysis or support.

If entries are retained, establish archival and retention policies. An error store should not grow forever simply because cleanup was never designed.

Recovery Must Be Observable

Track at least:

• recoverable backlog count;
• oldest failed item;
• recovery throughput;
• recovery success rate;
• repeated failure count;
• items awaiting manual action;
• duplicate suppression events.

These metrics tell operations whether recovery is actually converging.

A Practical Decision Sequence

When designing failure recovery, ask:

1. Can repeating the operation create a duplicate business effect?
2. Is the error transient or deterministic?
3. Do we need the original payload or only a source identifier?
4. If we re-fetch, is current state acceptable?
5. Does ordering matter?
6. How much recovery traffic can the target tolerate?
7. What happens after the event-retention window expires?
8. Do we delete successful recovery entries or retain them for audit?

Final Principle

A retry mechanism is not a recovery architecture.

Reliable integration platforms deliberately separate immediate retries, transport replay and business reprocessing. They preserve enough context to recover safely, protect downstream systems during backlog drain, and make duplicate prevention part of the design rather than an afterthought.