Architecture

Failure Scenarios and Error Handling

The integration flow consists of multiple steps, each requiring different failure characteristics and recovery strategies. The integration sequence diagram illustrates these failure scenarios.

integration sequence diagram

Insurance System

Failure during event creation in step 1 does not lead to inconsistencies. The atomicity property of the database transaction ensures that business data and the corresponding event are committed together or not at all.

Event Producer / Outbox Publisher (Steps 2-4)

  • Steps 2 or 3 failures (polling outbox, publishing to Kafka) do not change system state and support retries.

  • Step 4 failure (marking event as processed) creates duplicate messages through event reprocessing on the next poll.

  • The FS-CD Adapter’s idempotent consumer pattern prevents duplicate messages from causing data inconsistencies.

Message Broker (Kafka)

Kafka failures can affect steps 3, 5, 9, or 10. Kafka’s distributed architecture and replication ensure high availability. Temporary Kafka outages result in message delivery delays but not message loss.

FS-CD Adapter (Steps 5-10)

  • Steps 5-8 failures (consuming message, loading data, transformation, calling FS-CD) do not change system state and support retries.

  • Steps 9-10 failures (acknowledging message, error routing) may cause duplicate processing, but the idempotent consumer implementation prevents inconsistencies.

Transient and Persistent Failures

Failures can be categorized into two types, requiring different resolution strategies:

Type Characteristics Examples Resolution Strategy

Transient

Temporary, non-deterministic, self-healing

Network timeouts, temporary database locks, SAP system overload

Automatically resolved by reprocessing after a delay

Persistent

Permanent, deterministic, reproducible, not self-healing

Software bugs, configuration errors, malformed event payloads, missing data

Requires manual analysis and code/configuration fixes before reprocessing

Retry Mechanism and Error Handling

The FS-CD Adapter implements a multi-layered error handling strategy:

Exponential Backoff Retry

When transient errors (network issues, temporary SAP unavailability) occur in steps 6, 7, or 8, the FS-CD Adapter uses exponential backoff retries to handle failures automatically:

  • First retry after a short delay (e.g., 60 seconds)

  • Subsequent retries with progressively longer intervals (e.g., previous interval times 5: 300s, 1,500s)

  • Maximum number of retry attempts, initial interval and the multiplier are configurable in class KafkaConfig.

Dead-Letter Topic

When retry attempts exceed the maximum or a persistent error occurs (e.g., validation error), the system publishes the message to a dead-letter topic for manual analysis.

Dead-letter topic workflow:

  1. Failed message is published to dead-letter topic with error details

  2. Operations team is notified (via monitoring/alerting)

  3. Root cause analysis is performed

  4. If persistent error: Code or configuration is fixed

  5. Message is reprocessed after fix is deployed

  6. If transient error: Message is reprocessed once the issue is resolved