Model partial completion explicitly
A saga coordinates local transactions across a business workflow. When later work cannot complete, earlier effects may need compensating operations. The goal is an acceptable business state, not the illusion that every external effect can be rolled back.
Distributed transactions exist, but their availability and operational constraints depend on the participants. A database transaction cannot normally roll back an already sent email or an external payment simply because those calls originated in one monolith. Saga pattern.
An order workflow with honest states
Consider an illustrative order that reserves stock and authorizes payment before confirmation:
| State | Next action | Recovery concern |
|---|---|---|
| Order accepted | Reserve inventory | Duplicate reservation requests |
| Inventory reserved | Authorize payment | Request times out after provider acceptance |
| Payment outcome unknown | Reconcile provider status | Avoid a second authorization |
| Payment authorized | Confirm order | Persist confirmation before notifying |
| Cancellation required | Release or reverse applicable effects | Compensation can also fail |
The exact order depends on product and payment semantics. A failed email should not automatically refund a successfully completed purchase unless that is the intended business policy.
Choreography and orchestration
Choreography distributes reactions among services; orchestration makes workflow transitions explicit in a coordinator. Both need durable state, correlation identifiers, timeouts and clear ownership.
An orchestrator held only in an in-memory try/catch cannot recover after its process exits. Persist enough information to determine the next safe step after restart. Avoid a gap where an external action succeeds but the only record of that success is lost.
Where a local database change must lead to a message, a transactional outbox can record the intent in the same local transaction. A relay publishes it later. Consumers still need duplicate handling because publication and acknowledgement may be retried.
Idempotency crosses the network boundary
“Check whether a refund exists, then create one” is not sufficient. Two workers can both pass the check, and a process can crash after the provider accepts a request but before the database records it.
Create a durable identifier for the intended operation, enforce uniqueness locally and pass the provider's supported idempotency key on retries. Stripe documents that repeated requests with the same retained key return the original result, and that key retention and parameter matching matter. It is not an unlimited lifetime guarantee. Stripe idempotent requests.
After ambiguous failures, reconcile through the provider's supported status mechanisms. Never assume a timeout proves that no charge or refund happened.
Compensation needs its own operations plan
A reversal can incur fees, be delayed or be impossible after fulfillment. Define which effects are compensable, which require a forward correction and which need manual review.
Record every attempt, its outcome and the reason for intervention. Bound automatic retries and expose stuck workflows to an operator. A compensation failure must remain visible instead of being swallowed by another catch block.
Test crashes immediately before and after each durable write and external call. Test duplicate delivery and delayed events. These boundaries reveal whether the workflow can recover correctly; a successful happy path does not.
Sardar Mukhshaf
Full-Stack & DevOps Engineer
I build scalable web applications, secure backend systems, and production-ready cloud infrastructure. My engineering focus spans interface design, API architecture, deployment pipelines, and observability. Follow for more deep dives into system architecture, performance engineering, and emerging technologies.
More from the practice
Keep the useful ideas close.
Follow new articles with your preferred RSS reader.
Open the RSS feed ↗
