When you process creator payouts across hundreds of campaign accounts simultaneously, every batch run becomes an asynchronous minefield. On Curved Lab, our platform must atomically debit platform treasury balances, credit creator wallets, append immutable double-entry ledger entries, and trigger external bank transfers via Stripe Connect. In our initial launch, uncoordinated async execution led to silent lost updates, double-debits, and phantom payouts under concurrent load.
This article breaks down how we re-architected the payout pipeline from the ground up: replacing optimistic race conditions with MongoDB multi-document ACID transactions, isolating non-transactional external side-effects, and building an automated reconciliation loop.
The Concurrency Breakdown: Lost Updates & Race Conditions
The original payout dispatcher fetched pending payout rows and resolved them using standard Promise.all() loops. Without document-level transaction boundaries, two concurrent worker threads processing separate creator records frequently read identical platform balance snapshots before either had written their deductions back to the database.
The second write silently overwrote the first, producing a catastrophic lost update. The database reflected funds that had already been disbursed, leading to discrepancies between the platform ledger and the actual cash reserves held at the bank.
Engineering the ACID Isolation Boundary
To eliminate lost updates, we wrapped the entire database mutation sequence within a MongoDB replica set session using withTransaction(). Under multi-document ACID semantics, all balance checks, reservations, and ledger postings execute with snapshot isolation.
Rather than reading balances into Node.js application memory and calculating new balances locally, we implemented conditional atomic operators directly inside the query pipeline: combining $gte floor guards with atomic $inc decrements. If another thread mutates the document concurrently, MongoDB raises a transient WriteConflict error, which triggers an automated exponential backoff retry rather than corrupting ledger state.
The External Side-Effect Problem: Stripe Is Not in Your Database
A fatal design flaw in distributed systems is placing third-party HTTP calls inside a database transaction. If Stripe times out after 15 seconds, your MongoDB transaction times out or aborts. Worse, if Stripe successfully processes the transfer on their server but drops the network connection before returning a 200 OK, the database transaction rolls back while the real money has already left your bank account.
We solved this by establishing a strict Two-Phase Settlement Pattern:
- Phase 1 (Database Transaction): Atomically reserve funds and record the ledger intent in MongoDB. Zero external network calls.
- Phase 2 (Idempotent External Transfer): Execute the Stripe transfer using the deterministic MongoDB Payout
_id as Stripe's Idempotency-Key.
- Phase 3 (Final Settlement): On Stripe confirmation, transition the payout to completed and release the reserved fund hold.
Production Outcomes & Operational Metrics
By enforcing strict ACID boundaries and decoupling external API calls with deterministic idempotency keys, Curved Lab's payout pipeline achieved 100% financial accuracy across high-volume batch runs:
- Zero write conflicts: Over 40 concurrent payout threads process simultaneously with automatic backoff retries handling the rare 0.3% row collisions.
- Zero financial drift: The platform double-entry balance and Stripe bank balance match to the cent across every settlement cycle.
- Resilient network recovery: Network timeouts during Stripe transfers are automatically resolved by the reconciliation worker without human intervention.
