How to Build a Reconciliation System for High-Load AI Provider Webhooks
AI provider platforms run on two clocks: requests complete now, while providers calculate billing later. Under load, usage webhooks are delayed, duplicated, or missed. A webhook cannot be the final billing authority.
Use three paths: synchronous ledger accounting, a replay queue, and periodic reconciliation.
Synchronous accounting
Before returning 200, authenticate and validate the event, deduplicate by provider request ID, and commit the usage record plus atomic quota decrement in one transaction. Block locally when the raw balance goes below zero.
requestId must be unique; store nano-USD integers. Keep request history and operational quota separate; provider spend corrects drift.
Bounded negative balance
Zero is unsafe when concurrent or streaming requests finish after their preflight checks. Allow controlled debt, but separate:
- Enforcement: block below zero.
- Storage floor: cap the displayed balance, for example at
-$1.
The floor limits customer-facing exposure but must not hide debt used for alerts and reconciliation. A top-up can restore the balance; reconciliation repairs provider-side enforcement.
Fire-and-forget after commit
Fire-and-forget is for recoverable side effects: notifications, analytics, sampled checks, and provider calls backed by retries. It is never for the first usage write or quota decrement.
A robust implementation commits accounting first, then provider blocking runs. Next.js after() is bounded post-response work, not a durable queue. For critical side effects, write a transactional outbox row and retry it with a worker.
If one item in a webhook batch fails, returning 200 may prevent a retry. Persist the batch in an inbox or return a retryable failure when event IDs are idempotent.
Replay and reconcile
The provider usage API is the recovery path. A replay queue checkpoints work, retries safely, and deduplicates by request ID. Run it on a schedule with a lookback window for delayed usage.
Replay must not charge the same organization twice. If the webhook applied the operational deduction, replay fills missing history and reconciliation owns the final organization balance.
correct_balance = credits_paid - provider_cumulative_spendAn hourly job compares this value with the operational balance, corrects material drift, preserves raw debt, and repairs block state. Cumulative provider spend catches dropped callbacks and local ingestion bugs.
The operating blueprint
Make accounting idempotent and transactional. Store exact integer money. Allow bounded debt, but block on raw negative balance. Keep side effects after commit and use an outbox when they cannot be lost. Replay with checkpoints. Give each balance domain one owner. Reconcile against provider spend on a schedule.
The goal is not a perfect webhook. It is recoverability: webhooks stay responsive, replay recovers missing events, and reconciliation converges on provider truth.