Back to portfolio
Shopify WebhooksIdempotencyTransactionsInventoryJob Queues

Backend Reliability Case Study

Building Reliable Shopify Webhook Pipelines

Shopify webhooks look simple at first: receive an event, read the payload, create an order. In production, the real challenge is reliability.

I built a Shopify order webhook pipeline that converts Shopify orders/create events into complete internal marketplace orders while protecting the system from duplicates, partial writes, and inconsistent inventory.

Key outcome

Reliable order ingestion under retries and concurrency.

The pipeline prevents duplicate orders, duplicate inventory deduction, partial writes, missing customer data, and inconsistent multi-vendor order records.

The Problem

A Shopify order is not just an order in this system. One webhook has to create or find the customer, map Shopify line items to internal products and variants, choose the right vendor, calculate discounts and shipping, create payment records, split the order across sellers, generate delivery records, update inventory, and notify vendors and customers.

If any one of those steps fails halfway, the database can become inconsistent. If Shopify retries the same webhook, the same order can be created twice. If two webhook requests arrive at nearly the same time, both can try to reserve the same stock.

The main goal was simple: process every valid Shopify order once, completely, and safely.

Pipeline Overview

The webhook endpoint is mounted under /api/shopify-webhook/shopify-order.

When Shopify sends an orders/create event, the handler behaves like a reliable order-ingestion pipeline instead of a simple HTTP controller.

  • Logs the webhook metadata.
  • Checks whether the Shopify order already exists.
  • Acquires an in-memory lock for that Shopify order ID.
  • Re-checks idempotency after the lock.
  • Starts a database transaction.
  • Creates or reuses the customer account.
  • Resolves products, variants, vendors, discounts, and inventory.
  • Creates order, payment, order item, order detail, and delivery records.
  • Deducts inventory only after all validations pass.
  • Commits the transaction.
  • Sends emails and push notifications asynchronously.

Idempotency First

The first reliability layer is idempotency. Before doing any heavy work, the system checks whether an order with the same shopifyOrderId already exists with a successful status. If it does, the webhook returns a safe response instead of creating duplicate records.

The pipeline also performs the same check again after acquiring a lock. This second check matters because another webhook request may have completed while the current request was waiting.

  • Fast duplicate detection before processing.
  • Race-condition protection after lock acquisition.

Handling Concurrent Webhooks

Shopify can send duplicate webhooks within seconds. To avoid two requests processing the same Shopify order at the same time, I added a per-order in-memory lock.

Each Shopify order ID gets its own lock. If another request for the same order is already running, the next request waits until the first finishes. Once the first request commits, the waiting request checks the database again and exits safely.

This prevents duplicate orders, duplicate inventory deduction, and duplicate vendor allocation.

Transaction-Based Order Creation

The most important part of the pipeline is the database transaction. If any step fails, the transaction rolls back. That means the system avoids half-created orders where payment exists but delivery does not, or inventory is deducted but order details are missing.

Inventory is updated near the end of the transaction, after product linking, vendor assignment, pricing, payment status, order items, and deliveries are all validated.

  • Customer account, if needed.
  • Main order record.
  • Payment record.
  • Order payment management record.
  • Order items.
  • Seller-specific order details.
  • Delivery records.
  • Inventory updates.
  • Hub inventory updates.

Hub-Aware Vendor Assignment

One of the more interesting parts of the pipeline is vendor selection. The system supports HUB and NORMAL order flows.

For hub-based fulfillment, the customer's latitude and longitude are taken from the Shopify shipping address. The system finds active hubs whose service radius covers the customer location. If the ordered product is available in one of those hubs, vendors are selected from hub inventory.

If no hub can fulfill the product, the system falls back to the normal vendor-selection flow. Vendor candidates are ranked by margin, and distance can be used as a tie-breaker. This lets the system balance business profitability with operational practicality.

Payment Awareness

The webhook pipeline understands different Shopify payment states, including prepaid orders, Cash on Delivery orders, and partially paid orders.

For each order, it calculates total product amount, shipping charges, discount amount, collected amount, remaining amount, and payment status. This data is saved into payment and order payment management tables, making the internal system aware of whether the order is paid, unpaid, or partially paid.

Background Jobs for Side Effects

Emails are not sent directly inside the transaction. Instead, they are pushed into a Postgres-backed job queue.

This keeps the webhook fast and prevents email failures from breaking order creation.

  • Persistent job records.
  • PENDING, PROCESSING, COMPLETED, and FAILED states.
  • Retry attempts.
  • Backoff delays.
  • Recovery of stuck jobs after server restart.

Delivery Webhooks

The backend also handles delivery updates from Shiprocket. The delivery webhook acknowledges the request immediately, then maps Shiprocket shipment statuses into internal delivery statuses.

It updates delivery records using AWB, channel order ID, or split order ID depending on what the webhook payload contains. When an order is marked delivered, the system can also trigger payment settlement updates.

What This Architecture Solves

This pipeline solves several real production problems across order ingestion, inventory consistency, payment awareness, and delivery synchronization.

  • Duplicate Shopify webhooks.
  • Concurrent webhook processing.
  • Partial database writes.
  • Incorrect stock deduction.
  • Missing customer data.
  • Multi-vendor order splitting.
  • Hub-based fulfillment.
  • COD and partial payment tracking.
  • Async email and notification delivery.
  • Delivery status synchronization.

What I Would Improve Next

The next hardening step would be strict Shopify HMAC verification before processing the payload. The current implementation logs webhook HMAC metadata, but production-grade webhook security should verify the raw request body against Shopify's shared secret.

I would also persist webhook events in a dedicated table with webhook ID, topic, payload hash, processing status, and retry metadata. That would make debugging, replaying, and auditing webhook events easier.

Conclusion

Building a reliable Shopify webhook pipeline is less about receiving a webhook and more about designing for failure.

The strongest parts of this implementation are idempotency, per-order locking, transaction-safe writes, inventory consistency, hub-aware vendor selection, and asynchronous side effects.

The result is a backend pipeline that turns a Shopify order event into a complete internal marketplace order without losing reliability when retries, concurrency, payment complexity, or delivery updates enter the picture.