Designing Async Catalog Upload Systems

Product approvals should feel fast for admins, but Shopify uploads can be slow and unreliable. Shopify API calls may fail because of rate limits, validation errors, duplicate products, network issues, or temporary platform errors.

If the admin approval API tries to upload everything synchronously, the request can timeout or leave the system in an unclear state.

The goal was to save product approvals immediately, then upload the catalog to Shopify asynchronously with tracking, retries, and monitoring.

The catalog upload system is built around two persistent models: ShopifyUploadBatch and ShopifyUploadProduct.

A batch represents one upload run. Each product or variant inside that batch gets its own tracking row. This makes the upload flow asynchronous, observable, and recoverable.

Shopify catalog upload is not a good fit for a single HTTP request. A batch may contain many products and variants, and each upload can require multiple Shopify API calls.

By moving the upload into a background processor, the admin approval flow stays fast. The API only needs to create a batch and queue the work. The actual Shopify sync happens separately through cron-based processing.

Each upload batch stores enough metadata to show the admin panel exactly what happened after approval. The batch can move through statuses like PENDING, PROCESSING, COMPLETED, FAILED, and CANCELLED.

Each product inside a batch has its own upload record. This is important because one batch can partially succeed. Instead of failing the entire batch blindly, the system can show exactly which products uploaded successfully and which ones need attention.

The processor groups upload rows by internal_product_id. This avoids creating duplicate Shopify products for every variant.

If a Shopify product already exists, the system adds new variants to that existing product. If no Shopify product exists, it creates a new Shopify product with its variants. This keeps the Shopify catalog cleaner and matches the internal product-variant structure.

The batch processor runs sequentially and waits between product groups. This protects the system from hitting Shopify rate limits too aggressively.

Instead of firing many API calls in parallel, it processes product groups carefully and adds a delay between uploads. That design favors reliability over raw speed, which is usually the right tradeoff for catalog sync systems.

When a product upload fails, the system categorizes the error. This makes failures easier to debug from the admin panel because a validation error needs different handling than a Shopify rate-limit error or network timeout.

Failed products can be retried. The retry API resets failed product rows back to PENDING if they are still under the retry limit. The cron processor then picks them up in the next run.

This avoids manual database changes and gives admins a clean way to recover from temporary upload failures.

Long-running background jobs can fail silently if the server restarts or a process crashes.

To handle that, the system checks for batches stuck in PROCESSING for more than 30 minutes. Those batches are automatically marked as failed, and their processing products are marked as failed too. This prevents uploads from staying in a forever-processing state.

The system includes monitoring APIs for listing batches, viewing a single batch summary, viewing products inside a batch, viewing a product upload detail, and retrying failed products.

The API also calculates useful metadata like processing time, success rate, status counts, and formatted error messages. This turns the async system into something admins can actually understand and operate.

Catalog upload is only one part of Shopify sync. The backend also includes scheduled inventory synchronization.

A daily cron job calculates approved available stock from internal inventory and updates Shopify inventory levels. This keeps Shopify stock aligned with the internal marketplace inventory.

The inventory sync also respects category-based Shopify store configuration, so products can be routed to the correct Shopify store setup.

This async catalog upload design solves several real production problems around admin speed, background processing, retry behavior, observability, and inventory consistency.

The next improvement would be making the batch processor multi-instance safe using a database-level lock or atomic claim query. The current processor prevents duplicate work inside one Node.js process, but distributed deployments need stronger locking.

I would also add structured logs and webhook-style notifications for batch completion, so admins can be notified when a large catalog upload finishes or fails.

Async catalog upload systems are about more than sending product data to Shopify. They need persistence, observability, retry behavior, rate-limit awareness, and recovery from stuck states.

This implementation turns product approval into a reliable background workflow. Admins can approve products quickly, while the backend safely handles Shopify product creation, variant updates, error tracking, retries, monitoring, and inventory sync behind the scenes.