Incremental Migration & Integration App
Self-hosted migration platform connecting any two data sources — Shopify, BigCommerce, Cin7 — with support for 90,000+ record migrations. Development stopped, project archived.
Overview
Built a self-hosted migration application that connects any two data sources (Shopify, BigCommerce, Cin7) and incrementally pushes data from source to target. The code is complete and designed for deployment on self-hosted infrastructure as a replacement for the existing serverless system — not yet deployed to production.
Source code: github.com/Tulay-Tech/conduit
The Problem
Clients switching between e-commerce platforms needed their data migrated — products, customers, orders, pricing — without downtime and without data loss. The existing process ran on serverless infrastructure, and combined with the custom pricing ETL system, was costing $1K–$7K/month in compute. Both this app and the pricing ETL system were proposed to run on the same self-hosted infrastructure at ~$400/month total.
Architecture
The system centers on an Express server running on a Digital Ocean Droplet — a single serverful application handling all ETL jobs and API orchestration.
How data flows:
- System A (sources) — Shopify shops, BigCommerce stores — sends data to the Express server via dynamic cron jobs per tenant
- The Express server transforms and loads data into a PlanetScale database with replication (local PostgreSQL for development)
- Transformed data is pushed to System B (targets) — new BigCommerce stores, new Shopify shops, Cin7 instances
Orchestration: A self-hosted Inngest server orchestrates the workflows, but the application server still runs the actual operations. Inngest handles scheduling, retries, and step-level durability — the Express server handles the business logic and API calls.
Frontend: A React SPA provides the management interface for configuring tenants, monitoring sync status, and triggering manual migrations.
Why Self-Hosted
The key insight was that migration and integration workloads are long-running and predictable — the opposite of what serverless is optimized for. A single Digital Ocean Droplet running the Express server could handle the same workload at a fraction of the cost because:
- No per-invocation billing for long ETL jobs
- No cold start penalties on workflows that run for minutes
- Inngest provides the durability guarantees (retry, resume) without needing serverless infrastructure
- PlanetScale handles the database scaling independently
Design Decisions
Adapter pattern: The application abstracts each data source behind a common interface — any source can push to any target. Adding a new platform requires one adapter, not N new integration paths.
Dynamic cron per tenant: Each tenant gets its own sync schedule rather than a global cron. This lets high-volume clients sync more frequently without affecting smaller tenants.
Incremental sync: Tracks what’s been migrated via cursors, so the system picks up where it left off. Supports both one-time migrations and ongoing integrations.
Scale
The system was designed to support 90,000+ record migrations with zero data loss — handling both one-time migrations (client switching platforms) and ongoing integrations (keeping two systems in sync).
Status
Development stopped and the project was archived — there was no internal adoption path. The codebase is complete enough to demonstrate the architecture and approach, but it was never deployed to production.