BigQuery / Data Warehouse Sync for ERPNext

We build a dedicated Frappe app — a proper module with its own DocTypes, hooks.py wiring and scheduler events — that keeps a cloud data warehouse continuously synced with ERPNext. A Warehouse Sync Settings single DocType holds the BigQuery project, dataset, service-account credentials (stored via Frappe's encrypted Password fields) and the global sync cadence. A child-table Warehouse Sync Mapping lets you select exactly which DocTypes to replicate and map individual fields, rename columns, cast types, or exclude sensitive fields. The engine does incremental change-data capture: rather than dumping full tables, it watches modified timestamps plus on_update/after_insert/on_trash doc events registered in hooks.py, so only changed rows are shipped since the last high-water mark. A bench scheduler event (configurable from every few minutes to hourly) batches the queued changes and loads them into BigQuery via the load-job/streaming API, writing soft-deletes as tombstone rows so your warehouse stays a faithful mirror.

On first run and whenever your ERPNext DocTypes change, the app auto-creates and evolves the target schema: it introspects each DocType's fieldtypes, generates the matching BigQuery table schema, and applies additive column changes automatically when you add a custom field — no manual DDL. Every sync run is recorded in a Warehouse Sync Log DocType (rows read, rows loaded, bytes, duration, errors) so you get a full audit trail, and a monitoring layer raises alerts through ERPNext Notifications / email and can post to a webhook when a run fails, lags, or a row count deviates from expectation. Access is governed by Frappe's own permission system and a dedicated role profile, and admin-only whitelisted methods (@frappe.whitelist()) let your team trigger a manual backfill or re-sync a single DocType from the ERPNext desk. The same engine targets Snowflake, Redshift or a plain Postgres warehouse — BigQuery is simply the most common request.

Because this is build-to-order, nothing is a shrink-wrapped download. We start with a short scoping call to confirm your warehouse, the DocTypes and fields in scope, the sync cadence and your ERPNext version (we support Frappe/ERPNext v15 and v16). We then build against your requirements, review it on a staging bench with your real DocType structure, and hand over the installed app plus the git repository. Typical delivery is 2–4 weeks from confirmed scope depending on how many DocTypes and downstream transformations are involved.