Fabric & Data Lake Export Pipeline

The problem

Getting Dynamics 365 Finance & Operations data out of the application and into a lakehouse looks like a configuration exercise until you actually do it. Then you meet the real work.

There are thousands of tables. Nobody can tell you which forty actually matter for the reports finance signs off on. InventTrans, LedgerJournalTrans and SalesLine carry recIds and enum integers, not the labels your business users read on screen. Financial dimensions live in a combination table that has to be flattened before anyone can slice by cost centre. A change to a table's structure during a service update quietly breaks a downstream refresh at two in the morning. And the first time someone builds a Direct Lake report on raw staging output, the model is so wide and so unmodelled that the report is slower than the F&O form it replaced.

So the export lands, and six months later the lakehouse is a copy of the ERP rather than an analytics platform.

What ECOSIRE builds

ECOSIRE builds and deploys the pipeline that moves F&O data into your lakehouse and shapes it into something people can report on. The transport layer is chosen with you during scoping — link to Microsoft Fabric, export to Azure Data Lake Storage, or bring-your-own-database — because the right choice depends on your licensing, your data residency requirements and what your BI team already runs. What we build on top is the same either way: a curated, documented, incrementally refreshed model.

Table and entity selection

We work backwards from your reports. Rather than exporting everything and hoping, we select the tables and data entities that feed the questions you actually ask: order-to-cash, procure-to-pay, inventory movement and valuation, production, project accounting, or whichever domains are in scope. Every selected object is documented with its grain, its keys, its refresh behaviour and the reports that depend on it.

Bronze, silver and gold layers

Raw output lands in a bronze layer, untouched, so you always have the source of truth. A silver layer resolves the things that make raw F&O data unusable: enum integers translated to their labels, recId relationships resolved to readable business keys, DataAreaId retained so multi-legal-entity reporting is possible rather than accidental, and deleted-row handling applied so a deleted record does not linger forever in your reports. The gold layer holds the star schemas your reports bind to — conformed date, customer, vendor, item, site and warehouse dimensions, and fact tables at an agreed grain.

Financial dimensions, properly flattened

Ledger dimensions are unpacked into named columns — cost centre, department, business unit, project, whatever your dimension set uses — with the default dimension and the ledger dimension both handled. This single piece of engineering is usually the difference between a finance team that trusts the lakehouse and one that keeps exporting to Excel.

Incremental refresh

Full reloads of transactional tables are not viable past a certain size, and they are the main reason refresh windows creep past the start of the business day. We implement watermark-based incremental loading using change tracking, ModifiedDateTime, or the change-feed mechanism appropriate to the transport we selected. Late-arriving updates, back-dated postings and record deletions are all handled explicitly rather than assumed away, with a periodic reconciliation load to catch anything the incremental logic missed.

Schema drift handling

When Microsoft's service updates or your own extensions add, widen or rename a column, the pipeline detects the change against a stored schema contract, applies additive changes automatically, and raises an alert for breaking ones instead of failing silently at three in the morning. Schema versions are retained so you can see when a column appeared and why a historic figure looks different.

Orchestration and operations

Refresh is orchestrated on a schedule you control, with dependency ordering so gold never rebuilds from a half-loaded silver. Row counts, load durations, watermark positions and failures are logged to a monitoring table and surfaced as an operations report. Where F&O-side extraction jobs are involved, they run on the standard batch framework in their own batch group so they do not compete with month-end postings for capacity.

Reporting layer

We deliver a semantic model over the gold layer with relationships, measures, date intelligence and role-level security aligned to your legal entity structure. Where Direct Lake is the target, tables are partitioned and typed for it specifically — because a model that works fine in import mode can fall back to DirectQuery and lose the performance you moved to Fabric for. We build and hand over a starter set of agreed reports so the platform arrives with something people use on day one, not an empty workspace.

Optional write-back and enrichment

Where in scope, we can bring external data into the same model — storefront traffic, planning outputs, third-party logistics events — and, using Power Platform or the data management framework, feed curated results back into F&O as journal imports or forecast entries rather than leaving them stranded in a report.

Who this is for

Organisations running F&O that need analytics beyond what standard reporting delivers: consolidated reporting across legal entities, history retained longer than the application keeps it conveniently queryable, blending ERP data with non-ERP sources, or feeding a data science team. It suits companies that already have or intend to have a Fabric or Azure data platform and need the F&O side engineered properly.

How delivery works

1. Scoping call. We review your reporting requirements, your F&O version and topology, your legal entity structure, your existing data platform and your licensing position, then agree the domains and tables in scope.

2. Fixed quote. You get a written scope — named domains, named layers, named reports — and a fixed price before development starts. The listed price reflects a defined domain scope; adding domains changes the quote and we say so up front.

3. Build. ECOSIRE develops the extraction configuration, transformation layers, semantic model and any F&O-side X++ extension components. Lead time is typically two to four weeks from signed scope for a focused domain scope, longer for broad multi-domain programmes.

4. Install in test. The pipeline is deployed against your sandbox environment. We run a full initial load and several incremental cycles, then reconcile output figures against F&O reports with your finance and operations team until the numbers agree.

5. Production. After reconciliation sign-off, we deploy against production, run the initial historical load in a window that does not disrupt users, and hand over the refresh schedule.

6. Support. A support window covers defect fixes, refresh failures and configuration help after go-live. Additional domains and new reports are quoted as enhancement work.

This is built for your environment after a quote. There is no ready-made package to download.