Context
As part of an ERP migration, the client needed to move 98 high-volume, complex JSON business objects (stores, warehouses, users, receipts, inventory, and more) from Azure to Snowflake to feed Qlik in near real time.
Approach
Hundreds of SQL and Scala queries were generated automatically from specifications to flatten complex objects. The system creates and feeds more than 150 tables and views updated in real time. Data is ingested from ADLS into Snowflake through Kafka and ADF, with orchestration handled by streams and tasks.
Impact
BI teams receive minute-by-minute data feeds while the system processes millions of files each day. The organization transitioned to the new analytics system and then deprecated the legacy platform.
What we delivered
- Automatic SQL generation from XML metadata to speed up deployments.
- Incremental updates orchestrated by Snowflake Tasks with dedicated monitoring.
- Custom Scala functions to denormalize nested JSON.
- Coaching for analysts to leverage the new real-time feeds.
- Environment management via Snowflake API (test / pre-prod / prod).
Critical work deserves more than generic software.
Start with a real use case, the stakeholders involved, and the available data to identify the next useful step.
FDTI / Contact
Let's keep it simple.
We only use your contact details to reply: no newsletters, no unsolicited outreach, and no commercial sharing with third parties.