Context
As part of an ERP migration, migrate 98 high-volume JSON business objects with complex structures (shops, warehouses, users, receipts, stocks...) from Azure to Snowflake to feed Qlik in near real-time.
Approach
Automatic generation of hundreds of SQL and Scala queries from specifications to denest complex objects. The system produced and feeds 150+ tables and views updated in real-time. ADLS→Snowflake ingestion via Kafka/ADF and orchestration via streams and tasks.
Impact
Minute-level feeds for BI teams processing millions of files per day. Transition to the new analysis system followed by deprecation of the legacy one.
Business objects
98
Latency
<1 min
What we delivered
- Automatic SQL generation from XML metadata to speed up deployments.
- Delta updates orchestrated by Snowflake Tasks with dedicated monitoring.
- Custom Scala functions to denormalize nested JSON.
- Coaching for analysts to leverage the new real-time feeds.
- Environment management via Snowflake API (test / pre-prod / prod).
Want to explore a similar use case?
Let's align on your next data, app or AI priority and draft an actionable roadmap in days.