Context
As part of an ERP migration, migrate 98 high-volume JSON business objects with complex structures (shops, warehouses, users, receipts, stocks...) from Azure to Snowflake to feed Qlik in near real-time.
Approach
Automatic generation of hundreds of SQL and Scala queries from specifications to denest complex objects. The system produced and feeds 150+ tables and views updated in real-time. ADLS→Snowflake ingestion via Kafka/ADF and orchestration via streams and tasks.
Impact
Minute-level feeds for BI teams processing millions of files per day. Transition to the new analysis system followed by deprecation of the legacy one.
Business objects
98
Tables and views
150+
Latency
<1 min
What we delivered
- Automatic SQL generation from XML metadata to speed up deployments.
- Delta updates orchestrated by Snowflake Tasks with dedicated monitoring.
- Custom Scala functions to denormalize nested JSON.
- Coaching for analysts to leverage the new real-time feeds.
- Environment management via Snowflake API (test / pre-prod / prod).
Want to explore a similar use case?
Let's align on your next data, app or AI priority and draft an actionable roadmap in days.