Data Enlights · Automotive
A lakehouse for an automotive group in Ghent
Terabytes of factory data in a medallion lakehouse, batch and streaming, from one config-driven framework.
- Automotive
- Data platform
- Streaming data
The problem
The group aims for 50% of revenue from non-fossil products, and its factories produce terabytes of data that needed one reliable platform.
The solution
A bronze/silver/gold lakehouse with a config-driven Python framework for both batch and streaming, and an Azure AD access model.
The outcome
About 50% lower cloud cost for the platform, mostly by optimising the streaming jobs, with batch and streaming running on one framework.
The full story
Where it started
The group's factories around the world produce terabytes of data from machines, production systems and warehouses. It needed one reliable platform for both batch loads and real-time streams.
The platform
A bronze/silver/gold lakehouse on Azure Databricks with Delta Lake. Changes in production databases arrive through Qlik Replicate and IBM CDC, live events through Event Hubs, and Databricks Autoloader picks up new files. One configuration-driven Python framework runs both batch and streaming, with access managed through Azure AD.
Halving the bill
We analysed the Azure and Databricks costs job by job. Most of the saving came from rewriting the streaming jobs so they do the same work with far less compute, which brought the cloud cost of the platform down by about half.
Governance
Unity Catalog and MLflow on Databricks govern the data and the models, including the machine learning models used in the warehouses.
Beyond the platform
The platform feeds logistics work too: a warehouse optimisation across four sites, and an application in development that predicts how long an order needs to be prepared and proposes a matching delivery slot, built with Azure Machine Learning and MLflow.
Tech
Ask Pai
Ask Pai about this case
Pai is an AI. It only knows what is written on this page.
Pai is an AI. Answers come from this case page. Don't share sensitive data.
Chats are kept for 30 days unless you switch that off in Privacy and choices.