_How cloud microservices improved data extraction process from Windchill for luxury goods manufacturer

Summary

An esteemed luxury group – renowned for its exquisite high-end watches, jewelry, fashion, and accessories, with a global presence spanning luxury markets in Europe, Asia-Pacific, the Americas, and the Middle East – is using a fully custom framework for its in-house ETL (Extract-Transform-Load) process.

They extract data from Windchill daily, transform it into a predefined format, and store it in a custom external database, an interface for other departments for further processing and reporting. This process took a lot of time to complete and consumed many resources. Additionally, it was very error-prone. TT PSC has been assigned to fully refactor this framework to make it more reliable and performant.

Business Challenges

• Extraction jobs should be executed in parallel, allowing the use of the full power coming from the clustered environment. There is a timeframe in which all jobs should end every day. #UseEnvironmentFullPower
• Extractions should be as resilient, standalone, reliable, and as possible, with graceful error handling. #ResilientDataExtractions
• Daily execution of extraction jobs should have a manageable schedule. #ScheduleManagement
• Extracted data should be available in both the custom interface database and BigQuery service in the Google Cloud Platform. #DataAvailability

Solution

TT PSC performed a deep refactoring of the Extractions Framework, basing the whole project on several cloud microservices utilizing Event Driven Architecture principles.

With this new architecture, our customer, the dominant luxury fashion group, can run several extraction processes on each Method Server on each node in a cluster. This gives around 40 parallel jobs executed in a production environment. Each of them, as a standalone, monitored job, can manage their errors without impacting anything else.

TT PSC has used several modern technologies to achieve this solution. Each Method Server contains a client that consumes messages from an asynchronous message broker queue on which scheduled jobs arrive. Such jobs are then processed by the extractor engine. The schedule is produced by an external cloud microservice called Extractions Scheduler, written in the Spring Boot framework. A separate web application for managing this schedule is also created in the Vue framework. Additionally, for each successful extraction job, there’s a standalone service that exports the data to Google Cloud Platform storage.

Main results and advantages

Moving to the new architecture implemented by TT PSC resulted in an overall reduction in data extraction processing time from about 10 to just 3 hours, with more than 40 extraction jobs running daily. New asynchronous event-based scheduling significantly increased the resilience and overall stability of the platform, and the microservice approach allowed a more comfortable development process, reducing the possible regression and side effects.