Data Platform Migration

A comprehensive solution for migrating the data platform for legal portfolio of a P&C insurer’s corporate IT to Azure public cloud

A major P&C Insurance company with operations across 18 countries in 900 locations globally wanted to phase out its existing, on-prem tools in use like Teradata, Informatica, Cognos, etc. to a cloud environment leveraging the managed services (PaaS) model for better cost and operations management.

Trianz was engaged to develop a comprehensive solution for migrating the insurer’s data platform for legal portfolio, part of the corporate IT, to Azure public cloud by re-creating the entire ETL process using native Azure services.

Business Challenge

• Dependency on environment ownership – Currently, Teradata & Informatics are owned by different groups.

• Data centralization – No single source of truth & multiple redundant tools, internal administrative dependencies and bottlenecks.

• Environment upgrades and patches – Regular environment upgrades, patches and restarts impacting application availability.

• Limited scalability and tuning – Scaling the current infrastructure on ad-hoc needs, which is expensive.

Tools and Technology Stack

• Data LakeStore

• SQL DW

• Data Factory V2

• U-SQL

• Databricks

• Python

• SQL Server

• Power BI

• R

• DataVision+

• Visual Studio Team Services

The Approach

• Data & ETL Migration: Move the data from on-prem using ADF V2 to Azure, Data Lake Store and perform the ETL operations using Azure Data Lake Analytics using U-SQL. Perform the data orchestration, leveraging the ADF V2 and load to SQL DW and replacing Teradata and Informatica.

• Data Validation: Automate QA with Trianz Data Vision+ to validate data on an on-prem vs. Data Lake Store on Azure.

• Data Science: Provide data scientists with the capability of leveraging the Azure Databricks service to perform advanced analytics and ad-hoc querying.

• Security: Leverage Azure AD service with Service Principal MFA.

• Visualization: Rewrite the canned reports using Power BI and replaced Cognos.

• DevOps: Leverage MS VSTS to create a CI/CD pipeline to achieve end-to-end automated build and deployment process.

• Monitoring: Use Azure Monitoring service to monitor the service and infrastructure health and availability.

Transformational Effects

• Data Lake with Azure Analytics approach was unique and instrumental in engaging business upfront and throughout the solution, full ownership of the environment with complete set of data centralized in Data Lake Store.
• Replaced Teradata, Informatica etc., with native Azure services. Robust environment scalability with better performance results.
• End-to-end automated deployment process leveraging DevOps to increase the frequency of the data availability to the business
• Advanced analytics and data science capabilities with Azure Databricks.