Scalable Analytics Platform on Cloud

An analytics platform that will scale with data growth

A major financial services corporation’s principal business is to process payments between the banks of merchants and the card issuing banks or credit unions, which leverage the brand’s debit or credit services. As part of the financial inclusion initiatives globally, the client in partnership with financial institutions, governments & NGO’s, is leveraging its core digital technologies to advance agriculture, transform informal to formal financial distribution channels and increase the reach of health service by providing visibility into the overall value chain.

The client needed an analytics platform that will scale with data growth from identified sources, handle data ingestions processes, transform data, and continue to ingest new data sources to develop new program level KPI’s & generate insights on how each program is performing.

Business Challenge

• Existing analytics process is not robust and scalable, and is time-consuming.

• Nil or inconsistent view of data / insights; prone to manual errors.

• Existing platform can’t support the growing external reporting needs and the user community with program maturity.

• Dependent on the merchants & banks to share the information systems.

Tools and Technology Stack

• Data LakeStore

• SQL DW

• Data Factory V2

• SQL Server

• Databricks

• Power BI

• Python

• R

• .NET

• ANGULARJs

• Visual Studio Team Services

Approach

• Data & ETL: Move the data from on-prem using ADF V2 to Azure Data Lake Store and perform the ETL operations using Azure Data Lake Analytics using U-SQL & SQL Stored procedures. Perform the data orchestration leveraging the ADF V2 & load to SQL DW.

• Visualization: Rewrite the canned reports using Power BI

• User Portal : A user portal is developed using MS.Net and Angular.js to render the power Bi reports and display the KPIs based on the user role access

• Data Science : Data scientists provided with the capability of leveraging Azure Databricks service to perform advanced analytics & ad-hoc querying

• Security :Leveraged Azure AD service with Service Principal MFA and authorization maintained in

• DevOps: Leverage MS VSTS to create a CI/CD pipeline to achieve end-to-end automated build and deployment

• Monitoring:Use Azure monitoring service to monitor the service and infrastructure health and availability

Transformational Effects

• Data Lake with Azure Analytics was unique and instrumental in engaging business upfront & throughout the solution.
• End-to-end automated deployment process leveraging DevOps increased the frequency of the data availability to the business. Also, the platform provided robust environment scalability with better performance results.
• Comprehensive data security framework by leveraging cloud based encryption techniques helped secure the PII and other sensitive data. Actionable insights from data were generated using PowerBI with different user access roles.