• Developed end-to-end ML pipelines on Databricks for preprocessing, feature engineering, and model training using PySpark and Delta Lake.

• Developed a Universal Data Layer (UDL) leveraging AWS S3, Apache Iceberg and AWS Glue Catalog to enable a zero-copy data sharing mechanism.

• Developed features for Self-Serve Data Observability Service which automates configuration of Data

Reliability Policies – Data Quality, Data Recon, Data Drift and Schema Drift.

• Developed features for Self-Serve Data Publishing Service [Reverse-ETL] to publish data to external

systems.

• Developed CI/CD pipelines to automate deployment of DBT models across environments.

Technology Used: Java, Python, DBT, SQL, PySpark, Databricks, Apache Iceberg, Airflow, Spring Boot

• Developed features for Data Distributor – a config driven tool which processes the data in

real-time and batch.

• Developed a BDP Monitoring application that monitors data quality and safeguards pipeline

integrity.

• Developed API to manage Kafka consumers running using Spring Boot.

• Optimized SQL queries and enhanced database performance by 45%, accelerating reporting for clinical decision-making and administrative planning.

Technology Used: Python, Java, React, SQL, Spark, SpringBoot, Kafka, Kubernetes

• Developed a real-time streaming ingestion framework with Spark Structured Streaming and Kafka, delivering data to Snowflake for analytics.

• Developed Spark jobs for OfferMatch components using the Spark DataFrame API in Scala.

• Developed features for Cloud Digital Artery - enables rapid development and deployment of data

pipelines.

• Developed and productionized ML models for insurance risk prediction, including validation and ongoing monitoring, driving a 70% improvement in performance.

Technology Used: Python, Java, Apache Spark, Snowflake, Machine Learning, Tableau