Project: Financial Data Platform Modernization

• Led migration of legacy Airflow/EMR pipelines to Databricks, improving scalability and auditability,

resulting in a 15% cost reduction.

• Designed and implemented PySpark ETL pipelines across raw, cleansed, and curated layers with

full lineage and version control, cutting future development effort to half.

• Orchestrated data optimization by applying Liquid Clustering on financial datasets for regulatory

reporting, shrinking storage costs by 30% and improving query performance by 4x.

• Established a robust data governance framework around Snowflake, ensuring 99.9% data accuracy

for regulatory compliance and enabling 75+ users to access insights via Cognos under IAM controls.

• Led an engineering squad of 35 engineers, overseeing sprint planning, risk management,

stakeholder alignment, and production support, which reduced defects by 30%.

• Managed the migration of 2+ years of legacy financial data from S3 to Delta Lake with full

compliance and decommissioned outdated EMR workflows with zero business disruption.

• Python developer for a SaaS Auto-Scaling tool using threshold-based algorithm design.

• Integrated custom scaling algorithms into OpenStack to optimize compute usage.

• Pioneered Golang-based components for a multi-language PaaS deployment platform that helped

reduce latency by 15% and cut down build times by 12%.

• Managed backend deployment via Kubernetes & Docker ensuring high availability.

Project: Pharma Data Lake on AWS Databricks

• Spearheaded data lake implementation, integrating pharma data from diverse sources into AWS

Databricks, improving data processing speeds by 35%

• Consolidated customer data into Redshift using optimized stored procedures, freeing up 15+ hours

weekly for strategic initiatives.

• Created a migration framework using S3 and RDS, automating data transfer and validation, ensuring

99.9% data accuracy.

• Optimized ETL pipelines within Databricks, reducing processing time by 30% and improving data

quality for business insights.

• Implemented 7+ data governance policies within the data lake, ensuring regulatory compliance and

enhancing data security.

• Mentored a data engineering team, resulting in a 4x increase in team productivity in data lake

development and ETL processes.

• Automated data validation processes, reducing manual validation effort by 80% and ensuring data

accuracy across the data lake.

• Built a robust, scalable AWS-based data lake using Python, PySpark, and serverless technologies to

centralize data from 20+ disparate sources, enabling advanced analytics capabilities.

• Developed Lambda jobs to trigger EMR workloads and manage Parquet/Glue metadata.

• Automated serverless functions with AWS Lambda to distribute real-time notifications via SNS,

diminishing incident response times by 30% and enhancing system uptime.