Worked on making Apache Spark performant, resilient, scalable and cloud native:

• Improved Spark cluster downscaling by building features like RDD Cache decommissioning, Shuffle offloading. Contributed the same to OpenSource.

• Improved Spark TPCDS performance by more than 2X. Researched, Designed and Implemented multiple SQL optimizations - Pre-Aggregation, CNF-DNF Predicate pushdown, Better Sort order selection, Join reordering improvements, Inner to Semi join conversion, Smart shuffle key selection, Distinct pushdown, Adaptive Hash Aggregates etc in Apache Spark.

• Worked on Sub-Query Fusion Optimization for Eliminating Redundant I/O in Spark.

• Designed and developed features such as direct writes, parquet metadata caching, executor packing, proactive shuffle data cleanup, recover partition improvements, smart executor sizing which improves spark performance and cluster utilization.

• Made SQL optimizations in Spark Catalyst optimizer such as Skew join, UDF pushdown.

• Designed and developed a new Spark Datasource connector to support HiveAcid tables in Spark.

Created a new cloud agnostic, multi engine framework Cloudman from scratch which supports multiple clouds - AWS, Azure, Oracle OPC and multiple engines - Hadoop, Hive, Presto, Spark. Gave a talk in "Strata+Hadoop" conference on the same.

Added multiple features in Cluster orchestration layer - Heterogeneous clusters, Smart AZ selection, Parallel master-slave bringup, cluster management for Public/Private subnet in VPC, Spot loss handling, Metrics collection framework.