San Francisco, California, United States
• Member of Apache Spark Platform Team,
• Experience on Spark Query Compilation and Execution,
• Extending Spark Adaptive Query Execution(AQE),
• Improving Spark AQE Skew Mitigation Support for SortMergeJoin and ShuffleHashJoin,
• Extending Spark Memory Management/Consumers and Physical Operators Disk Spilling Support,
• Fixing Spark Driver and Executor Side Memory Problems,
• Tuning Spark Shuffle Partitioning,
• Extending ShuffleHashJoin & BloomFilterJoin features and get production ready for critical and complex queries,
• Working on Spark Scalability, Resilience and Performance,
• Lead Engineer of Progressive Spark Upgrades (Spark is backbone of Prism Data Analytics Platform),
• Lead Engineer of Architectural Improvements of Spark Observability Platform
• Experience on Rust, Scala, Java, Akka, Functional Programming, Distributed Query Execution, Big Data Technologies(e.g: Apache Spark, Hadoop YARN, HDFS, Kafka, S3, AWS, Kubernetes),
• Working on Apache Spark In-house and OSS Patches, ETL Batch Data Processing Pipelines, Distributed Computing Challenges, Performance Improvements.