Was a founding engineer of Amazon EMR Serverless - a new Amazon EMR offering clients to setup an EMR cluster 75% faster. Eliminates the need to setup, maintain, or configure EC2 hosts or EKS clusters.
•
Architected and built a multi-tenant, distributed, backend service that serves as the resource manager for clients’ distributed applications’ (i.e. Spark and Hive) workers with >99.9% availability.
•
Designed, implemented, and launched EMR Serverless’ custom images, enabling customers to install/configure packages optimized for their workloads and integrate with current CI/CD best practices. 100+ customers successfully submitted 5000+ jobs within the first two months of the launch.
•
Improved container start time by reducing Docker images pull times by 90%. Worked cross-functionally with other AWS service teams and the Amazon Linux team to upgrade a core Docker dependency to achieve this.
•
Implemented pre-initialized capacity support, a feature to reduce application start time, on EMR Serverless’ resource manager service.
•
Collaborated with downstream services to overcome capacity constraints brought on by heavy users, those who consumed at least 4,000 vCPU’s concurrently, of EMR Serverless.
•
Led the implementation of operational metrics collection and monitoring dashboards in parts of EMR Serverless to improve debugging and proactively identify issues.
Developed a system that maps over 30 billion products to an internal brands database daily to discover products and brands not present in the Amazon Catalog and to determine brands’ selling potentials.
•
Maintained a platform built on top of AWS Mechanical Turk to allow users to consolidate worker responses manually, increasing accuracy of responses when compared to the default string comparison.
•
Developed a feature to allow users to choose if a workers gets paid based on quality, reducing our users’ costs.
•
Developed a feature to retrieve responses before all questions (1000+ per batch) were answered, reducing the wait time from days to minutes for downstream consumers
•
Coordinated the handoff of an internal platform to another team and documented parity requirements for their effort to consolidate with their own platform to unify functionality across multiple orgs.