Cosco is a distributed data service which implements Spark shuffle data repartitioning. Up to 1000PB of data are ingested per day across a distributed infrastructure of 10k machines across 8 data center regions Shared Data Writer

Led development of Share Data Writer (SDW) project which optimizes memory usage during Spark shuffling jobs. Optimization done by implementing multi-threading with shared data buffers. Memory usage reduced by 60% from 4GB per executor to 1.5 GB per executor.

Implemented a new data package format for Spark shuffles to support correct encoding and decoding in high-concurrency use cases.

Translation Routing Service

●Led the development on ‘Human Translation Worker’ which ingests translation tasks from upper stream service. This service helped over 50+ organizations to globalize their business. It processes 60k files per day with peaks 220k files per day.

●Designed and implemented the Machine Translation Worker component which pulls and pre-processes jobs before sending them to a back-end translation engine. It services 15 requests per second with peaks 25 requests per second.

●Designed a queuing and task distribution service for translation workflow and connect with many worker services to process jobs. It handles 10k jobs per day and with peaks 100k job per day.

●Implemented using AWS services:EC2,Elastic Container Service(ECS), DynamoDB,S3 ,Simple Queue Service(SQS), Lambda + Step Function.

Translation Automation API

●Designed and implemented an API that allow user to fetch project information and get notified when project status changed.

●Led design of an authentication solution to allow Amazon-external users to access Translation Automation API using JSON web tokens.

●Implemented as a Restful API and used SQS + SNS for project status notification.

Translation Project Management UI

●Led the project management UI development for Amazon Translation System.

●Implemented with Typescript and Polaris for ReactJS(AWS)

●Developed a workflow translation framework, which can translate json based workflow file into Apache Flink program. Increased the SnapLogic, Inc data processing ability 10x larger.

●Implemented using Apache Flink and AWS services: Elastic Map Reduce(EMR) and S3 for data storing.

●Developed a data processing system to streamline analysis workflows for data scientists monitoring AWS-impacting, but Amazon-external provider networks.

●Implemented using Apache Spark and AWS services: Elastic Map Reduce (EMR), Lambda for execution scheduling and S3 for data persistency.

●Ingests datasets of 10GB daily, architected to be scalable to 1TB+ data streams.