• Managing 10+ PB data and 100TB memory

• Handles 34B daily events

• Operating more than 2,000 jobs

• Built highly available DataPlatform Infrastructure using AWS, Terraform. Plus, built an infrastructure RESTful API, Cost optimized/tracking service with Django

• Built CICD pipeline to build custom AMI with Packer

• Built Airflow services in multiple regions

• Manage hundreds of Elasticsearch, Cloudera Hadoop, Qubole, Kafka clusters as SRE

• Used Puppet to build on-premise host

• Used Docker/Kubernetes to build containerized services

Architect entire platform service from scratch

• Machine Learning platform as a service(PaaS)

• RESTFul API service with Django framework

• Kubernetes with Dockerized services

• Deploy with Ansible Playbook

• Datapipe with Apache Spark, Hive and Airflow

• Introduced Apache Airflow for efficient workflow scheduling and applied to the datapipe.

• Building new data pipeline of 100TB/day, 2M/sec data set, using Spark, HBase, Hive and HP Vertica

Build huge impact products to advertising industry

• Architect scalable ad data pipeline from scratch using Apache Spark(40B+/Month)

• Built scalable real time Apache Spark Streaming pipeline with Apache Kafka(100K+ records/sec)

• Maintainable data workflow management using Spotify Luigi

• Micro service architecture(Fraud detection, Delivery report, Segment audience, Machine Learning for CTR optimization and etc)

• Lambda architecture(Real time + Batch)

• Built Big Data process on top of Amazon cost effective EMR infra structure

• Integrated data processing with external companies for BI

• Data process optimization with Apache Parquet

• AWS Athena, Quicksight integration