HANDS-ON EXPERIENCE: Built scalable data pipelines ground up on AWS EC2 Ubuntu instances under Amazon Virtual Private Cloud, using technology including Spark, Kafka, ZooKeeper, Airflow, Cassandra, Flask/Dash within 3 weeks.

Experience

2023 — NowBoxSoftware Engineer

2023 — Now

United States

2022 — 2023Pure StorageSoftware Engineer

2022 — 2023

Mountain View, California, United States

2020 — 2022AppleSoftware Engineer

2020 — 2022

Cupertino, California, United States

2020 — 2020Insight Data ScienceInsight Data Engineering Fellow

2020 — 2020

San Francisco Bay Area

Reddit-Trend-Bitcoin

Analyzed how users' behavior of Reddit cryptocurrency community made impacts on bitcoin price. Produced analytic reports for bitcoin investors to draft a more strategic plan.

Developed a Scala/Spark batch processing pipeline. Transformed 5TB of JSON data in AWS S3 into 1TB of Parquet data and accelerate Spark SQL query speed from 6 hours to 1.5 hours.

Ingested real-time events data streams by Kafka coded in Java, processed micro-batch data with Spark Structured Streaming, and stored in Cassandra for web access from Flask/Dash.

Unified historical and real-time data by running Airflow periodically.

Education

University of Southern California

Master's degree

National Tsing Hua University

Experience

Education

Master's degree

Bachelor's degree