Streaming data processing at Pinterest

Presto

Build up Presto@Uber to power multiple user cases including dynamic pricing, driver promotion, ML feature engineering, A/B testing and so on.

• Parquet Reader optimization. Support nested column in QueryPlan and Parquet Reader, speedup query performance by 2-10X.

https://eng.uber.com/presto/

• Presto Geospatial Query Optimizer. Build and query geospatial index during query execution, without user query rewriting. dropping down vcore seconds by 3X, execution time by 1.7X.

https://www.oreilly.com/ideas/query-the-planet-geospatial-big-data-analytics-at-uber

https://dataworkssummit.com/san-jose-2018/session/geospatial-data-platform-at-uber/

• Cache File Metadata in Presto, reducing HDFS Namenode RPC by 50%

• Support Security in Presto as IPO readiness efforts

• Developing common Proxy Layer to intelligently redirect queries to multiple query engines.

Query Translator

Translate Vertica SQL to Presto SQL

Ads&Data team in Yahoo

Designed and Implemented the Next Generation Yahoo’s User Profile Platform that powers Yahoo user modeling, profile serving and audience insights. This data platform supports fast data ingestion, random access and interactive ad-hoc query using state-of-art technologies including Spark, HBase, Kafka and Hive. Presented on Hadoop Summit 2016 San Jose.

https://www.slideshare.net/HadoopSummit/yahoos-next-generation-user-profile-platform-63920409

Working with Science Team to develop CPC Advertising Pipeline, in which machine learning models are trained and used in Real-time Bidding.

Worked in Yahoo Finance.

Built replay module across quotefeed servers, which can be used for raw data analysis, functional testing and performance testing.

Implemented library that converts Y!Finance data type to protocol buffer message using reflection.

Designed and implemented in-memory storage system using mdbm and boost asio. Improved QPS

by 120%

Implemented a series of scripts for performance investigation by leveraging tools like netstat, strace and powerful profilers like perf, systemtap and google perftools. Helped developers easily target high CPU usage functions, memory leak and root cause of latency.