Focus on stream processing infrastructure at LinkedIn.

As a main contributor, I work on the next generation real-time monitoring and log processing platform at eBay. This work involves dealing with petabytes of data being generated from machines every day, to design an efficient system with respect to cost and performance to ensure that any degradation in the health of any critical services can be detected and remediated within minutes or even seconds. This work utilizes the technologies such as Kafka, Flink, Prometheus, ElasticSearch, Kubernetes, Thanos, Ceph and etc.

Lead to design and implement the new features for Thanos, and contributing back to Thanos community.

★2019 Spot Award Winner. I helped the team to solve the performance issue on metrics ingress service. After the optimization, it can serve 5X traffic than before.

Lead to design and develop a brand new stream processing platform based on Flink, which will provide a secure, scalable and unified streaming platform to support efficient and fault-tolerant stateful stream processing in eBay.

★2017 Spot Award Winner. As a main contributor, I designed and developed a new service based on Rheos platform which empowers near real-time data movement cross different zones, specifically for EAZ to Site Lockdown. This service successfully reduced the time to move the data between the different zones with security compliance from hours to seconds, which started the new chapter of real-time data movement between zones in eBay.

★2017 Luminary Award Winner. I participated in the design and development of a new data platform (Rheos), which provides a near real-time buyer experience, seller insights, and a data-driven commercial business at eBay. Rheos provides the necessary life-cycle management, monitoring, and well-architected standards and ecosystem for the near real-time streaming data pipelines. Currently, the pipelines consist of Kafka, Storm and stream processing applications. Shared and non-shared data streams can be run on these pipelines. By the end of 2016, nearly 100billion messages flowed through the pipelines in Rheos daily. In 2017, Rheos is expected to handle 15 times of the current traffic.

As a main contributor, I worked with my team and developed the following 6 core components: Core Service; Schema Registry Service; Mirroring Service; Health Check Service; Kafka Proxy Service; Lifecycle Management Service.

Lead in designing and developing a robust backend infrastructure framework. It supports automatic creation of real-time denormalized index data for both Cassandra cluster and Elasticsearch cluster.

This framework includes five core components:

1. A core code processor engine based on Java annotation processor, Mustache and Antlr. It will generate denormalized index related Java classes automatically.

2. A real-time online job based on Spark streaming with Kafka and an offline job based on Spark. The online job will generate real-time indices for Cassandra and Elasticsearch; The offline job will run index data migration operation and index data repair operation.

3. A core message queue library based on Apache Kafka.

4. A core serialization library based on Kryo.

5. A core maven plugin which is an automatic job generator based on ASM(Java bytecode analysis library) and Maven

Lead in designing Rabbitmq cluster architecture, in order to improve system response speed, decouple the different project modules and support offline job. Make API response 2-3 times faster than before.

Lead in optimizing cloud-based email retrieval system, to support quick full text retrieval from more than 1 billion existing emails, and 0.7 million daily increasing mail indices.

Research and design new architecture for multilingual email retrieval, especially improve the accuracy and speed of mixed-language email retrieval. Research on Solr and Lucene source code in-depth, designed custom multilingual text analyzer component. This new architecture can improve query speed 2-4 times faster on average than old Solr architecture.

Optimize index build process in-depth between Cassandra and Solr cloud, design and implement a brand new architecture for this build process, using Hadoop and custom SSTable export tool to convert SSTable data to Solr documents, and update these data to Solr Cloud directly.

Design and develop an enterprise mail system, based on Cassandra cluster and Nginx, ELB load balancing technology, which can support huge business throughput of around 3000 employees' business gracefully.

Design and develop email retrieval system with Solr cloud technology; integrate Solr cloud and Cassandra, which makes full text query and index performance more powerful.

Design and develop a new file system to make this enterprise mail system more versatile, which is based on Amazon S3 and file reference counting algorithm. In addition to the basic file management operations, it supports many advanced features, such as file sharing and file private link.