Scaling the real-time meeting and AI chat infrastructure to support large-scale, rapidly growing traffic.

Full-stack development including: Developing gRPC-based services to remotely operate motion controllers for laser, robots, and various other services on premises, in Docker or on AWS; Modeling study of embedded systems using finite state machine and behavior trees; Researching and developing a Bazel-based mono-repo build system that supports C/C++, Go, Python, Protobuf, gRPC and cross-platform toolchains for embedded systems and cross- language bindings/dependencies (open sourced at https://github.com/thx123/bazel-examples).

Owned the self-driving vehicle sensor calibration service that persists, updates and pushes latest calibration of tens of thousands of sensors to the entire self-driving car & truck fleet operating in multiple cities, incl. that for cameras, lidars, radars, GPS, microphones, etc;

Developed an off-board batch data processing pipeline and related APIs to automatically detect, analyze, and re-calibrate sensors from newly imported self-driving vehicles’ sensor logs, filing work orders on faulty sensors, applying structured labels to affected systems downstream, and aggregating results across the self-driving vehicle fleet for visualization and trending analysis;

Google-wide Logging Infrastructure (Sawmill) Team (Jan 2016 – Sep 2020):

o Developed a near-realtime and globally-distributed logs tailing system adopted by major streaming platforms including YouTube and Ads. It grew from my 20% project into a mission-critical and team-managed system, onboarded by SRE with 24×7 support in multiple time zones. It improved streaming data discovery latency by 8x @50p, 3x @99p, and 6x @99.9p (“p” for percentile), in average, across all collected log types, and flattened latency spikes from 10 minutes to below 30 seconds. Other side benefits include load and data center outage reductions, better data abstraction, simplified client APIs, and better monitoring and alerting of streaming data latencies;

o Owned multiple large streaming & batch data processing pipelines for Google Instant Web Search, Near-realtime security monitoring, CloudSQL, and App Hosting (Scale: 40K+ CPUs, 30+TB RAM, 30+GB/sec data ingestion rate in Google private cloud). Works included streaming latency root-cause analysis, major refactorings, upgrades, migrations, deprecations, and stability/resource tuning;

o Member of the Logs Analysis team managing the Logs Proxy Service, one of the top 10 compute-intensive production services at Google, consuming 1.6M CPUs globally, running 24×7 with full SRE support; Made numerous improvements, incl. L4 cache on SSD in Google data centers to reduce hard drive spindle cost and improve data read/write latencies (earned Tech Infra 2019 Feats of Engineering Team Award);

o Member of the Logs Metadata team managing Google-wide metadata service for all log types.

Enterprise Search (Topaz) Team (Aug 2014 – Dec 2015):

o Full stack engineering work including performance/latency measurements, automatic regression tests, visualization and optimization on both front & back ends; owner of multiple logs processing pipelines, worked closely with Dev, SRE, Test and Privacy/Legal teams on various backend features and product launches;

Worked in the search infrastructure team. Participated in the project to build a complete Tweet indexing and searching system for every public tweet since 2006.

Blog post: https://blog.twitter.com/2014/building-a-complete-tweet-index