Building the recommender system that powers the Twitter Home Timeline, the largest scale and most impactful consumer product at Twitter (> 75% of time on Twitter is spent on Home).
Experienced in building scalable and fault-tolerant distributed systems. Shipped end-to-end improvements to the entire online and offline codepath covering critical online serving infrastructure, feature engineering, performant data pipeline and running advanced A/B tests at scale.
Optimized online serving path component with SLO of 10M predictions/sec, 50M QPS. Improved service tail latencies by > 20% and achieved massive reduction in ranking failures. Improved service scalability, reliability, and maintainability.
Redesigned streaming and offline data pipelines to increase production training data collection by 30, O(100TB), while removing training/inference discrepancy. This resulted in significant model and ranking quality gains on Twitter’s largest consumer facing product surface: +16M UAM, +500k unique engagers.
Migrated core service components out of a legacy codebase into a more performant framework.
Keywords: distributed systems, backend, ML, Python, Java, Scala, AWS, GCP, BigQuery, SQL, ETL, Kafka, Hadoop