Architect of Search Indexing Pipeline

• Led a team of engineers to create the next generation indexer for eBay’s search engine. Designed the indexer from ground up and authored about 85% of the framework code in Scala. The indexer which run on spark and standalone is over 3x faster than the current generation indexer.

• Decreased the search query latency by over 30% by designing and implementing a mechanism to reorder the documents in the search index.

• Reduced the size of the search index by 30% by developing a compression mechanism to compress similar documents (MSKU) in the index.

• Designed a system to identify and rewrite a full update on a document into a partial in-place update which reduced fragmentation of the index over time by over 60%.

• Designed a system which can squash multiple incremental updates to the query nodes into 1 incremental update. This decrease the cold start time of the query node by 50%

• Designed and implemented a system to collect and aggregate the click through information for the answer modules on all of eBay’s search and browse pages. The aggregated information is then used to rank these modules using a Bandit Algorithm.

• Designed and implemented a system to collect and aggregated and query logs generated by over 10,000 machines which are part of eBay’s search engine. The system does streaming joins on over 300TB/day of logs and able to respond to queries about the search engine’s performance in under 1 second.