Search at Amazon

Product Search at Amazon.

Working on resiliency, fault tolerance, cluster management, and distributed systems for Amazon Elasticsearch Service.

Contributions (as technical lead) -

* "Autotune for Amazon ES" - a self adapting feedback loop mechanism for intelligently optimizing Elasticsearch clusters.

* Hyper-scale shard allocation and fault tolerance in Ultrawarm enabled Amazon ES clusters.

* A strongly consistent framework for in place configuration updates in a distributed system.

* Self Healing framework to auto-heal clusters.

Other projects I've helped build:

* Split brain avoidance mechanisms

* Internal monitoring systems

* Internal architecture of the service across control and data plane.

* Different parts of a domain's lifecycle supporting scaling updates and configurational changes.

I'm routinely involved in operational deep dives and mentoring other engineers.

Part of the core team of engineers that created and launched Amazon Elasticsearch Service.

Technical Lead for:

• Cluster management and shard allocation in Amazon Elasticsearch

• Hyperscale shard allocation in Ultrawarm

• AutoTune for Amazon Elasticsearch

Committer on the Apache Lucene open source search engine. My current focus has been indexing and merging in Lucene, and I'm actively exploring other areas. I help with reviewing and committing changes, making Lucene enhancements, and maintaining the code base in general.

I implemented a concurrent, non-blocking, transactional version of the addIndexes() API, that leverages concurrent background merges to enable users to combine indexes with a low add-to-search latency. The change is foundational in unlocking the ability to decouple indexing and search shards in applications that use segment replication.

I'll speak more about this change and decoupling indexing from search at my talk in the upcoming ApacheCon NA, '22, on October 6th at New Orleans, Louisiana - https://www.apachecon.com/acna2022/schedule.html