Databricks SQL

SQL Gateway Team

Synthetic Simulation Team

• Developed data lineage platform and SQL parser with Python to index ETL, table, and cross-system level lineage for all data processes across DoorDash as part of the Data Catalog project

• Built interactive lineage graph view, allowing engineers to perform impact analysis, data governance, and root cause analysis

• Created metadata extractor to catalog all registered ETL jobs and make them searchable through a web interface using Neo4j and Elasticsearch

• Designed and developed a parallel resource fetching component using Spring, enabling Alexa backend services to gather resource data in parallel while handling requests

• Improved user latency on average by 180 ms, up to a 60% reduction in latency after integrating the component

• Implemented load testing to provide thorough insight on end-to-end latency metrics and improvements