Redesign and migration of GTCS Sweeper Microservices

• Authored system design documents and collaborated with senior team members, discussing application requirements and design trade-offs.

• Migrated 6+ GTCS sweeper microservices to AWS, enhancing efficiency, scalability, and resilience of the applications.

• Utilized TypeScript within AWS CDK for cloud resources, establishing an efficient data pipeline that improved sweeper services by 30%.

• Developed an event-driven data pipeline, transitioning from a continuous check to event-triggered, reducing latency by 50%-70%.

• Integrated Amazon SQS for message decoupling and initiated SNS notifications upon message reception.

• Programmed Lambda applications in Java, optimizing design based on execution time, complexity, and costs.

• Adopted Git for versioning and collaborated with multiple team members.

• Designed CI/CD pipelines for application releases, using Agile methodologies, from personal tests to production deployments.

• Managed on-call duties, addressing 15-20 urgent tickets weekly, ensuring optimal user experience and ticket prioritization.

• Participated in weekly DevOps and Bug Bash meetings, identifying system issues and formulating stability plans.

Google Map's Transit Data Pipeline

• Spearheaded the integration of public transportation system data from two major U.S. cities, San Francisco and Boston, and two mid-sized Canadian cities within Google Maps, encompassing timetables, route maps, and other geographical data.

• Maintained the accuracy of GTFS static data and real-time updates for specific regions within Google Maps, leveraging in-house data processing tools for pipeline data validation and modifications.

• Collaborated in cross-departmental meetings with over 25 different transportation agencies and data providers to obtain updated transit data, addressing data discrepancies by making model alterations and adjustments.

• Utilized Java and internal automation for the data ingest process, facilitating data visualization, validation, and adjustments, leading to a 30% efficiency boost in team data processing.

• Defined requirements for RESTful APIs endpoints for the backend team, allowing for data filtration based on diverse criteria and display on the application's dashboard.

• Conducted spatial and transit data comparative analysis before and after updates, ensuring data integration and validation outcomes met anticipated standards.

Apple Map's Territory/Transit/Water Data Pipeline System:

• Integrated geospatial data, including administrative regions, postal codes, hydrology, and road systems for over 20 countries.

• Collaborated in maintaining Apple's Neutron Data Model, adapting product definitions and data outputs for various countries.

• Developed data model designs and integration workflows using Scala/Python, enhancing data pre-processing efficiency by 20%.

• Utilized PostgreSQL to pre-process data inconsistencies and discrepancies, optimizing the computational logic of Scala programs and improving SQL query performance by 50%.

• Employed Hadoop for big data processing, increasing team productivity by 25%.

• Established new repository IDs with Quark and imported integrated data for further validation.

• Executed over 100 validation tasks using Spectrum to monitor data quality.

• Managed and developed over 150 batch data modification codes, adjusting logic based on validation results.

• Visualized and edited map data using tools like Fusion X and QGIS.

• Collaborated with teams using Git for version control.

• Conducted spatial and data quality analysis, ensuring the accuracy of data integration processes.