Maintained the core data pipeline, processing terabytes of data daily using Hadoop MapReduce, Airflow and AWS EMR, and updating security ratings for over 1 million entities in AWS RDS and Redshift databases
•
Designed the MySQL backend and Django model of a new API in the BitSight portal that could show customers their upcoming security rating change estimation
•
Optimized the Rapid Assessment Service that took one domain as input, searched associate IPs and secondary domains, and generated security ratings. Reduced search time from over 5 min to 90% less than 1 min. Generated security ratings of 250,000 entities for a customer in two weeks.
•
Integrated the revenue data of entities from external APIs into BitSight rapid assessment API. Built caching mechanism to reduce response time and also avoid exceeding rate limits of external APIs.