Boston, Massachusetts, United States
• Built backend & infra for microservices to provide process and insights from 300TB data per day.
• Improved team's productivity by migrating to monorepo and local-focused development setup which includes test suites for quick validation feedback, reduced feedback loops from 20 mins to under 5 mins per iteration.
• Proposed and led end to end development of orchestrating tasks in data pipeline with AWS StepFunctions, cut the overall process by 67%
• Coordinated across teams to derive performance insight with Python (numpy, pandas, Spark).
• Scaled metric services to process more than 1 billion data points per day, reduced time taken from 10 minutes down to under 30 seconds by utilizing AWS Lambda, which cut computation servers cost by 80%.
• Standardized API style for streamlining development. Fixed multiple bugs related to query performance and concurrency across services.
• Set up CI/CD pipelines and integration tests for the entire platform.
• Stack: C#, Python, ReactJS, Serverless, Spark, AWS Kinesis, postgresql, SQS, Lambda, eks