Implemented realtime system usage event capture in the production stack across multiple microservices
Singlehandedly designed and implemented the company’s first iteration of an analytics data processing pipeline in 2019
•Batch ingestion, batch transformation, storage, presentation via BI tools (AWS Quicksights)
•Built with AWS s3, AWS lambda functions, AWS SQS, AWS kinesis firehose, redshift, dbt.
Designed and implemented second iteration of data processing pipeline in 2021 with primary goal of scalability.
•Implemented generalized producer consumer framework that eliminated bottlenecks for scaling the previous pipeline design by streaming data instead of batch processing.
•Reduced latency from event capture to end reporting from 1 hour to < 1 minute.
•Ingested ~1MB/min during peak loads.
•Built with the custom python pipeline framework, AWS SQS, kibana, redshift, dbt, and elasticsearch.