San Jose, California, United States
An easy-to-deploy, reliable, & highly scalable log observability platform for distributed applications. After a one-step deployment to AWS, it autoscales using Kubernetes & maintains high reliability with a Kafka cluster.
• Developed a streamlined deployment process using AWS CDK that provisions cloud infrastructure, reducing a complex 15+ step workflow to a single command.
• Engineered a high-throughput log streaming pipeline using Kafka with 3-broker replication, designed to handle several terabytes of logs daily while ensuring reliable data delivery.
• Containerized and deployed the Unilogs platform components into a Kubernetes cluster, enabling automatic horizontal & vertical scaling to handle 4+ TBs of logs per day.
• Built a CLI tool for deploying Dockerized Vector log shipper, automating parsing for common & custom log formats.
• Implemented end-to-end security for log shipping infrastructure by leveraging AWS SDK to dynamically retrieve Kafka endpoints & TLS certificates, while integrating SASL authentication to strengthen data protection protocols.
• Resolved Kubernetes service permissions according to AWS IAM specifications, enabling log aggregators to write to S3 buckets for long-term storage.
• Authored & edited a comprehensive case study with technical documentation and infographics for unilogs.github.io.