Building Apollo's Data Platform from ground up for internal analytics.
● Tech lead for AI Agents platform - a Python API service on k8s/GKE. Built out observability for the service on Grafana - with Prometheus, OTEL, logs, alerts-as-code. Built out LLM quality monitoring platform, including LLM tracing, LLM regression framework, working on LLM evaluations at the moment
● Tech lead for real-time ingestion pipelines from production MongoDB into data lake, merging 30M CDC events (720GB) per hour into analytical storage, using Apache Beam, Spark, and Iceberg
● Built and maintained pipelines that ingest 100TB of production MongoDB data to Snowflake data warehouse weekly, with no impact on production, at 1/10 the cost of paying for a Saas, using Airflow and inhouse pipeline design
● Built and maintained infrastructure (Apache Airflow) to orchestrate data pipelines
● Built and maintained dev tools for the Data Platform team, such as CI/CD, monitoring, dev environments, etc.