Geospatial Data Monitoring Platform
•Worked as a core software engineer in a geospatial big data company, which constructs data pipelines to process satellite images, and leverages artificial intelligence to get some insights on a global scale.
•Designed and built a data monitoring platform, where data monitors and alerts can be visualized, configured, and set up along the geospatial data pipeline to improve data reliability and quality.
•Upgraded database schema on PostgreSQL for time-series data storage, set up alert rules based on historical data analytics using Grafana, and realized the monitoring of image count, delivery delay, and ingestion delay.
•Dockerized the data monitors, streamlined the deployment and management of Kubernetes applications using helm, and automated the whole process with continuous integration(CI/CD) on Jenkins.
•Deployed the monitoring platform into production, and reduced the time delay of anomaly discovery by 70%.