As part of the data architecture team I am very fortunate to be exposed to a great deal of technologies. My responsibilities typically include:
● Use Golang, InfluxDB, Telegraf, and Kapacitor to develop and actively maintain a custom monitoring system for our Confluent Kafka + Snowflake driven pipeline.
● Develop and maintain a self monitoring, scheduled task runner with Terraform, AWS Batch, and more to support our machine learning pipeline.
● Develop tools with bash, Python, or Go to aid in operating and managing our Snowflake data warehouse.
● Routinely add and work on existing Kafka producers and consumers using Python.
● Develop Python scripts to aid in operations for our Snowflake data warehouse and Kafka pipeline.
● Use the Python Pyramid web framework to create and maintain backend services.
● Heavily use AWS, Docker, Terraform, and Ansible to create pieces for and maintain our existing data pipeline infrastructure.