Conducted tests to determine optimal Flume Agent configurations in cloud environments.

Designed, codified, and deployed infrastructure using Terraform, SaltStack and Github Actions.

Updated Dstillery-tech-stack specific Apache Hive dependencies to work on Google Dataproc Clusters.

Implemented metrics to monitor throughput, filtration, and failure rates of over 100 billion daily events through Apache Kafka topics and Apache Flume Agents (macro view of pipeline health).

Designed metrics to monitor success rates & failure reasons in Flume Sinks and Interceptors (micro view of pipeline health).

Built Grafana dashboards to visualize such metrics for both business and engineering stakeholders.

Wrote system to unify GCP security credential Secret access across all Dstillery applications via SaltStack.

These contributions aided in a company-mandated cloud migration which saves Dstillery millions of dollars per year in data center costs and minimizes spending on cloud computation and storage costs.

Managed migration of 7 core Hive databases from on-prem Hadoop environment to GCP Dataproc clusters.

Reconfigured Flume sinks (streaming Protobuf messages w/ Hive table data) to GCS buckets instead of HDFS directories.

Wrote a program for Hive table partition management, deployed via GCP Cloud Functions.

Estimated, planned, and tested GCP resource usage to optimize configurations to minimize storage/computing costs.

Developed core service to enable all 17 Groovy apps to interface with Hive in GCP environments.

Redesigned Spring Boot, Groovy, and Python pipeline components to run in GCP compute instances.

Utilized agile development to work on a variety of services that make up the platform, mostly based in Java Spring Boot and Python Flask.

Worked to integrate Managed ML Flow into the ML model deployment platform (in addition to the current AWS based offerings).

Implemented backoff functionality (http call retry - including unit tests), and resource auto-scaling across the entire platform.

Co-developed a Slackbot using Elixir for general error reporting with another software engineer.

Learned good software engineering practices, especially test-driven development, code organization (namely separation-of-responsibilities), and story-point estimation.

Focused on efficiency for cloud pipeline deployment; helped to increase existing performance by 300%.

Frequently communicated with data scientists (customers).