Continued to scale our datastore infrastructure as our user base and traffic grew substantially in 2020; focused more effort to improve security of our data and support our nascent services platform:

• Implemented oss Redis clusters for new services in all environments

• Implemented Hashicorp Vault clusters in all environments

• Revamped MySQL slow query guardrail

• Comprehensive site traffic replay/load-testing (and contributed to goreplay project to speed up its traffic replay function)

• Reviewed new technology PoCs and architecture proposals as member of Technology Council

• Improved MySQL CIS compliance

Manage and expand our datastore infrastructure; work closely with our SRE team to improve the performance, reliability, and scalability of our site.

• Managed production MySQL clusters and an enterprise Redis clusters in three data centers, using terraform, ansible, and custom automation in bash and python

• Implemented and deployed ProxySQL clusters to scale and protect our production MySQL clusters

• Redesigned a workflow/process used by our Data team to remove dangerous load on our production clusters and improve data aggregation workflow

• Reviewed MySQL schema alter and related Django/Python pull requests

• Created an annual road map for our team

• Created and reviewed new technology proposals

• Created automation scripts and runbooks to failover our production data stores to the disaster recovery data center

• Optimized MySQL configurations and ran game days and load tests on a number of production clusters

• Improved the reliability of our development database refresh process

• On-call rotation and incident reviews

• Onboarded and mentored junior engineers

Technologies used: ansible, terraform, python, mysql, proxysql, haproxy, twemproxy/nutcracker, rabbitmq, celery, memcached, redis enterprise cluster, docker

I ended up working more on production site-related projects, mainly centered around Udemy's databases and message queues.

• Provisioned and configured production MySQL instances

• Automated MySQL replication repair

• "Ansibilized" a number of production clusters, including rabbitmq, backup, log aggregator, kafka/zookeeper

• Tuned and tested production MySQL cluster configuration

• Wrote tests for many ansible roles

• Implemented public API endpoints to allow 3PL partners to query customer orders and automated the fulfillment process

• Implemented features in batch workers that synchronize and process customer data from a number of partners: Shopify, eBay, Amazon, Squareup, etc.

• Updated legacy codebase to migration our system to a new sharded database (Vitess): swapped legacy mysql driver; reimplemented low-level database methods; updated queries that could not pass the new database guardrails

• Collaborated with our Ops team: updated all MySQL nodes to use faster disks in order to add replication to our database cluster; provisioned many classes of servers (GCE) and updated our Ansible scripts when necessary; added monitoring to many services using in-house grafana system as well as New Relic; maintained our deployment and continuous-integration service and added feature to block deployment of untested branches

• PHP; Python; MySQL; Vitess; Laravel; Ansible; Django; API