I worked on getting events quickly and reliably into the database for a while for a variety of products as part of the Ingestion/Pipeline team. Then I worked on getting those same events back out of the database quickly and reliably for even more products as part of the ClickHouse team.

• Led the technical design and implementation of the Metadata Change Detection feature, ensuring that Parse.ly’s stored copy of publisher on-page metadata is kept up-to-date with the most recently published version, addressing a long-standing customer pain point with Parse.ly’s data quality.

• Implemented several internal and research & development projects, such as: added snapshot data diff testing to Parse.ly’s batch processing system to make it more resilient against the introduction of accidental data quality issues; implemented a proof-of-concept to demonstrate the viability of unifying the streaming and batch processing paths of Parse.ly’s lambda architecture into a single logical code path that could be run in both environments, decreasing the overall conceptual complexity of the existing system and increasing streaming-side performance through the extensive use of Apache Arrow to decrease transfer serialization overhead.

• Other responsibilities as an experienced member on a small team, such as extensive stabilization and hardening of legacy systems, incident response and resolution, mentorship of junior engineers through reading groups and pairing sessions, etc.

Early employee/engineer (#6), with a wide variety of responsibilities but primarily focused on backend storage and streaming data systems, including:

• feature development, ranging from large efforts (such as Similar Issues, which uses locality-sensitive hashing to identify related collections of events based on various attributes/dimensions, and the backend component of the Metric Alerts feature, a subscription engine for triggering scheduled queries on streaming time series data) to smaller improvements (such as email alert digests, billing quota notifications, and weekly report scheduling and delivery)

• member of the small Search and Storage team that migrated the core event data from materialized aggregations in PostgreSQL to an unmaterialized representation in ClickHouse, and was individually responsible for the integration with existing systems as well as validating the data integrity and performance of the new system

• various internal performance and reliability projects, including a high performance change data capture library using the PostgreSQL logical replication interface to emit upwards of 20,000 replication messages/sec to Kafka from a single Python process; a high performance multiprocessing stream consumer toolkit in Python; distributed query deduplication to shed load and shield ClickHouse from the compounding effects of long-running, resource intensive queries

• developer productivity improvements such as adopting gradual typing (with mypy) and Python 3

• research and development work, such as prototyping a time series anomaly detection strategy

• critical bug fixes and incident resolution, code reviews, interviews and interview process improvement, technical blog posts, internal technical documentation, customer and community support, conference staffing, and the other responsibilities that come with being an early employee/engineer at a successful startup during various stages of the company lifespan

Worked on a variety of different backend projects (many of which were eventually open-sourced), including

• leading and/or contributing to larger initiatives such as a pure-Python Kafka client (samsa, later adopted by Parse.ly and rebranded as pykafka); the design and implementation of a fanout activity stream backed by Cassandra to power Disqus Home; and pgshovel, a trigger-based change capture suite for PostgreSQL using PgQ, ZooKeeper, and Kafka

• smaller efforts such as implementing a Redis-based email notification digest storage backend, an optimized CSS inliner (toronado) as well as various tech debt reclamation projects