Willow is an open-source framework for building Change Data Capture pipelines. It reflects database changes to a cache in near real-time. Read more at https://willow-cdc.github.io/
•Developed framework to simplify Change Data Capture pipeline creation from 9 steps to 4 steps, leveraging open-source tools such as Debezium and Apache Kafka.
•Resolved Debezium configuration issues which prevented multiple pipelines from connecting to the same database (PostgreSQL, Debezium).
•Ensured Willow works properly when provided a minimum privileged database user, avoiding the need to provide Willow with superuser privileges.
•Architected and implemented a RESTful API, facilitating pipeline creation and teardown (Node.js, Express).
•Designed and constructed a custom Kafka connector to transform Kafka events and send to a cache (Node.js, KafkaJS, Redis).
•Built user-friendly UI that streamlines pipeline management (React, Material UI).
•Authored technical case study, providing an in-depth analysis of Willow's problem domain, system design, and engineering decisions (willow-cdc.github.io/case-study ).
