California, United States
Project Title - SnapLogic Persistent Event Streaming
SnapLogic is an Integration Platform as a Service (iPaaS) that enables quick development of sophisticated data and application pipelines using a visual drag and drop user interface. The current SnapLogic execution engine does not provide durable and persistent storage for in-flight data, instead, most processing is done in memory. We developed a prototype of a SnapLogic Persistent Event Streaming service that extends the existing architecture by providing a native streaming service. Ultimately, this service can be integrated with the native Snaplex execution environment so that SnapLogic customers can easily deploy and leverage event streaming. In this project, we built Kafka-like components, a durable and persistent logging mechanism, and distributed consensus. Our implementation can survive node failures. We also built Streaming Service Snaps that allows a SnapLogic user to build visual pipelines that can read and write to our streaming service. We also supported many features of kafka like timely deletion of data, consistency and replication management.