Pinterest is a visual discovery tool to help people find what they love and try an idea offline. I was in Data Eng team, which handles a massive amount of offline data processing.
•worked on a DAG (Directed Acyclic Graph) engine https://github.com/pinterest/pinball and DAG composer for large scale offline data processing, which is similar to https://airflow.apache.org.
•worked on a microdata framework to break Pinterest's core pin/board/user data pipelines into smaller and more manageable ones, which achieves better isolation and faster data refresh.
