Automating the data load process by using Airflow triggers to spin up Docker containers into a Kubernetes cluster

Worked on streaming source data using Kafka producers and consumers to always have the latest data

Using SQL and Jupyter Notebook to test data transformations and validate data pulled from DB2, TM1 and Java databases