Designing, Developing & Deploying Scalable Machine Learning Models in Production (Airflow, Spark, Python, Kubernetes, MLFlow)

Spark, Kafka (Kafka Streams), Airflow, Kubernetes, Cloud, Java, Scala, Python, Go, GrapphQL, Protobufs, Cassandra, Elastic Search, CI-CD.

Building IoT Products on Cloud with Cloud Native Support (K8s, Docker, Jenkins, Spinnaker)

Building Data Pipelines using Airflow, Spark, Kafka

Designed and Developed spark-protobuf. EPIC - https://issues.apache.org/jira/browse/SPARK-40653

• Designed and Implemented spark-protobuf support from the ground up.

• Added from_protobuf and to_protobuf functions in Scala, Java, and PySpark

• Made changes to new use error classes for Spark-Protobuf exceptions

• Worked on fixing bugs, unit tests, and adding documentation

https://github.com/apache/spark/pull/37972

https://github.com/apache/spark/pull/38212

https://github.com/apache/spark/pull/38344

https://github.com/apache/spark/pull/38515

https://github.com/apache/spark/pull/38603

• Designed and Implemented Scaler functions for array types, and string types. array_reverse_int, array_reverse_string, array_sort_int, array_sort_string, array_index_of_int, array_index_of_string, array_contains_int, array_contains_string, strpos, strrpos, strrpos, to_utf8, normalize, normalize, split, replace, hammingDistance, array_concat_int, array_union_string, array_union_int, array_remove_int, array_remove_string, array_distinct_int, array_distinct_string, array_slice_int, array_slice_string

https://github.com/apache/pinot/pull/6446

• I worked on bug fixes such as style checks and nit checks.

• Made changes to use only google annotations GuardedBy instead JavaX

• Made changes to set Test timeout higher for SegmentManagerThreadSafetyTest

• Added StructuralSearchInspection, Prohibit check on Thread.getState()

https://github.com/apache/druid/pull/8394

https://github.com/apache/druid/pull/8386

https://github.com/apache/druid/pull/8060

https://github.com/apache/druid/pull/7889

https://github.com/apache/druid/pull/7890

• Designed and Implemented Helm chart to build Apache Kudu Kubernetes cluster. A cloud-native Support.

• Added Kudu kubernetes statefulset manifest and worked on docker image changes.

• Implemented kudu client tools for Hadoop and spark import/export(csv,parquet,avro)