part of ML/AI group

Tech Stack : Apache Kafka, Apache Spark, GCP, Kubernetes, Helm charts, Python, Java, Airflow, MongoDB, Prometheus, Grafana/Loki, Redis, Terraform

ML/AI - Anamoly detection(Isolation Forest), Prediction using FB Prophet, LLM

• Project involves creating an Automated system which does the reconciliation of Apple Employee Compensations in Apple Payroll and HR Systems.Automating the payroll checks and Increasing their frequency will increase the accuracy of payroll and decrease the workload for both Apple and ADP with regards to last minute escalations and fixes.

• Technology Stack : Apache Spark, Python, Apache Kafka, Apache Parquet, JSON, Incorta (In-Memory Data Warehousing & Visualization tool), JSON, Unix, Github

Accomplishments/Responsibilities:

• Led the Design, Architecture and Development activities for the Incorta Audit and Data Warehousing development project.

• Key participant in Design and Architecture decision making, closely working with upstream and downstream teams in Apple to finalize Non-Functional requirements, and translate them to Technical solutions

• Designed/developed Data model and data ingesting pipelines.

• Used Pyspark (primarily Analytical functions) to read data from parquet files, and create Semantic layer.

• Anchored Proof of Concept (PoC) development to validate proposed solutions and reduce Technical risk.

• Involved in Providing critical Technical/Architecture related inputs to help finalize the Plan/Schedule and resource requirements.

• Involved in interviewing/hiring & mentoring team members, assigning/monitoring tasks and deliverables, providing Technical guidance to developers in the project.

• Supporting the application during Warranty period, and helping resolve/troubleshoot complex technical problems

The project involves using iTunes data for Analytics/Reporting. It involves interfacing with Business & Core Induction teams to understand Analytics requirements, translating to Technical architecture/BI

Solutions

Responsibilities include - Design, Development of aggregates, Identifying & implementing Architecture related improvements/automation, mentoring/helping Junior developers(w.r.t. Technical solutions, Scala/Spark coding standards etc.)

Technology Stack includes – Apache Spark, Scala, Cloudera 5.11.1 (Hadoop, Hive), Jupyter notebooks,

Apple Private Cloud (S3, EC2, Jenkins, Splunk)

Accomplishments :

Designed/developed Analytics solutions - reading data in parquet format, using Scala + Apache Spark to implement complex Business Logic

worked on POC to evaluate/configure using Jupiter notebooks with Scala - to enable complex logic & visualization charts. (By default, advanced visualization is available only if we use Python on Jupiter notebook)

Description: The project involves setting up highly available, scalable, centralized Real-time Data pipeline for DataCenter Data. The data is used for Analytics/Reporting.

Technology Stack – Hortonworks 2.x platform, Apache/Confluent Kafka, Apache Spark, AWS(S3, EC2), Hadoop, Prometheus, Grafana, Hive, Scala, Jenkins

Accomplishments/Responsibilities:

• As part of Phase 1 : Developed data pipelines using Apache Spark + Scala moving data from Kafka to HDFS/Hive (on Hortonworks 2.x platform).

• As part of Phase 2 : Developed data pipelines using Apache Spark + Scala moving data from Kafka to AWS/S3.

• Confluent Kafka - Evaluated platform capabilities, published best practices, implemented/optimized key Confluent modules - Control Center, Auto DataBalancer, Schema Registry, Replicator, Kafka Connect, Data Stream monitoring etc

• Evaluated & Defined Hadoop security capabilities for HDP 2.x components - HDFS, Kafka, Hive, Hbase, Spark, OpenTSDB, Grafana

• Implemented Security for HDP 2.x, some of the components implemented include

• Kerberos for Authentication

• SSL for Confluent Kafka

• Apache Ranger (role based access)

• Apache Knox (perimeter security)

• Encryption using Hadoop KMS

• Defined best practices for Hadoop governance - encompasses Security, Lifecycle Management, Data Quality, Metadata management, Operations & reporting