I'm with Integration-At-Scale team on Informatica Cloud. We develop software for Informatica Mapping jobs. Clients create Mappings to read, transform, and store data. We optimize, compile, and generate Java/Scala code to create a data processing pipeline. We set up a Kubernetes cluster with Apache Spark, start and execute spark job on the cluster. (800+ customers use daily)

Designed, gathered, documented requirements, and implemented/developed The Serverless Runtime environment. A serverless runtime environment is an advanced serverless deployment solution that uses an isolated, single-tenant model. ETL as a service. (500+ customers use daily)

Developed a procurement service(AWS and Azure) that creates resources for the Serverless Runmine environment. Worked on a warm-pool service that contains pre-baked Kubernetes cluster nodes.

Developed a Cluster Creation Service for creating Ephemeral Kubernetes clusters for The Serverless Runmine environment.

Worked on a customized Kubernetes autoscaler project.

Developed a forecasting project, and developed a time series machine learning model on Apache Spark to predict the future state of the warm-pool nodes. This was the first Machine Learning model implemented with Spark in the team and saves cloud provider costs for customers on average %17.

Developed an always-running Apache Spark cluster for predicate pushdown queries to run in parallel with ETL processes. A single node local cluster in the microservice node, which accepts Spark SQL queries to run against databases. Improved overall performance of running ETL tasks by 23%.

Improved the code coverage(Java, SonarCube) for entire microservices from 62% to 90%.

Created integration test suits(Docker, postman/Newman) for entire AtScale microservices. (11 total)

Developed Restful microservices to run ETL jobs on Kubernetes Local/AWS/Azure clusters.

Created an internal document that details the workings of Apache Spark Internals.

Asurion, LLC, San Mateo, CA

Asurion, LLC is a privately held company based in Nashville, Tennessee that provides

mobile insurance, device protection, and support services for smartphones, tablets,

consumer electronics, and appliances.

● Data Ingestion;

o Batch, using Apache Spark (EMR, Scala) populating hourly data sets (parquet) on

AWS S3.

o Streaming, AWS Lambda, Apache Spark Steaming. Customers’ data collected

through AWS Kinesis. Spark Streaming runs 7/24 to populate the data on S3

(Parquet), DynamoDB, and Graph(neo4j). AWS Lambda is also used for real-time

processing.

● Data Insights. Apache Spark (EMR), AWS Lambda, DynamoDB, Elasticsearch, Athena.

They are shared via the RestAPI (Elastic Beanstalk) endpoint.

● Managed “Monitoring and Alerting” of all the products deployed on AWS.

● Managed to integrate the Big Data Analytics products from one VPC to another.

● Customer Knowledge Graph (customer interaction data >250M historical and current

customers and represented by 1B+ nodes and 5B+ relationships, neo4j). Customers’

journey data was cleaned, processed (Kinesis, Apache Spark (EMR) Lambda, Redshift,

Athena), and populated into the Graph. The data is used by Data Science teams,

visualization, and the Recommendation Engine.

● Helped the Data Science team to generate smarter reports.

● Generated several dashboards (Tableau, Spotfire).

● Managed our team’s AWS Taskforce initiative to learn more about AWS products and how

they can help to make better products at lower costs.

● Given several Knowledge Sharing sessions on Serverless (AWS Lambda, Step Functions,

AWS Kinesis), new technologies, Monitoring, Spark Streaming, etc.

● Managed DevOps, QA, and cost analysis for the team.

Carrier IQ is the leading provider of Mobile Intelligence Solutions to the Wireless Industry. As the only embedded analytics company to support millions of devices simultaneously. Carrier IQ gives Wireless Carriers and handset Manufacturers valuable information about their customers’ mobile experience. Details are at http://www.carrieriq.com