Seasoned generalist developer with 13+ years of experience in different areas of backend development, be it working on cloud services, building managed cloud services providing infrastructure to run large scale services, building platforms that support running large scale services, Or working with big data at...
2022 — Now
Cupertino, California, United States
Developer in AIML Data Platform and Infrastructure team
Authored and managed Apache Flink based streaming pipeline to process real-time ingestion of trillions of Siri messages per day.
Built custom framework (based on Apache Calcite) to generate data pipelines helping in real-time ingestion of trillions of messages per day which can be converted to either Apache Spark or Apache Flink workflow in a self serve fashion using only configs.
Backend developer in the Apple iCloud Services group.
Own (and authored) Apache Spark jobs working on big data in real time (Millions [double digit] of events per second)
Own (and authored) Apache Airflow pipelines syncing the spark aggregated data to Apache Druid, compacting the data
Own (and manage) the Apache Druid data, compaction, and overall lifecycle of the data
Own (and authored) the service to deliver the insights from aforementioned aggregated data to users with a capability to slice and dice. The service works on a pluggable model supporting scenarios for multiple teams that need to connect to different databases and convert the data to a common data model.
Own (and authored) client library to talk to Apache Superset programmatically, which is published within the company and multiple teams are building their service on top of it.
Own (and authored) multiple services used internally by several partner teams across multiple orgs.
2020 — 2022
Greater Seattle Area
Working as a software engineer to provide managed Azure offering of PostgreSql DB as a service.
1. Part of the team working on moving the platform to serverless infra instead of managing our own
infrastructure for long term viability.
2. Working on security aspects of the service to ensure service can be GA.
3. Reduced the time it takes to provision a DB instance from >5 mins to under 2.5 mins by caching the pre-provisioned instances and assigning them to user on demand.
2017 — 2020
Greater Seattle Area
Part of Azure Service Fabric team.
Azure Service Fabric is a distributed systems platform ; akin to Kubernetes. Major contributions:
1. Service Fabric Mesh
i. Implemented monitoring story for the multiple microservices of Service Fabric Mesh Resource Provider.
ii. Designed and implemented a new microservice to manage the MSI Identity for a SF Mesh application. The code was implemented abstractly using interfaces, so that, other resource provider services and make use of the code and provide MSI identity for their resources.
2. Designed and implemented the APIs to expose container (both docker and clear) stats, and events to users via Azure portal.
3. Designed and implemented the APIs to expose container (both docker and clear) logs via a custom log driver.
i. This empowered users to bring their own sidecars to collect logs.
ii. In addition to it, worked on infrastructure to enable one touch experience for users where they tell platform which diagnostics pipeline to use and platform enables it for users.
4. Port managed C# code to CoreCLR net standard v2.0. The porting enabled activation of many functionalities as well as increased the code coverage of tests on linux.
5. Designed and implemented changes to introduce a first class concept where a node type in SF cluster can be declared to support only stateless workloads. This allowed the system to treat stateless workloads differently than stateful workloads and speed up actions related to stateless workloads.
i. This required changes at multiple layers of the system, viz, Service Fabric Resource Provider service, SF runtime which generated cluster manifest as well as load balancer to treat stateless workloads differently.
6. Support low priority VMs for Service Fabric clusters.
Open source contributions:
Implemented docker plugins API for Docker.Dotnet library for C#.
https://github.com/Microsoft/Docker.DotNet/releases/tag/3.125.2
2014 — 2017
Greater Seattle Area
As part of Managed Service Identity (MSI).
1. Member of two people team owning one of the Azure's core infra services.
2. Service's idea was to automate the process of providing an "Identity" to Azure resources. The same identity can then be used to identify the owning resource through AAD and allow access to other resources.
3. Being a new service, I worked on it from scratch. Covered all aspects of the service, including setting up service basics, DR story, monitoring and alerts, in addition to designing and implementing the features.
2013 — 2014
In Kiwi, I worked on android games.
Highlights:
1) Design and implement new features in the game in Java using libgdx platform. Few major ones are:
a) Player segmentation - Segregate users based on their game play in the game and provide different features and deals based on this info.
b) Ranking of players - Rank players for the feature "team challengers" on the basis of score they are achieving in the game.
2) In addition to working on the client side, I also worked on server side using Php, mysql and Redis (NoSql db).
Education
Indian Institute of Technology (BHU), Varanasi