Building the platform that enables data collection which is used in the training of frontier LLMs

Building large scale distributed database systems.

Part of the core data engineering team responsible for the metadata storage layer at Box.

Built micro-services with Scala/Java & Kubernetes that ensure high uptime of the DB cluster. Developed and managed data pipelines with Kafka for publishing database change events.

Worked on services to facilitate database sharding, monitoring, connection management.

Worked with Scala, Java, Python, Akka, ProxySQL, Slick ORM, Kafka, MySQL, Kubernetes. Provided engineering on call support for live services.

Developed algorithms and data pipelines for image processing and machine learning for scientific applications such as seismic volume interpretation & microchip fabrication.

Built a platform for the complete machine learning lifecycle including data collection, processing, exploratory analysis, training & managing ML models, deployment to production. Used Python and libraries like SciPy, NumPy, Keras, Dask, ZARR.

Worked with AWS technologies for training and deploying ML models.

• Worked on the development of a machine learning system for disease diagnosis & treatment.

• Optimizing and parallelizing algorithms for analyzing large data sets.

• Streamline deployments using robust CI/CD pipelines, Gitlab, Docker and Ansible.