Engineered 8-stage data mining pipeline for neural machine translation evaluation.

Implemented bi-text data preprocessing, SPM Tokenization, Model-Vocab generation, and model training for potentially millions of sentences in Python, Bash & Perl.

Built registry for 100+ Jobs in Python: CRUD registration, querying, batched jobs failure handlers

Our work was recently open-sourced: https://github.com/facebookresearch/stopes

Docs: https://facebookresearch.github.io/stopes/