● Designed and implemented ETL pipelines using PySpark in Kubernetes clusters, processing 10 TB raw stock dataset

● Developed three optimized internal tools for researchers using Python & C++, optimizing data processing time by 60%

● Implemented a scalable and efficient API service for stock data management, utilizing Python and deployed on Ruitian Cloud for increased performance and scalability

● Utilized advanced Docker features such as network and data volume management to ensure data consistency and quality

As a part of this job role, I enhance application performance and attract direct investment of $ 54K by executing new API routes, architecting new ORM structures, and refactoring code.

One of my key tasks is to clean/manipulate raw data and implement feature selection algorithms from 20 TB data of French election data with preprocessors.

My achievements are as follows:

● Implemented a robust application for analyzing Twitter users' demographics and psychographics using NLP and Hadoop

● Conducted MapReduce jobs on a 20 TB raw dataset from Twitter, significantly reducing data processing time by 80%

● Improved Twitter User prediction accuracy to 95% through the application of FrameAxis machine learning algorithm

● Fulfilled the $54K contract of INCAS project offered by DAPRA, meeting all expectations from the stakeholders

During this tenure, I supported back-end communication by developing REST API.

It is worth mentioning that I initialized AWS servers and managed databases as DBA.

It was me who identified and resolved issues to streamline processes.

My key successes for this role are:

● Developed a web-based stock risk analysis platform using Django and React to provide valuable insights for traders

● Improved data retrieval speed by 50% from AWS through the implementation of database indexing and caching strategies

● Deployed and maintained RDS databases to host large amounts of stock data, ensuring a routine dataflow of up to 20 Gb