High Performance Computing in the Cloud

Smart Grid Modeling and Simulation Project - Sponsored by U.S. Department of Energy. The project goal is to use industry standard data and novel three-phase model for power system dynamics simulation to model and simulate large-scale power grids. As a team leader, I was involved in two parts of job. First is to design the data structure for industry standardized data. Second is to implement the critical modules of the software simulator including dynamic initialization, Y-bus computation and 3-phase Y-bus computation. Meanwhile mentoring two other entry-level C programmers was also a great part of my responsibility.

Environment:

PETSc, Linux C

A Scalable Performance Anomaly Detection Framework in Hadoop via Nonparametric Clustering:

Our work aims at providing a scalable and practical performance anomaly detection framework for cloud environments. In contrast to traditional detection schemes, our design features a decentralized analysis based on hierarchical grouping and a novel anomaly identification method via non-parametric clustering. We evaluate our framework in a Hadoop environment under various workloads. The experimental results show that our design outperforms traditional methods significantly in the context of complex anomaly patterns and high failure probability.

Responsibilities:

• Establish the experiment environment which is a 64 node cluster computer in the SCS lab.

• Deploy Hadoop and its Map/Reduce framework on all 64 nodes.

• Write benchmark monitor program to monitor CPU, memory, disk I/O and network I/O.

• Write benchmark monitor program to monitor Hadoop I/O and number of tasks being assigned by Map/Reduce framework.

• Write programs to simulate CPU hog, memory leak, disk hog and network hog.

• Inject the fault into certain amount of node and use our decentralized analysis and novel anomaly identification method to see if those fault nodes can be quickly detected.

Environment:

Java 1.6, Hadoop 0.20, Ubantu Linux, Linux C

Bill Payment System:

The system enables Standard Chartered Bank’s personal client paying all kinds of utility bills, including gas, electricity, water, land line or mobile phone bills via their online banking system. Customer can choose to pay each bill by themselves or enroll in auto recursive payment so they won’t miss the due date any more. The technical challenge of the project is to establish a new payment gateway through Fu Fei Tong, an external bill collecting system. At the mean time, all the business logic including online bill payment transactions, day end batch reconciliation and report issuing needs to be carefully designed so that all the exceptional case can be handled by business operation team.

Responsibilities:

• Conducting requirement elicitation meetings with business operation team.

• Writing technical documents including BRD(Business Requirement Document), FSD(Function Specification Document) and Architecture Design Document.

• Do project planning and cost estimation.

• Do architecture design by creating Architecture Design Document.

• Do system decomposition, task assigning and track the progress during the whole project.

• Write test plan and conduct system integration testing and user acceptance testing. Tracking defects and assigning bug fixing task.

• Go production and do system maintenance.

Environment:

Frontend: Java EE

Backend: C on IBM AIX

Database: Oracle 9i