Worked with Symantec's Cloud Platform Engineering group responsible for developing big-data cloud platform hosted in AWS and having ability to host and process petabytes of data into a secure data lake. Mainly responsible for the architecture, design of the big-data platform and ingestion pipelines running over the platform.
Currently the data-lake hosts multiple petabytes of data and is hosted on AWS. Data-lake allows applications to consume incoming events in real-time or in batch-mode. Initial version of the platform was hosted on private cloud built on top of Openstack.
Also architected and designed, from ground-up, Symantec's first marketplace for security applications to allow 3rd party developers to securely consume Symantec's data and offer valuable analytics to Symantec's customers.
Technology Stack: Apache Storm, Hadoop, Hive, Pig, Zookeeper, Kafka, Redis, Python, Java, AWS infrastructure and services, Openstack.
Also worked with the security and compliance BU on a product called Control Compliance Suite
Played a major role in architecture revamp of the product wherein multiple deployment layers were merged into a robust, efficient and minimal architecture. Played key role in coming up with high-level architecture, design and proposed/implemented several performance improvements scaling the product from 20-30k servers to 150k servers.
Technology Stack: C++ for agent and scan server, .Net/C# and related technologies for management server.