New York City Metropolitan Area
Fixed Income Real time alerts team
Rewrote our teams Cassandra client and achieved these results:
Reduced Cassandra reads team by 20x for a batch of 100 securities by caching prepared statements and performing them concurrently, BEFORE: 50th percentile, 90th percentile AFTER: 50th percentile, 90th percentile (brought job’s total processing time down from 3 hours to 30 minutes)
Performed async Cassandra writes and cached prepared statement, achieving 8x speed enhancement for a batch of 100 securities. BEFORE: AFTER: (brought job’s total time down from 30 minutes to 10 minutes)
Identified bottlenecks throughout ETL pipeline for (BVAL GSAC) job through grafana, and increased relevant Kafka instances bringing total e2e time down from 9 hours to ~40-50 minutes
Charged Cassandra compaction strategy saving 143k/year reducing droppable tombstone ratio from 95% down to 12%. Decreased the amount of SSTables per read down from ~20 to 4 speeding up median read latency by around 27%.
Threaded API calls to service scaling up application to allow users to request up to 2.4 million securities rather than 500k before timing out
Setup teams docker containers for integration testing, allowing cleaner integration testing by allowing for api requests to route to containers, mocking db containers, and mocking a Cassandra db.