Inference for multimodal models, specifically Nova Sonic and Nova 2 Sonic

Building platform for conversational AI for phone calls, orchestrating LLMs, transcription models, and voice synthesis models. Joined as one of the first technical hires, focusing on core technical challenges in model performance, conversation latency, system scalability, and reliability. Notably focusing on fine-tuning models (customer agents, voicemail detection), ML lifecycle pipelines (data collection/cleaning, training infra, model evaluation suites), observability (Sentry integration, system logging), and core features (integration of model providers like Anthropic, model fallback mechanisms, interruption handling, etc.). Led internal and customer projects, designed our engineering on-call process, helped with candidate interviews and hiring, and contributed to open-source repo (https://github.com/vocodedev/vocode-core).

Researched 3D topological reconstruction techniques using foundation and large-language models. This research was done prior to the release of modern multimodal LLMs like GPT4-vision.

Designed a novel process to retrieve CAD models and iteratively orient and align them in images, utilizing the robust nature of LLMs and foundation models trained on massive-scale knowledge sources.

Orchestrated multiple SotA models for CV and vision-language tasks including GroundingDINO, Segment Anything, InstructBLIP, ZoeDepth.

Built custom scripts for synthetic data generation and iterative rendering in Blender, and conducted multiple experiements to evaluate retrieval and alignment accuracy, running on an A100 HPC cluster.

Conducted computer vision research with Dr. Alvin Wan under Prof. Joseph Gonzalez. Researched using temporal Neural Radiance Fields (NeRFs) for novel-view synthesis and generation of unseen vehicle trajectories from existing images captured by self-driving vehicles. Additionally researched using panoramic stitching to improve efficiency of multi-view computer vision for self-driving vehicles.