# Ben Rachbach > Machine Learning and Evaluation Product Manager Location: San Francisco Bay Area, United States Profile: https://flows.cv/benrachbach Working to put our long-term future on a better track. I've worked as a software engineer, as a researcher, and as the first employee of a startup getting it off the ground. I currently work to make Elicit's (elicit.com) answers more accurate and vettable. ## Work Experience ### Machine Learning and Evaluation Product Manager @ Elicit Jan 2021 – Present | San Francisco Bay Area I product manage the team that makes Elicit give accurate and vettable answers to users' research questions • Set the roadmap for ML features and eval. I talk to key customers to learn how we need to improve Elicit's answers, then I translate this into specs for the ML team and eval designs • Started and lead the 2-person eval team at Elicit. This team creates evals for all Elicit features that let an ML engineer run a quick command to learn whether the change they just made was an improvement, and to have justified confidence in that answer • Led our work on data extraction from its inception -- worked very closely with key users to build a minimum viable product that fit their needs, then product-managed extending the MVP. Data extraction is now the core feature of Elicit and we make constant improvements, e.g. https://email.elicit.com/deliveries/dgTd9ggDANJo0WgBjbfL0NY-iZqxBk4O86BA • Designed the tasks and evals for our paper on answering questions about papers using iterated decomposition: https://arxiv.org/pdf/2301.01751 • Designed and created the dataset for a high-fidelity approach to evaluate Elicit's search which allowed us to improve precision by around 78% (see https://email.elicit.com/deliveries/dgTd9ggDAKHze6DzewGQc_h8U7zEMbsdsvUZ6pw=) (When Ought spun off Elicit, I moved along with the entire team from Ought to Elicit. That happened Summer 2023, but my role transitioned earlier so I've gone with the role transition date here) ### Technical Product Manager @ Ought Jan 2020 – Jan 2021 | San Francisco Bay Area Ought pivoted to building a forecasting product. I worked as a technical product manager, data scientist, and software engineer on the new product • Led the team that won Metaculus's El Paso COVID forecasting competition: https://pandemic.metaculus.com/questions/4161/el-paso-series-supporting-covid-19-response-planning-in-a-mid-sized-city/ • Launched the first prototype of literature review in Elicit, which led us to pivot to literature review ### Experimenter @ Ought Jan 2019 – Jan 2020 | San Francisco Bay Area Led our research on Factored Cognition/Iterated Distillation and Amplification. This research was Ought's main focus as an organization. Our main report on this research: https://ought.org/updates/2020-01-11-arguments • Set and carried out our research plan. Created ~weekly loop of running an experiment, analyzing the results, and then running a tweaked experiment • Managed a team of ~15 contractors pretending to be AIs in our experiments (pre-GPT-3!). Created onboarding curriculum to teach this large part-time team to do some really odd work (to trick each other as part of our experiments) We got negative results from these experiments, so we abandoned them ### 1st employee @ Ought Jan 2018 – Jan 2019 | San Francisco Bay Area Helped get Elicit off the ground. Mostly ops and hiring. • Put fundamental policies and procedures in place (e.g. Gusto for payroll and HR) • Created our first ops handbook ### Co-Founder @ Sovereign Finance Jan 2018 – Jan 2018 Explored using satellite imagery for economic development: https://medium.com/alan-do/empowering-emerging-economies-ai-satellite-imagery-and-economic-analysis-311e168465cf ### Software Engineer @ Wonder Workshop Jan 2016 – Jan 2018 | San Francisco Bay Area iOS/Android/Chromebook/Windows app development • Led team of developers to create a web IDE for programming robots (React, Redux, TypeScript, S3, Web Bluetooth): https://code.makewonder.com/cue • Headed cross-functional team to define and scope company’s first app aimed at schools • Led performance team of 5 engineers. Reduced app's response time to user input by 66% • Analyzed app analytics data and redesigned educational challenges to cut user failure rate by 17% • Launched cue iOS/Android app on schedule to support hardware launch Business Efficiency Improvements • Architected and built web purchase order system processing $1M+ in annual orders • Process automation: Halved time for staff to review purchase orders Website Development • Managed all aspects of website development, including engineering, scoping and prioritizing requests with stakeholders, and DevOps (AngularJS, Node/Express, PostgreSQL, Chai, AWS EC2/ELB, Ansible). Site gets over 1M monthly page views. • Optimized page load time with caching and automatic image compression • Modernized backend: upgraded all sites from Node 0 to Node 6 What this all means: I created tools for kids to program and invent with robots! ### Refugee Aid Design Consultant @ Various Jan 2015 – Jan 2016 | Haiti // Uganda Devised program prototypes for the American Refugee Committee and Soylent CSR based on Human-Centered Design research in refugee camps in Haiti and Uganda. ### Research Analyst @ GiveWell Jan 2013 – Jan 2016 | San Francisco Bay Area • Led research on salt iodization charities that helped lead donors to give over $1M in 2016 • Analyzed the scientific literature to prioritize philanthropic causes ranging from climate change to increasing labor mobility ### Math Tutor @ Match Community Day School Jan 2012 – Jan 2013 Put together curriculum and lesson plans and tutored ELL 3rd graders in math. 90% passed the state standardized exam. ### Research Fellow @ Fulbright Association Jan 2011 – Jan 2012 Led qualitative research team to improve math and Chinese learning software in rural Chinese schools ## Education ### B.A. with High Honors in Chinese and Education Studies Swarthmore College ### Catalina Foothills High School ### Software Engineering Hack Reactor ## Contact & Social - LinkedIn: https://linkedin.com/in/brachbach - Portfolio: https://benrachblogblog.wordpress.com/ --- Source: https://flows.cv/benrachbach JSON Resume: https://flows.cv/benrachbach/resume.json Last updated: 2026-03-31