Jing Yu Koh
jingyuk@cs.cmu.edu
I am a research scientist on the Llama team at Meta, working on multimodal LLM agents with Ruslan Salakhutdinov and Daniel Fried. My goal is to develop general purpose agents that can automate computer work.
I'm on a leave of absence from a PhD in machine learning at Carnegie Mellon University. Prior to starting my PhD, I worked at Google Research in Jason Baldridge's team from 2019-2022, where I researched vision-and-language problems and generative models. Before that, I completed my undergraduate studies at the Singapore University of Technology and Design summa cum laude (highest honors) in 2019.
My first name is "Jing Yu" and informally I go by the nickname "JY". 许靖宇 is my name in Chinese. I'm from Singapore.
News
- (Aug 2024) Gave invited talks about Tree Search for LM Agents at UW, CMU, and Meta.
- (Jun 2024) Personal update: I've joined the Llama team at Meta to build multimodal agents!
- (Jun 2024) Excited to share a new preprint: Tree Search for Language Model Agents.
- (Spring 2024) Gave invited talks about VisualWebArena at NUS, Jane Street, and Cohere for AI (recording).
- (Feb 2024) Awarded the Jane Street Graduate Research Fellowship. Thank you Jane Street!
- (Jan 2024) Excited to share a new preprint: VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks.
- (Sep 2023) 1 paper accepted to NeurIPS 2023!
- (Summer 2023) Gave an invited talk about GILL at DLCT and Cohere For AI (slides, recording).
- (Apr 2023) 1 paper accepted to ICML 2023!
- (Spring 2023) Gave invited talks at Microsoft Research, Apple AI/ML, Georgia Tech, and the London ML Meetup (recording, slides).
- (Dec 2022) I made a bet on LLM capabilities with my office mate Ben Chugg. Bubble tea is on the line.
- (Nov 2022) Parti was accepted to TMLR with a Featured Certification!
- (Oct 2022) In the spirit of paying it forward, I'm sharing my Statement of Purpose publicly. Hope it helps future applicants!
- (Jul 2022) After 2.73 wonderful years at Google, I've left to pursue my PhD at Carnegie Mellon University!
- (January 2022) 1 paper accepted to ICLR 2022!
- (December 2021) Serving as a reviewer for CVPR 2022.
- (July 2021) 1 paper accepted to ICCV 2021!
- (July 2021) Presenting an invited talk at Microsoft Research.
- (July 2021) Serving as a reviewer for NeurIPS 2021.
- (March 2021) 1 paper accepted to CVPR 2021!
- (January 2021) 1 paper accepted to ICLR 2021!
- (October 2020) 1 paper accepted to WACV 2021!
- (July 2020) 1 paper accepted to ECCV 2020!
- (October 2019) Officially joined Google as an AI Resident in Mountain View, California.
Selected Publications [Google Scholar]
2024
VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks
ACL, 2024.
As seen on: Wired.
