Jing Yu Koh

jingyuk@cs.cmu.edu

I am a first year PhD student in the Machine Learning Department at Carnegie Mellon University. I am advised by Daniel Fried and Ruslan Salakhutdinov. I work on grounded language understanding, usually in the context of vision-and-language problems.

Prior to this, I was a Research Engineer (and previously an AI Resident) at Google Research, where I worked on vision-and-language problems and generative models. I completed my undergraduate studies at the Singapore University of Technology and Design summa cum laude (highest honors) in 2019.

My first name is "Jing Yu" and informally I go by the nickname "JY". I'm from Singapore.

News

  • (Feb - Mar 2023) Gave an invited talk at Microsoft Research, Apple AI/ML, Georgia Tech, and the London ML Meetup (recording, slides).
  • (Jan 2023) New preprint! We ground LLMs to enable multimodal processing and generation.
  • (Dec 2022) I made a bet on LLM capabilities with my office mate Ben Chugg. Bubble tea is on the line.
  • (Dec 2022) 1 paper accepted to AAAI 2023.
  • (Nov 2022) Parti was accepted to TMLR with a Featured Certification!
  • (Oct 2022) In the spirit of paying it forward, I'm sharing my Statement of Purpose publicly. Hope it helps future applicants!
  • (July 2022) After 2.73 wonderful years at Google, I've left to pursue my PhD at Carnegie Mellon University!
  • (January 2022) 1 paper accepted to ICLR 2022!
  • (December 2021) Serving as a reviewer for CVPR 2022.
  • (July 2021) 1 paper accepted to ICCV 2021!
  • (July 2021) Presenting an invited talk at Microsoft Research.
  • (July 2021) Serving as a reviewer for NeurIPS 2021.
  • (March 2021) 1 paper accepted to CVPR 2021!
  • (January 2021) 1 paper accepted to ICLR 2021!
  • (October 2020) 1 paper accepted to WACV 2021!
  • (July 2020) 1 paper accepted to ECCV 2020!
  • (October 2019) Officially joined Google Research in Mountain View, California.

Selected Publications [Google Scholar]

2023

Grounding Language Models to Images for Multimodal Generation

Grounding Language Models to Images for Multimodal Generation

Jing Yu Koh, Ruslan Salakhutdinov, Daniel Fried

Preprint, 2023.

Simple and Effective Synthesis of Indoor 3D Scenes

Simple and Effective Synthesis of Indoor 3D Scenes

Jing Yu Koh*, Harsh Agrawal*, Dhruv Batra, Richard Tucker, Austin Waters, Honglak Lee, Yinfei Yang, Jason Baldridge, Peter Anderson (* denotes equal contribution)

In AAAI Conference on Artificial Intelligence (AAAI), 2023.

2022

Scaling Autoregressive Models for Content-Rich Text-to-Image Generation

Scaling Autoregressive Models for Content-Rich Text-to-Image Generation

Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, Yonghui Wu

In Transactions of Machine Learning Research (TMLR), 2022.

2021

Pathdreamer: A World Model for Indoor Navigation

Pathdreamer: A World Model for Indoor Navigation

Jing Yu Koh, Honglak Lee, Yinfei Yang, Jason Baldridge, Peter Anderson

In The International Conference on Computer Vision (ICCV), 2021.

Vector-quantized Image Modeling with Improved VQGAN

Vector-quantized Image Modeling with Improved VQGAN

Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, Yonghui Wu

In The International Conference on Learning Representations (ICLR), 2022.

PDF
Cross-Modal Contrastive Learning for Text-to-Image Generation

Cross-Modal Contrastive Learning for Text-to-Image Generation

Han Zhang*, Jing Yu Koh*, Jason Baldridge, Honglak Lee, Yinfei Yang (* denotes equal contribution)

In The Conference on Computer Vision and Pattern Recognition (CVPR), 2021.

Text-to-Image Generation Grounded by Fine-Grained User Attention

Text-to-Image Generation Grounded by Fine-Grained User Attention

Jing Yu Koh, Jason Baldridge, Honglak Lee, Yinfei Yang

In The IEEE Winter Conference on Applications of Computer Vision (WACV), 2021.

2020

SideInfNet

SideInfNet: A Deep Neural Network for Semi-Automatic Semantic Segmentation with Side Information

Jing Yu Koh, Duc Thanh Nguyen, Quang-Trung Truong, Sai-Kit Yeung, Alexander Binder

In The European Conference on Computer Vision (ECCV), 2020.


Projects

ModelZoo

Model Zoo curates pre-trained deep learning models and code, making it easy for researchers to find models for various frameworks.

Web Application
Yu Sheng

The CNY Yusheng app is a reference app designed to help you learn more about unique Chinese New Year traditions and the yusheng dish.

iOS Application
Pixel Warrior

Castle-defense strategy game for iOS. Permanent death and level randomisation make no two games alike.

iOS Game