( events)   Timezone: »  
Mon Dec 13 08:00 AM -- 12:00 AM (PST)
CtrlGen: Controllable Generative Modeling in Language and Vision
Steven Y. Feng · Drew Arad Hudson · Tatsunori Hashimoto · DONGYEOP Kang · Varun Prashant Gangal · Anusha Balakrishnan · Joel Tetreault

Over the past few years, there has been an increased interest in the areas of language and image generation within the community. As generated texts by models like GPT-3 start to sound more fluid and natural, and generated images and videos by GAN models appear more realistic, researchers began focusing on qualitative properties of the generated content such as the ability to control its style and structure, or incorporate information from external sources into the output. Such aims are extremely important to make language and image generation useful for human-machine interaction and other real-world applications including machine co-creativity, entertainment, reducing biases or toxicity, and improving conversational agents and personal assistants.

Achieving these ambitious but important goals introduces challenges not only from NLP and Vision perspectives, but also ones that pertain to Machine Learning as a whole, which has witnessed a growing body of research in relevant domains such as interpretability, disentanglement, robustness, and representation learning. We believe that progress towards the realization of human-like language and image generation may benefit greatly from insights and progress in these and other ML areas.

In this workshop, we propose to bring together researchers from the NLP, Vision, and ML communities to discuss the current challenges and explore potential directions for controllable generation and improve its quality, correctness, and diversity. As excitement about language and image generation has significantly increased recently thanks to the advent and improvement of language models, Transformers, and GANs, we feel this is the opportune time to hold a new workshop about this subject. We hope CtrlGen will foster discussion and interaction across communities, and sprout fruitful cross-domain relations that open the door for enhanced controllability in language and image generation.

Opening Remarks (Short Intro)
Invited Talk #1 - Control in Dialogue: When does it work? (Jason Weston) (Invited Talk)
Invited Talk #1 Q&A (Short Q&A)
Invited Talk #2 - Disentangling Faithfulness and Extractiveness in Abstractive Summarization (He He) (Invited Talk)
Invited Talk #2 Q&A (Short Q&A)
Invited Talk #3 - Disentanglement for Controllable Image Generation (Irina Higgins) (Invited Talk)
Invited Talk #3 Q&A (Short Q&A)
Invited Talk #4 - Neuro-Logic and Differentiable Controls (Yejin Choi) (Invited Talk)
Invited Talk #4 Q&A (Short Q&A)
Virtual Coffee/Networking Break (Break)
Discussion Panel and QA Session (Discussion Panel)
Virtual Poster Session #1 (Poster Session)
Lunch Break (Break)
Demonstrations (Live-Streamed Demos)
Invited Talk #5 - Off the Beaten Path: Domain-Agnostic ML for Controllable Generation and Beyond (Alex Tamkin) (Invited Talk)
Invited Talk #5 Q&A (Short Q&A)
Invited Talk #6 - Generating and Editing Images Using StyleGAN and CLIP (Or Patashnik) (Invited Talk)
Invited Talk #6 Q&A (Short Q&A)
Virtual Coffee/Networking Break (Break)
Virtual Poster Session #2 (Poster Session)
Invited Talk #7 - Controllable Text Generation with Multiple Constraints (Yulia Tsvetkov) (Invited Talk)
Invited Talk #7 Q&A (Short Q&A)
Best Paper Awards and Closing Remarks (Closing Remarks)
GatherTown Open for Continued Socializing (Networking and Socializing)
Learning to Compose Visual Relations (Poster)
Hamiltonian prior to Disentangle Content and Motion in Image Sequences (Poster)
LUMINOUS: Indoor Scene Generation for Embodied AI Challenges (Poster)
Fair Data Generation using Language Models with Hard Constraints (Poster)
Controllable Paraphrase Generation with Multiple Types of Constraints (Poster)
Controlling Conditional Language Models with Distributional Policy Gradients (Poster)
Continuous Emotion Transfer Using Kernels (Poster)
XCI-Sketch: Extraction of Color Information from Images for Generation of Colored Outlines and Sketches (Poster)
Sampling from Discrete Energy-Based Models with Quality/Efficiency Trade-offs (Poster)
Learning Representations for Zero-Shot Image Generation without Text (Poster)
Controlled Cue Generation for Play Scripts (Poster)
Diamond in the rough: Improving image realism by traversing the GAN latent space (Poster)
Neural Abstructions: Abstractions that Support Construction for Grounded Language Learning (Poster)
Self-supervised Enhancement of Latent Discovery in GANs (Poster)
PPL-MCTS: Constrained Textual Generation Through Discriminator-Guided Decoding (Poster)
C^3: Contrastive Learning for Cross-domain Correspondence in Few-shot Image Generation (Poster)
Sound-Guided Semantic Image Manipulation (Poster)
Towards Unsupervised Content Disentanglement in Sentence Representations via Syntactic Roles (Poster)
MIDI-DDSP: Hierarchical Modeling of Music for Detailed Control (Poster)
Robust Text Generation using Sequence-to-Sequence Pre-Training (Poster)