Timezone: »
Textual grounding is an important but challenging task for human-computer inter- action, robotics and knowledge mining. Existing algorithms generally formulate the task as selection from a set of bounding box proposals obtained from deep net based systems. In this work, we demonstrate that we can cast the problem of textual grounding into a unified framework that permits efficient search over all possible bounding boxes. Hence, the method is able to consider significantly more proposals and doesn’t rely on a successful first stage hypothesizing bounding box proposals. Beyond, we demonstrate that the trained parameters of our model can be used as word-embeddings which capture spatial-image relationships and provide interpretability. Lastly, at the time of submission, our approach outperformed the current state-of-the-art methods on the Flickr 30k Entities and the ReferItGame dataset by 3.08% and 7.77% respectively.
Author Information
Raymond A. Yeh (University of Illinois at Urbana–Champaign)
Jinjun Xiong (IBM Research)
Wen-Mei Hwu
Minh Do (University of Illinois)
Alex Schwing (University of Illinois at Urbana-Champaign)
Related Events (a corresponding poster, oral, or spotlight)
-
2017 Oral: Interpretable and Globally Optimal Prediction for Textual Grounding using Image Concepts »
Wed. Dec 6th 12:35 -- 12:50 AM Room Hall A
More from the Same Authors
-
2021 Spotlight: Generic Neural Architecture Search via Regression »
Yuhong Li · Cong Hao · Pan Li · Jinjun Xiong · Deming Chen -
2021 Spotlight: Per-Pixel Classification is Not All You Need for Semantic Segmentation »
Bowen Cheng · Alex Schwing · Alexander Kirillov -
2023 Poster: A Simple Solution for Offline Imitation from Observations and Examples with Possibly Incomplete Trajectories »
Kai Yan · Alex Schwing · Yu-Xiong Wang -
2022 Poster: CEIP: Combining Explicit and Implicit Priors for Reinforcement Learning with Demonstrations »
Kai Yan · Alex Schwing · Yu-Xiong Wang -
2022 Poster: DigGAN: Discriminator gradIent Gap Regularization for GAN Training with Limited Data »
Tiantian Fang · Ruoyu Sun · Alex Schwing -
2022 Poster: Learnable Polyphase Sampling for Shift Invariant and Equivariant Convolutional Networks »
Renan A. Rojas-Gomez · Teck-Yian Lim · Alex Schwing · Minh Do · Raymond A. Yeh -
2022 Poster: On the Importance of Gradient Norm in PAC-Bayesian Bounds »
Itai Gat · Yossi Adi · Alex Schwing · Tamir Hazan -
2021 Poster: Generic Neural Architecture Search via Regression »
Yuhong Li · Cong Hao · Pan Li · Jinjun Xiong · Deming Chen -
2021 Poster: Bridging the Imitation Gap by Adaptive Insubordination »
Luca Weihs · Unnat Jain · Iou-Jen Liu · Jordi Salvador · Svetlana Lazebnik · Aniruddha Kembhavi · Alex Schwing -
2021 Poster: Per-Pixel Classification is Not All You Need for Semantic Segmentation »
Bowen Cheng · Alex Schwing · Alexander Kirillov -
2021 Poster: A Contrastive Learning Approach for Training Variational Autoencoder Priors »
Jyoti Aneja · Alex Schwing · Jan Kautz · Arash Vahdat -
2021 Poster: Why Lottery Ticket Wins? A Theoretical Perspective of Sample Complexity on Sparse Neural Networks »
Shuai Zhang · Meng Wang · Sijia Liu · Pin-Yu Chen · Jinjun Xiong -
2021 Poster: Class-agnostic Reconstruction of Dynamic Objects from Videos »
Zhongzheng Ren · Xiaoming Zhao · Alex Schwing -
2021 Poster: Perceptual Score: What Data Modalities Does Your Model Perceive? »
Itai Gat · Idan Schwartz · Alex Schwing -
2020 Poster: Not All Unlabeled Data are Equal: Learning to Weight Data in Semi-supervised Learning »
Zhongzheng Ren · Raymond A. Yeh · Alex Schwing -
2020 Poster: Towards a Better Global Loss Landscape of GANs »
Ruoyu Sun · Tiantian Fang · Alex Schwing -
2020 Oral: Towards a Better Global Loss Landscape of GANs »
Ruoyu Sun · Tiantian Fang · Alex Schwing -
2020 Session: Orals & Spotlights Track 22: Vision Applications »
Leonid Sigal · Alex Schwing -
2020 Poster: Removing Bias in Multi-modal Classifiers: Regularization by Maximizing Functional Entropies »
Itai Gat · Idan Schwartz · Alex Schwing · Tamir Hazan -
2020 Poster: High-Throughput Synchronous Deep RL »
Iou-Jen Liu · Raymond A. Yeh · Alex Schwing -
2019 : Poster Session »
Pravish Sainath · Mohamed Akrout · Charles Delahunt · Nathan Kutz · Guangyu Robert Yang · Joseph Marino · L F Abbott · Nicolas Vecoven · Damien Ernst · andrew warrington · Michael Kagan · Kyunghyun Cho · Kameron Harris · Leopold Grinberg · John J. Hopfield · Dmitry Krotov · Taliah Muhammad · Erick Cobos · Edgar Walker · Jacob Reimer · Andreas Tolias · Alexander Ecker · Janaki Sheth · Yu Zhang · Maciej Wołczyk · Jacek Tabor · Szymon Maszke · Roman Pogodin · Dane Corneil · Wulfram Gerstner · Baihan Lin · Guillermo Cecchi · Jenna M Reinen · Irina Rish · Guillaume Bellec · Darjan Salaj · Anand Subramoney · Wolfgang Maass · Yueqi Wang · Ari Pakman · Jin Hyung Lee · Liam Paninski · Bryan Tripp · Colin Graber · Alex Schwing · Luke Prince · Gabriel Ocker · Michael Buice · Benjamin Lansdell · Konrad Kording · Jack Lindsey · Terrence Sejnowski · Matthew Farrell · Eric Shea-Brown · Nicolas Farrugia · Victor Nepveu · Jiwoong Im · Kristin Branson · Brian Hu · Ramakrishnan Iyer · Stefan Mihalas · Sneha Aenugu · Hananel Hazan · Sihui Dai · Tan Nguyen · Doris Tsao · Richard Baraniuk · Anima Anandkumar · Hidenori Tanaka · Aran Nayebi · Stephen Baccus · Surya Ganguli · Dean Pospisil · Eilif Muller · Jeffrey S Cheng · Gaël Varoquaux · Kamalaker Dadi · Dimitrios C Gklezakos · Rajesh PN Rao · Anand Louis · Christos Papadimitriou · Santosh Vempala · Naganand Yadati · Daniel Zdeblick · Daniela M Witten · Nicholas Roberts · Vinay Prabhu · Pierre Bellec · Poornima Ramesh · Jakob H Macke · Santiago Cadena · Guillaume Bellec · Franz Scherr · Owen Marschall · Robert Kim · Hannes Rapp · Marcio Fonseca · Oliver Armitage · Jiwoong Im · Thomas Hardcastle · Abhishek Sharma · Wyeth Bair · Adrian Valente · Shane Shang · Merav Stern · Rutuja Patil · Peter Wang · Sruthi Gorantla · Peter Stratton · Tristan Edwards · Jialin Lu · Martin Ester · Yurii Vlasov · Siavash Golkar -
2019 : Coffee Break & Poster Session 2 »
Juho Lee · Yoonho Lee · Yee Whye Teh · Raymond A. Yeh · Yuan-Ting Hu · Alex Schwing · Sara Ahmadian · Alessandro Epasto · Marina Knittel · Ravi Kumar · Mohammad Mahdian · Christian Bueno · Aditya Sanghi · Pradeep Kumar Jayaraman · Ignacio Arroyo-Fernández · Andrew Hryniowski · Vinayak Mathur · Sanjay Singh · Shahrzad Haddadan · Vasco Portilheiro · Luna Zhang · Mert Yuksekgonul · Jhosimar Arias Figueroa · Deepak Maurya · Balaraman Ravindran · Frank NIELSEN · Philip Pham · Justin Payan · Andrew McCallum · Jinesh Mehta · Ke SUN -
2019 : Contributed Talk - Chirality Nets: Exploiting Structure in Human Pose Regression »
Raymond A. Yeh · Yuan-Ting Hu · Alex Schwing -
2019 : Poster Session »
Jonathan Scarlett · Piotr Indyk · Ali Vakilian · Adrian Weller · Partha P Mitra · Benjamin Aubin · Bruno Loureiro · Florent Krzakala · Lenka Zdeborová · Kristina Monakhova · Joshua Yurtsever · Laura Waller · Hendrik Sommerhoff · Michael Moeller · Rushil Anirudh · Shuang Qiu · Xiaohan Wei · Zhuoran Yang · Jayaraman Thiagarajan · Salman Asif · Michael Gillhofer · Johannes Brandstetter · Sepp Hochreiter · Felix Petersen · Dhruv Patel · Assad Oberai · Akshay Kamath · Sushrut Karmalkar · Eric Price · Ali Ahmed · Zahra Kadkhodaie · Sreyas Mohan · Eero Simoncelli · Carlos Fernandez-Granda · Oscar Leong · Wesam Sakla · Rebecca Willett · Stephan Hoyer · Jascha Sohl-Dickstein · Sam Greydanus · Gauri Jagatap · Chinmay Hegde · Michael Kellman · Jonathan Tamir · Nouamane Laanait · Ousmane Dia · Mirco Ravanelli · Jonathan Binas · Negar Rostamzadeh · Shirin Jalali · Tiantian Fang · Alex Schwing · Sébastien Lachapelle · Philippe Brouillard · Tristan Deleu · Simon Lacoste-Julien · Stella Yu · Arya Mazumdar · Ankit Singh Rawat · Yue Zhao · Jianshu Chen · Xiaoyang Li · Hubert Ramsauer · Gabrio Rizzuti · Nikolaos Mitsakos · Dingzhou Cao · Thomas Strohmer · Yang Li · Pei Peng · Gregory Ongie -
2019 Poster: Chirality Nets for Human Pose Regression »
Raymond A. Yeh · Yuan-Ting Hu · Alex Schwing -
2019 Poster: Graph Structured Prediction Energy Networks »
Colin Graber · Alex Schwing -
2019 Poster: STREETS: A Novel Camera Network Dataset for Traffic Flow »
Corey Snyder · Minh Do -
2019 Spotlight: STREETS: A Novel Camera Network Dataset for Traffic Flow »
Corey Snyder · Minh Do -
2019 Poster: TAB-VCR: Tags and Attributes based Visual Commonsense Reasoning Baselines »
Jingxiang Lin · Unnat Jain · Alex Schwing -
2019 Poster: Co-Generation with GANs using AIS based HMC »
Tiantian Fang · Alex Schwing -
2018 Poster: Deep Structured Prediction with Nonlinear Output Transformations »
Colin Graber · Ofer Meshi · Alex Schwing -
2018 Poster: Pipe-SGD: A Decentralized Pipelined SGD Framework for Distributed Deep Net Training »
Youjie Li · Mingchao Yu · Songze Li · Salman Avestimehr · Nam Sung Kim · Alex Schwing -
2018 Poster: Out of the Box: Reasoning with Graph Convolution Nets for Factual Visual Question Answering »
Medhini Narasimhan · Svetlana Lazebnik · Alex Schwing -
2018 Poster: GradiVeQ: Vector Quantization for Bandwidth-Efficient Gradient Aggregation in Distributed CNN Training »
Mingchao Yu · Zhifeng Lin · Krishna Narra · Songze Li · Youjie Li · Nam Sung Kim · Alex Schwing · Murali Annavaram · Salman Avestimehr -
2017 Poster: Dualing GANs »
Yujia Li · Alex Schwing · Kuan-Chieh Wang · Richard Zemel -
2017 Spotlight: Dualing GANs »
Yujia Li · Alex Schwing · Kuan-Chieh Wang · Richard Zemel -
2017 Poster: MaskRNN: Instance Level Video Object Segmentation »
Yuan-Ting Hu · Jia-Bin Huang · Alex Schwing -
2017 Poster: Asynchronous Parallel Coordinate Minimization for MAP Inference »
Ofer Meshi · Alex Schwing -
2017 Poster: High-Order Attention Models for Visual Question Answering »
Idan Schwartz · Alex Schwing · Tamir Hazan -
2017 Poster: Diverse and Accurate Image Description Using a Variational Auto-Encoder with an Additive Gaussian Encoding Space »
Liwei Wang · Alex Schwing · Svetlana Lazebnik -
2016 Poster: Constraints Based Convex Belief Propagation »
Yaniv Tenzer · Alex Schwing · Kevin Gimpel · Tamir Hazan -
2016 Poster: Learning Deep Parsimonious Representations »
Renjie Liao · Alex Schwing · Richard Zemel · Raquel Urtasun -
2015 Poster: Smooth and Strong: MAP Inference with Linear Convergence »
Ofer Meshi · Mehrdad Mahdavi · Alex Schwing -
2014 Poster: Efficient Inference of Continuous Markov Random Fields with Polynomial Potentials »
Shenlong Wang · Alex Schwing · Raquel Urtasun -
2014 Poster: Message Passing Inference for Large Scale Graphical Models with High Order Potentials »
Jian Zhang · Alex Schwing · Raquel Urtasun -
2013 Poster: Latent Structured Active Learning »
Wenjie Luo · Alex Schwing · Raquel Urtasun -
2012 Poster: Probabilistic Low-Rank Subspace Clustering »
S. Derin Babacan · Shinichi Nakajima · Minh Do -
2012 Poster: Globally Convergent Dual MAP LP Relaxation Solvers using Fenchel-Young Margins »
Alex Schwing · Tamir Hazan · Marc Pollefeys · Raquel Urtasun