We propose a method that can generate an unambiguous description (known as a referring expression) of a specific object or region in an image, and which can also comprehend or interpret such an expression to infer which object is being described.

We show that our method outperforms previous methods that generate descriptions of objects without taking into account other potentially ambiguous objects in the scene.

Our model is inspired by recent successes of deep learning methods for image captioning, but while image captioning is difficult to evaluate, our task allows for easy objective evaluation.

We also present a new large-scaledatasetfor referring expressions, based on

MS-COCO.

We have released the%B The Conference on Computer Vision and Pattern Recognition (CVPR) %C Las Vegas, Nevada %8 06/2016 %G eng %U https://github.com/ mjhucla/Google_Refexp_toolbox %0 Journal Article %J Proceedings of the IEEE %D 2016 %T A Review of Relational Machine Learning for Knowledge Graphs %A Maximilian Nickel %A Kevin Murphy %A Tresp, Volker %A Evgeniy Gabrilovich %Xdatasetand a toolbox for visualization and evaluation, see \url{https://github.com/mjhucla/Google_Refexp_toolbox}.

Relational machine learning studies methods for the statistical analysis of relational, or graph-structured, data. In this paper, we provide a review of how such statistical models can be “trained” on large knowledge graphs, and then used to predict new facts about the world (which is equivalent to predicting new edges in the graph). In particular, we discuss two fundamentally different kinds of statistical relational models, both of which can scale to massive data sets. The first is based on latent feature models such as tensor factorization and multiway neural networks. The second is based on mining observable patterns in the graph. We also show how to combine these latent and observable models to get improved modeling power at decreased computational cost. Finally, we discuss how such statistical models of graphs can be combined with text-based information extraction methods for automatically constructing knowledge graphs from the Web. To this end, we also discuss Google's knowledge vault project as an example of such combination.

%B Proceedings of the IEEE %V 104 %P 11 - 33 %8 Jan-01-2016 %G eng %U http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=7358050 %N 1 %! Proc. IEEE %R 10.1109/JPROC.2015.2483592 %0 Generic %D 2015 %T A Review of Relational Machine Learning for Knowledge Graphs: From Multi-Relational Link Prediction to Automated Knowledge Graph Construction %A Maximilian Nickel %A Kevin Murphy %A Volker Tresp %A Evgeniy Gabrilovich %Xhttp://hdl.handle.net/1721.1/100193