|Title||Grounding language acquisition by training semantic parsersusing captioned videos|
|Publication Type||Conference Paper|
|Year of Publication||2018|
|Authors||Ross, C, Barbu, A, Berzak, Y, Myanganbayar, B, Katz, B|
|Conference Name||Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP 2018),|
|Conference Location||Brussels, Belgium|
We develop a semantic parser that is trained ina grounded setting using pairs of videos cap-tioned with sentences. This setting is bothdata-efficient, requiring little annotation, andsimilar to the experience of children wherethey observe their environment and listen tospeakers. The semantic parser recovers themeaning of English sentences despite not hav-ing access to any annotated sentences. It doesso despite the ambiguity inherent in visionwhere a sentence may refer to any combina-tion of objects, object properties, relations oractions taken by any agent in a video. For thistask, we collected a new dataset for groundedlanguage acquisition. Learning a grounded se-mantic parser — turning sentences into logi-cal forms using captioned videos — can sig-nificantly expand the range of data that parserscan be trained on, lower the effort of training asemantic parser, and ultimately lead to a betterunderstanding of child language acquisition.
- CBMM Funded