ML Anthology

Multi-Modal References in Human-Computer Dialogue

Jeannette G. Neal, Zuzana Krifka Dobes, Keith E. Bettinger, Jong S. Byoun
AAAI 1988 pp. 819-825
/aaai/1988/neal1988aaai-multi/

Abstract

Multi-modal communication is common among humans. People frequently supplement natural language (NL) communication with simultaneous coordinated pointing gestures and drawing on ancillary visual aids. Similar multi-modal communication can facilitate human interaction with modern sophisticated information processing and decision-aiding computer systems. In this paper, we focus on the use of deictic pointing gestures with simultaneous coordinated NL in both user input and system-generated output. Key knowledge sources and methodology for referent resolution are presented. The synergistic mutual disambiguation of simultaneous NL and pointing is discussed as well as a methodology for handling inconsistent NL/pointing expressions and expressions that have an apparent null referent. This work is part of the Intelligent Multi-Media Interface Project [Neal & Shapiro, 1988] which is devoted to the development of intelligent interface technology that integrates speech, NL text, graphics, and pointing gestures for human-computer dialogues.

PDF AAAI Semantic Scholar

Cite

Text

Neal et al. "Multi-Modal References in Human-Computer Dialogue." AAAI Conference on Artificial Intelligence, 1988.

Markdown

[Neal et al. "Multi-Modal References in Human-Computer Dialogue." AAAI Conference on Artificial Intelligence, 1988.](https://mlanthology.org/aaai/1988/neal1988aaai-multi/)

BibTeX

@inproceedings{neal1988aaai-multi,
  title     = {{Multi-Modal References in Human-Computer Dialogue}},
  author    = {Neal, Jeannette G. and Dobes, Zuzana Krifka and Bettinger, Keith E. and Byoun, Jong S.},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year      = {1988},
  pages     = {819-825},
  url       = {https://mlanthology.org/aaai/1988/neal1988aaai-multi/}
}