Researching Human-AI Collaboration through the Design of Language-Based Query Assistance

Abstract

Interactions with artificial intelligence (AI) are uniquely difficult to design because of the complexity of its output and the uncertainty of its capabilities for designers. Additionally, AI can be error-prone and needs human oversight. To try to overcome this issue, we followed a research through design (RtD) process to develop a high-fidelity interface prototype and to investigate human-AI collaboration in a realistic scenario. Specifically, we developed a support answer assistant that collaborates with customer service representatives to answer questions from customers. Our contributions highlight interaction designs that allow people to guide AI towards more satisfactory output, or to improve the utility of unsatisfactory AI output by making it possible to review and edit AI output efficiently. Early studies showed that participants used these designs to leverage the AI to provide a first answer draft or to automatically complete what they wanted to write. We describe our design and evaluation process and discuss how the insights of this research can help improve human-AI collaboration. Finally, we discuss how explainable artificial intelligence (XAI) can help identify incorrect AI output.