Towards the Learning, Perception, and Effectiveness of Teachable Conversational Agents

Abstract

The traditional process of building interactive machine learning systems can be viewed as a teacher-learner interaction scenario where the machine-learners are trained by one or more human-teachers. In this work, we explore if teachable AI agents can reliably learn from human-teachers through conversational interactions, how this teaching process affects a teacher's performance in the task, and their trust on the agent. We introduce a teachable agent named Kai, that learns to classify news articles while also guiding the teaching process through conversational interventions. In a three-part study, where several crowdworkers individually teach Kai, we investigate whether this Learning by Teaching approach creates reliable machine learners, improves Turkers' performance and leads to trustable AI agents that crowdworkers would use. We present and discuss the results of the underlying classifier built from conversational interactions with other text classification algorithms. We also provide an evaluation of how crowdworkers perform a text classification before and after interacting with a teachable agent. Finally, we investigate the notion of trust that crowdworkers exhibit for their teachable agents in terms of delegating the work involving monetary compensation. Together, our results demonstrate the benefits of Learning by Teaching approach, in terms of the performance of the AI agent, the crowdworkers, and the dynamics of trust built from the teacher-learner interaction.