| dc.description.abstract | Artificial Intelligence (AI) is being increasingly used to assist complex decision-making such as financial investing. As most AI systems rely on black-box machine learning models, understanding how to support human decision-makers and gaining users' trust becomes important. Explainable Artificial Intelligence (XAI) has been proposed to address these issues by making the decision-making process of AI systems understandable to users. However, existing XAI approaches fail to take into account users' domain experience, and fail to support users with limited domain expertise. This work aims to fill this gap. We presented an approach to integrate domain expertise into XAI, and showed that this approach can have a number of benefits to users of XAI systems such as improved task performance and better assessment of XAI. The main contributions of this work include identifying the benefits of adding domain knowledge to XAI, demonstrating the usefulness of Cognitive Work Analysis (CWA) in XAI, and developing recommendations for future design of AI systems. First, through a Work Domain Analysis (WDA) approach, we identified opportunities to improve the existing XAI approaches by augmenting the explanations with domain knowledge and conducted an online study with 100 participants on users' perceptions of AI in a credit approval context. Results showed some benefits in improving user perceptions and highlighted the importance of contextual factors. Next, we introduced a testbed for exploring user behavior and task performance in a financial decision-making task. We designed decision-support aids based on domain knowledge and explored their effectiveness in an experimental study with 60 participants. In the study, participants engaged with an AI assistant and made investing decisions. Depending on the condition, participants had access to domain knowledge presented on a separate display, domain knowledge embedded in the AI assistant, or no access to domain knowledge. The results showed that participants who had access to domain knowledge relied less on AI when it was incorrect, and obtained better task performance. The effect of domain knowledge on perceptions of AI was limited. Next, we analyzed the user interviews that were part of the previous study. We identified users' mental models of AI and multiple ways they integrated the AI into their decision-making process. The analysis also revealed the complexity of designing for non-expert users, and we developed recommendations for future research and design. Finally, we conducted a Control Task Analysis and Strategies Analysis to synthesize the qualitative and quantitative findings and developed decision ladders and information flow maps. The analyses provided insights into the influence of AI on the decision-making process, challenges associated with non-expert users, and opportunities to improve AI user interface design. | en |
