Leveraging Interpretable Counterfactuals to Inform the Design of AI-Enabled Technologies for Human-Centered Automation in Healthcare

Abstract

Hypertension is a prevalent cardiovascular condition characterized by blood pressure exceeding 130/80 mmHg, which often remains asymptomatic and undiagnosed. In Canada, 1 in 5 people with hypertension are unaware that they have the condition. The increasing integration of artificial intelligence (AI) into digital health technologies (DHTs), such as wearables, offers opportunities to support hypertension screening and monitoring; however, high levels of automation in DHTs may provide users with limited information about the conditions that influence the measurements or limitations of underlying algorithms. This is particularly important in systems that decide when to involve the user in the decision-making and action selection process, where limited visibility into the system’s reasoning may affect how users interpret an unsuccessful measurement or the absence of a notification. This thesis investigates how explainable AI (XAI), model counterfactual explanations, large language models (LLMs), and behavior-informed interface design can be integrated to support more transparent and actionable communication between highly automated health systems and their users. The research first examined the use of explainable AI to evaluate binary photoplethysmography (PPG) signal quality classifiers beyond predictive performance alone. Statistical analysis identified the acquisition and contextual conditions associated with poor signal quality, and multiple binary classification approaches were evaluated using both performance and interpretability criteria. SHAP and LIME were used to examine the features associated with individual model classifications, while DiCE was used to generate model counterfactuals presenting feasible input changes capable of shifting predicted signal quality. This approach supported the identification of models and features that were not only predictive but also more suitable for developing actionable, user-facing guidance. The resulting model counterfactuals were subsequently incorporated into an LLM-assisted prompt-generation pipeline informed by the Fogg Behavior Model. Technical model outputs were translated into short, model-informed facilitator prompts that linked potentially modifiable measurement conditions associated with bad-quality predictions to practical corrective actions. A single reviewer then refined the generated messages, removing unsupported interpretations, improving alignment between model-identified conditions and recommended actions, and ensuring that the final prompts were appropriate for presentation on a wearable interface under the relevant movement conditions. To evaluate the transferability of this corrective feedback approach to an interactive wearable measurement context, the resulting prompts were implemented in a Figma prototype of the wearable interface using unsuccessful smartwatch electrocardiogram (ECG) measurement scenarios. Twenty-eight participants interacted with prototype smartwatch interfaces and evaluated the clarity, usefulness, confidence, information needs, and intended actions associated with the proposed and existing interfaces. Participants generally valued the proposed interface’s ability to explain a measurement condition likely contributing to unsuccessful measurements, provide corrective action, and communicate clear next steps. Reported retry intention varied across scenarios, and the proposed interface did not produce a statistically significant overall increase in reported retry intention relative to the existing interface. These findings indicate that the value of explanatory prompts extends beyond encouraging immediate repetition of a measurement and includes supporting users’ understanding, confidence, and context-sensitive decision-making about whether to retry, wait, adjust the device, or seek additional information. Based on the technical and human-centered findings, this thesis proposes ACT-AI, a design framework for integrating explainability, actionable counterfactual guidance, LLM-assisted interpretation, and behavior-informed interface design into highly automated digital health technologies. The framework advances the design of AI-enabled systems by contributing an approach for making system limitations and model-identified factors more accessible to users while supporting calibrated human involvement when corrective action is possible.