Browsing by Author "Dautenhahn, Kerstin"

Now showing 1 - 17 of 17

Accessible Integration of Physiological Adaptation in Human-Robot Interaction
(University of Waterloo, 2021-09-21) Kothig, Austin; Dautenhahn, Kerstin
Technological advancements in creating and commercializing novel unobtrusive wearable physiological sensors have generated new opportunities to develop adaptive human-robot interaction (HRI). Detecting complex human states such as engagement and stress when interacting with social agents could bring numerous advantages to creating meaningful interactive experiences. Bodily signals have classically been used for post-interaction analysis in HRI. Despite this, real-time measurements of autonomic responses have been used in other research domains to develop physiologically adaptive systems with great success; increasing user-experience, task performance, and reducing cognitive workload. This thesis presents the HRI Physio Lib, a conceptual framework, and open-source software library to facilitate the development of physiologically adaptive HRI scenarios. Both the framework and architecture of the library are described in-depth, along with descriptions of additional software tools that were developed to make the inclusion of physiological signals easier for robotics frameworks. The framework is structured around four main components for designing physiologically adaptive experimental scenarios: signal acquisition, processing and analysis; social robot and communication; and scenario and adaptation. Open-source software tools have been developed to assist in the individual creation of each described component. To showcase our framework and test the software library, we developed, as a proof-of-concept, a simple scenario revolving around a physiologically aware exercise coach, that modulates the speed and intensity of the activity to promote an effective cardiorespiratory exercise. We employed the socially assistive QT robot for our exercise scenario, as it provides a comprehensive ROS interface, making prototyping of behavioral responses fast and simple. Our exercise routine was designed following guidelines by the American College of Sports Medicine. We describe our physiologically adaptive algorithm and propose an alternative second one with stochastic elements. Finally, a discussion about other HRI domains where the addition of a physiologically adaptive mechanism could result in novel advances in interaction quality is provided as future extensions for this work. From the literature, we identified improving engagement, providing deeper social connections, health care scenarios, and also applications for self-driving vehicles as promising avenues for future research where a physiologically adaptive social robot could improve user experience.
Can Social Robots Improve People’s Attitudes towards Individuals who Stutter?
(ACM, 2025) Körner, Jule; Chandra, Shruti; Dautenhahn, Kerstin
Public attitudes towards stuttering are rooted in stereotypes and misconceptions, leading to negative reactions and discrimination against individuals who stutter. Previous research highlights the positive impact of educational interventions on people’s attitudes towards stuttering. The potential of social robots as an educational tool in the context of stuttering awareness remains unexplored. In the present study, we investigate whether a social robot can improve public attitudes when giving an interactive presentation on the topic. We compare its impact with a tablet-only condition. Additionally, we differentiate between two robot conditions - one in which the robot imitates stuttering and another where the robot has fluent speech. In the robot conditions, visuals are shown on a tablet.We used a co-design approach and incorporated the perspectives and experiences of two individuals with lived experiences of stuttering into our study design. A user study with 69 participants reveals significant improvements in attitudes across all three conditions, with no significant difference between conditions. However, participants perceived the robot as significantly ‘warmer’, more ‘attractive’ and ‘novel’ when compared to the tablet. These findings provide valuable insights into the potential of social robots as intervention techniques for improving attitudes in the field of stuttering.
Developing a Semi-autonomous Robot to Engage Children with Special Needs and Their Peers in Robot-Assisted Play
(University of Waterloo, 2021-05-31) Mahdi, Hamza; Dautenhahn, Kerstin
Despite the wide variety of robots used in human-robot interaction (HRI) scenarios, the potential of robots as connectors whilst acting as play mediators has not been fully explored. Robots present an opportunity to redefine traditional game scenarios by being physical embodiments of agents/game elements. Robot assisted play has been used to reduce the barriers that children with physical special needs experience. However, many projects focus on child-robot interaction rather than child-child interaction. In an attempt to address this gap, a semi-autonomous mobile robot, MyJay, was created. This thesis discusses the successful development of MyJay and its potential contribution in future HRI studies. MyJay is an open-source robot that plays a basketball-like game. It features light and color for communicative feedback, omni-directional mobility, robust mechanisms, adjustable levels of autonomy for dynamic interaction, and a child-friendly aesthetically-pleasing outer shell. The design process included target users such as children with special needs and therapists in order to create a robot that ensures repeated use, engagement, and long-term interaction. A hybrid approach was taken to involve stakeholders, combining user-centered design and co-design, exemplifying that children can be included in the creation process even when it is not possible to hold in-person co-design sessions due to COVID-19. Aside from the care taken to meet user requirements, the robot was designed with researchers in mind, featuring extensible software and ROS compatibility. The frame is constructed from aluminum to ensure rigidity, and most functional parts related to gameplay are 3D printed to allow for quick swapping, should a need to change game mechanics arise. The modularity in software and in mechanical aspects should increase the potential of MyJay as a valuable research tool for future HRI studies. Finally, a novel framework to simulate teleoperation difficulties for individuals with upper-limb mobility challenges is proposed, along with a dynamic assistance algorithm to aid in the teleoperation process.
Development of a Social Robot as a Mediator for Intergenerational Gameplay & Development of a Canvas for the Conceptualisation of HRI Game Design
(University of Waterloo, 2020-12-21) K Aravamuthan, Aishwarya; Dautenhahn, Kerstin; Boger, Jennifer
Intergenerational interaction between grandparents and grandchildren benefits both generations. The use of a social robot in mediating this interaction is a relatively unexplored area of research. Often Human-Robot Interaction (HRI) research uses the robot as a point of focus; this thesis puts the focus on the interaction between the generations, using a multi-stage study with a robot mediating the interaction in dyads of grandparents and grandchildren. The research questions guiding this thesis are: 1) How might a robot-mediated game be used to foster intergenerational gameplay? 2) What template can be created to conceptually describe HRI game systems? To answer the first question, the study design includes three stages: 1. Human mediator Stage (exploratory); 2. The Wizard-of-Oz (WoZ) Stage (where a researcher remotely controls the robot); 3. Fully/semi-autonomous Stage. A Tangram puzzle game was used to create an enjoyable, collaborative experience. Stage 1 of the study was conducted with four dyads of grandparents (52-74 years of age) and their grandchildren (7-9 years of age). The purpose of Stage 1 was to determine the following: 1. How do dyads of grandparent-grandchild perceive their collaboration in the Tangram game? 2. What role do the dyads envision for a social robot in the game? Results showed the dyads perceived high collaboration in the Tangram game, and saw the role of the robot as helping them by providing clues in the gameplay. The research team felt the game, in conjunction with the proposed setup, worked well for supporting collaboration and decided to use the same game with a similar setup for the next two stages. Although the design and development of the next stage were ready, the COVID-19 pandemic led to the suspension of in-person research. The second part of this thesis research focused on creating the Human-Robot Interaction Game Canvas (HRIGC), a novel way to conceptually model HRI game systems. A literature search of systematic ways to capture information, to assist in the design of the multi-stage study, yielded no appropriate tool, and prompted the creation of the HRIGC. The goal of the HRIGC is to help researchers think about, identify, and explore various aspects of designing an HRI game-based system. During the development process, the HRIGC was put through three case studies and two test runs: 1) Test run 1 with three researchers in HRI game design; 2) Test run 2 with four Human-Computer Interaction (HCI) researchers of different backgrounds. The case studies and test runs showed HRIGC to be a promising tool in articulating the key aspects of HRI game design in an intuitive manner. Formal validation of the canvas is necessary to confirm this tool.
Development of a Social Robot for the Study of Robot Roles in Robot-Assisted Language Learning
(University of Waterloo, 2024-04-25) Yang, Ya-Ting; Dautenhahn, Kerstin; Hu, Yue
The objective of this thesis encompasses both the development of a novel social robot, named ”Maple,” and the examination of its effectiveness in various roles within the context of education and Human-Robot Interaction (HRI). A study was conducted to compare the efficacy of two common roles that social robots can assume in HRI research concerning learning: as a tutor and as a peer-learner. Individual participants, including university students and staff, engage in a vocabulary learning task in Esperanto with Maple assuming either a tutor or a peer-learner role. The research findings indicate participants’ positive attitudes and perceptions towards both roles enacted by Maple. Usability assessments reveal high scores across both conditions, with participants reporting predominantly positive user experiences. While no significant difference in learning outcomes is observed between the two robot roles, participants exhibit a 30% increase in quiz completion speed when interacting with Maple in the peer-learner role, suggesting the peer-learner robot promotes and fosters meaningful learning. Thematic analysis of participants’ responses underscores their preferences for increased interaction with Maple as a peer-learner, acknowledgment of Maple’s capacity for companionship and interactivity, and envisioning practical applications of such learning systems in real-life settings. The insights obtained from this study not only contribute to understanding the effectiveness of social robots in educational settings but also offer valuable directions for future research and technology development in Human-Robot Interaction and Robot-Assisted Language Learning.
Engineering Social Learning Mechanisms for Multi-Agent Interaction
(University of Waterloo, 2021-09-14) Mateen Abdul Hamid, Owais; Dautenhahn, Kerstin; Nehaniv, Chrystopher
This thesis is strongly inspired by literature on animal social learning, applying it to multi-robot as well as human-robot interaction scenarios, Social learning, which can include complex or simple social mechanisms, allow us to understand cooperation and communication in animals, giving them better chances to survive for longer and thrive as a society. For this dissertation, to translate this understanding into socially rich behavior among multi-agent robots and Human-Robot Interaction, two experiments were conducted. The first experiment focused on how social learning might optimize cooperation among robots (in a robot 'society') for the problem of foraging. The task utilizes small and simple swarm robots to understand how such social mechanisms might play a role in establishing rules for emergent group behavior and how social rules might be engineered to gain useful effects in a group of robots. The study investigated exploratory behavior without interaction (asocial) and with interaction (social). The results from this exploratory study suggest that deterministic asocial exploration is best performed by a Spiral exploration mechanisms. However, these asocial exploration strategies are eclipsed by certain types of social reward sharing strategies as long as sharing occurs for at least half the lifetime of the robots. Sharing locations of reward caches for all time is of course the most optimal, but comes at the cost of communicating longer and hence using more energy both on the sender and receiver’s end. An analysis of a compromise strategy between completely asocial exploration and social reward location sharing is performed using strategies termed critical and conditional learning. It is found that the number of reward caches located through critical and conditional learning are intermediary to the two extremes, namely completely asocial and completely social foraging. The second experiment sought to understand if and how other types of social learning mechanisms such as observational conditioning can facilitate social information spread to human participants. The question of whether, and to what extent, a robot can influence a human's actions is asked through a study designed to understand if emotions displayed by a robot demonstrators can influence human observers. An immersive first-person gaming experience utilizing Unity was designed where a robot demonstrator reacted either positively or negatively to an external stimulus. Objective (position of player in-game) and subjective (Questionnaire) data collected on the human participants' reactions suggests that the virtual robot agent is successful in socially transmitting information. Through these studies, I seek to contribute to the understanding of the role simple social learning mechanisms can play in information transfer among human and robot agents, and to identify useful metrics for the detection of such social mechanisms.
Exploring Human Teachers' Interpretations of Trainee Robots' Nonverbal Behaviour and Errors
(University of Waterloo, 2021-04-22) Aliasghari, Pourya; Dautenhahn, Kerstin; Nehaniv, Chrystopher L.; Ghafurian, Moojan
In the near future, socially intelligent robots that can learn new tasks from humans may become widely available and gain an opportunity to help people more and more. In order to successfully play a role, not only should intelligent robots be able to interact effectively with humans while they are being taught, but also humans should have the assurance to trust these robots after teaching them how to perform tasks. When human students learn, they usually provide nonverbal cues to display their understanding of and interest in the material. For example, they sometimes nod, make eye contact or show meaningful facial expressions. Likewise, a humanoid robot's nonverbal social cues may enhance the learning process, in case the provided cues are legible for human teachers. To inform designing such nonverbal interaction techniques for intelligent robots, our first work investigates humans' interpretations of nonverbal cues provided by a trainee robot. Through an online experiment (with 167 participants), we examine how different gaze patterns and arm movements with various speeds and different kinds of pauses, displayed by a student robot when practising a physical task, impact teachers' understandings of the robot’s attributes. We show that a robot can appear differently in terms of its confidence, proficiency, eagerness to learn, etc., by systematically adjusting those nonverbal factors. Human students sometimes make mistakes while practising a task, but teachers may be forgiving about them. Intelligent robots are machines, and therefore, they may behave erroneously in certain situations. Our second study examines if human teachers for a robot overlook its small mistakes made when practising a recently taught task, in case the robot has already shown significant improvements. By means of an online rating experiment (with 173 participants), we first determine how severe a robot’s errors in a household task (i.e., preparing food) are perceived. We then use that information to design and conduct another experiment (with 139 participants) in which participants are given the experience of teaching trainee robots. According to our results, perceptions of teachers improve as the robots get better in performing the task. We also show that while bigger errors have a greater negative impact on human teachers' trust compared with the smaller ones, even a small error can significantly destroy trust in a trainee robot. This effect is also correlated with the personality traits of participants. The present work contributes by extending HRI knowledge concerning human teachers’ understandings of robots, in a specific teaching scenario when teachers are observing behaviours that have the primary goal of accomplishing a physical task.
Exploring the Use of Assistive Robotics in Play and Education for Children with Disabilities
(University of Waterloo, 2022-12-19) Azizi, Negin; Dautenhahn, Kerstin
Assistive technologies in general, and assistive robots in particular, are being studied extensively to maintain and increase the capabilities of individuals with disabilities. However, there are aspects in this field that have not been explored yet. This thesis investigates the use of assistive robots for different groups of children with disabilities, such as learning disabilities, and upper-limb disorders, where the use of robots as tools have not been widely explored. We began by exploring learning disabilities and their challenges. Students with a learning disability (LD) generally require supplementary one-to-one instruction and support to acquire the foundational academic skills learned at school. Because learning is more difficult for students with LD, students can frequently display off-task behaviours to avoid attempting or completing challenging learning tasks. Re-directing students back to their learning task is a frequent strategy used by educators to support students. However, there have been limited studies investigating the use of assistive technology to support student re-direction, specifically in a "real-world" educational setting. We investigated the impact of integrating a socially assistive robot to provide re-direction strategies to students. A commercially available social robot, QT, was employed within the existing learning program during one-to-one remedial instruction sessions. First, we conducted a pilot study to explore the impact of the robot on students' on-task behaviours and progress towards learning goals. The results of our mixed method analysis suggest that the robotic intervention supported students in staying on-task and completing their learning goal. Learning from the lessons of the pilot study, we designed a between-participant study with two conditions, control, and intervention with the QT robot to address the shortcomings of the pilot study. In the main study we aimed a) to evaluate the acceptance of the social robot by the users, i.e., instructors and students in a real-world educational setting; and b) understand the impact of the robot’s intervention on student's engagement during learning tasks over multiple learning sessions. Our qualitative analysis suggests that instructors and students showed positive attitudes towards the social robot in their one-to-one sessions. In addition, the students were more engaged with their task in the presence of the robot, and displayed fewer off-task behaviours in the intervention condition, compared to the control condition. These results suggest that a social robot can be used as an effective educational tool for instructors in boosting engagement and mitigating off-task behaviours for students with learning disabilities. Assistive technology can also be beneficial in play, especially for children that face barriers in physical activities due to their physical impairments. In the third study, we focused on children with upper-limb disorders and the lack of equipment and enjoyable experiences in games. While game-play is widely used in human robot interaction studies, using a robot as a play-mediator, where two individuals interact with each other through a robot, has not been fully studied yet. However, understanding the play dynamics of this type of game is an important step towards designing an engaging experience. In this work, participants played two collaborative games which involved teleoperating a mobile robot. Each game consisted in achieving the same task, but involved two different collaboration strategies: one where the players shared tasks and one where joint action was necessary. In this study, we focused on how both players collaborated with each other in terms of coordination and communication using video and joystick data. Due to Covid-19 restrictions, we were not able to recruit children with physical disabilities. Instead, we recruited university students to participate in the study to collect data. Results indicated different behavioural events, and observed different levels of communication among the two conditions. The present work contributes to robotic assistive technologies by providing support for children with learning disabilities and upper-limb disorders in different aspects of their life, such as education and play.
Exploring the Use of Social Robots in Promoting University Students’ Mental Well-Being
(University of Waterloo, 2023-12-12) Rasouli, Samira; Dautenhahn, Kerstin; Ghafurian, Moojan
Mental health problems within university student populations are a growing concern. In recent years, university students have reported increased symptoms of stress and anxiety, which can negatively impact their mental well-being and academic performance. However, many students do not seek or receive support for these challenges. With recent technological advances in social robots and their expanding capabilities, there is potential for social robots to extend and complement mental health care interventions and enhance access to care. This thesis introduces a novel application of social robots in robot-assisted interventions for addressing stress and anxiety in social situations. As an initial step to developing effective technology-based interventions, it is essential to identify design elements and functionalities that are perceived as engaging and useful by students. Therefore, our first study explores students’ perceptions of, and preferences for, using different types of intelligent agents (e.g., virtual agents, social robots, etc.) to support their mental well-being, specifically to cope with feelings of stress and anxiety in social situations typically encountered within a university context (e.g., engaging in a group discussion, delivering presentations, expressing opinions, etc.). Two online surveys were conducted: a pilot study on Amazon Mechanical Turk (N = 85) and a larger study at the University of Waterloo (N = 1054). The results provide insights into different design elements (e.g., confidentiality, accessibility and enjoyment) as well as social and technical capabilities (e.g., ability to understand conversation) to consider when designing intelligent agents to help address stress and anxiety among university students. The results also showed that public speaking is one of the most anxiety-provoking social situations at university, and animal-like robots were the most selected agents among the given choices of different types of intelligent agents to use for managing anxiety related to the students' most anxiety-provoking social situations. Participants also endorsed a variety of activities that they thought could be beneficial for managing anxiety related to their most anxiety-provoking situation, including receiving personalized coaching for activities and tasks that support mental well-being (e.g., meditation and yoga), getting help with thinking calming or positive thoughts, and getting help with building confidence, for which they indicated to be willing to use an intelligent agent. In the next step, we used the findings of our first study to design a robotic mental well-being coach aimed at assisting university students in managing public speaking anxiety. To this end, in our second study, we conducted collaborative co-design sessions with five mental health professionals (the envisioned secondary users) to identify the design-related needs, including robot behaviours and interactions. We employed a co-design approach to ensure that the design of the robot's behaviours and functionalities is grounded in evidence-based practices and aligns with the best practices of psychological support in this application area that can be delivered by a robotic coach. Furthermore, we carried out a participatory study involving six university students (the envisioned primary users) to gather their opinions for further improvements to the robotic coach. Students provided feedback on the behaviours and functionality of the robot and generally found the robot engaging, relaxing, knowledgeable, and beneficial for learning relaxation exercises. Lastly, after implementing final improvements to the behaviour and functionalities of the robotic coach based on feedback from students and with the assistance of mental health professionals, we conducted our third study with university students (N=50) to evaluate the usability and acceptability of the robotic mental well-being coach designed in the second study to help university students manage their public speaking anxiety. The findings demonstrated a usability score of 84.05 for the robotic coach and high acceptability among university students who perceived the robot as competent and knowledgeable. In addition, a significant improvement in participants' self-reported moods was observed following the study. The qualitative analysis suggests that university students had a positive impression of the robotic mental well-being coach and their interaction. Overall, the findings of this study are encouraging and indicate the potential of social robots as mental well-being coaches to support university students' public speaking anxiety. Future research should investigate the users' experiences and the effectiveness of the robotic mental well-being coach in assisting university students to cope with public speaking anxiety in the long term over multiple interaction sessions.
How Do Different Modes of Verbal Expressiveness of a Student Robot Making Errors Impact Human Teachers' Intention to Use the Robot?
(Association for Computing Machinery, 2021-11-09) Aliasghari, Pourya; Ghafurian, Moojan; Nehaniv, Chrystopher L.; Dautenhahn, Kerstin
When humans make a mistake, they often try to employ some strategies to manage the situation and possibly mitigate the negative effects of the mistake. Robots that operate in the real world will also make errors and therefore might benefit from such recovery strategies. In this work, we studied how different verbal expression strategies of a trainee humanoid robot when committing an error after learning a task influence participants’ intention to use it. We performed a virtual experiment in which the expression modes of the robot were as follows: (1) being silent; (2) verbal expression but ignoring any errors; or (3) verbal expression while mentioning any error by apologizing, as well as acknowledging and justifying the error. To simulate teaching, participants remotely demonstrated their preferences to the robot in a series of food preparation tasks; however, at the very end of the teaching session, the robot made an error (in two of the three experimental conditions). Based on data collected from 176 participants, we observed that, compared to the mode where the robot remained silent, both modes where the robot utilized verbal expression could significantly enhance participants' intention to use the robot in the future if it made an error in the last practice round. When no error occurred at the end of the practice rounds, a silent robot was preferred and increased participants' intention to use.
Human-aware Autonomous Vehicle Navigation in Pedestrian-rich Unstructured Environments
(University of Waterloo, 2024-12-11) Golchoubian, Mahsa; Lashgarian Azad, Nasser; Dautenhahn, Kerstin
Autonomous Vehicles (AVs) have the potential to enhance transportation safety, improve efficiency, and elevate quality of life. Despite significant advancements in AV technology, operating these vehicles in dynamic, crowded environments that requires frequent interaction with other decision-making agents remains challenging. A key example is the interaction between AVs and pedestrians. While most research has focused on these interactions in structured road settings, the complexity and diversity of AV navigation among pedestrians in unstructured environments (e.g., shared spaces, airport terminals) have been less explored. In such pedestrian-rich environments, AVs must be human-aware, meeting people's expectations while ensuring both their safety and comfort. At the same time, navigating these spaces requires reasoning about mutual interactions and accounting for the uncertainty in pedestrian behaviour. This thesis introduces a novel approach to address these challenges, presenting an integrated prediction and planning framework for AV navigation among pedestrians in unstructured shared environments. The thesis is structured into two main phases: a design requirement study and an algorithmic development phase. Given the novelty of this application, the first phase focused on understanding the perceptions and preferences of pedestrians regarding AV behaviour in common interactive scenarios within unstructured settings. Additionally, we examined the unique aspects of pedestrian behaviour in these environments, identifying common behaviours AVs must manage and gathering existing datasets that better represent pedestrian behaviour in such settings. This study highlights the importance of considering uncertainty in pedestrian behaviour, shaping the direction of the development phase. In the algorithmic development phase, we propose a novel proactive, uncertainty-aware Deep Reinforcement Learning (DRL) decision-making algorithm. This algorithm efficiently accounts for complex interaction effects with multiple pedestrians while maintaining reasonable computational time. The navigation algorithm is made proactive and farsighted by integrating the DRL motion planner with a data-driven pedestrian trajectory predictor. Our novel prediction model is designed to forecast pedestrian trajectories in highly interactive shared environments. It uses a collision risk metric to identify key interacting agents and encodes their effects through a newly engineered interaction feature which guide the learning process. During training, we prevented overconfident predictions and improved estimates of prediction uncertainty using an augmented loss function that incorporates uncertainty awareness. Unlike other DRL algorithms in this area, our model's DRL motion planner accounts for prediction uncertainty, integrating it into the reward function to encourage the AV to minimize collision probability with pedestrians over a prediction horizon. Additionally, the reward function design encourages socially aware behaviours, such as reducing speed during close encounters, respecting pedestrians' personal space, and adhering to social norms identified in our earlier design requirement study. We trained our model in a simulation environment that contains realistic pedestrian trajectory behaviour in the presence of vehicles in shared spaces. The simulation results demonstrate that our uncertainty-aware DRL navigation framework outperforms state-of-the-art DRL crowd navigation and uncertainty-aware Model Predictive Control (MPC) models, both in terms of efficiency and social behaviour aspects. Overall, this thesis contributes to the advancement of socially-aware crowd navigation algorithms beyond human-sized mobile robots to autonomous vehicles operating as mobility aids among pedestrians in unstructured environments. It demonstrates how agent interactions can be effectively modelled within prediction and planning modules, and how uncertainty in these predictions can be integrated into a DRL-based motion planner.
Integrating Affective Expressions into Robot-Assisted Search and Rescue to Improve Human-Robot Communication
(University of Waterloo, 2021-09-13) Akgun, Sami Alperen; Dautenhahn, Kerstin; Ghafurian, Moojan; Crowley, Mark
Unexplained or ambiguous behaviours of rescue robots can lead to inefficient collaborations between humans and robots in robot-assisted SAR teams. To date, rescue robots do not have the ability to interact with humans on a social level, which is believed to be an essential ability that can improve the quality of interactions. This thesis research proposes to bring affective robot expressions into the SAR context to provide rescue robots social capabilities. The first experiment presented in Chapter 3 investigates whether there is consensus in mapping emotions to messages/situations in Urban Search and Rescue (USAR) scenarios, where efficiency and effectiveness of interactions are crucial to success. We studied mappings between 10 specific messages, presented in two different communication styles, reflecting common situations that might happen during search and rescue missions and the emotions exhibited by robots in those situations. The data was obtained through a Mechanical Turk study with 78 participants. The findings support the feasibility of using emotions as an additional communication channel to improve multi-modal human-robot interaction for urban search and rescue robots and suggest that these mappings are robust, i.e., are not affected by the robot’s communication style. The second experiment was conducted on Amazon Mechanical Turk as well with 223 participants. We used Affect Control Theory (ACT) as a method for deriving the mappings between situations and emotions (similar to the ones in the first experiment) and as an alternative method to obtaining mappings that can be adjusted for different emotion sets (Chapter 4). The results suggested that there is consistency in the choice of emotions for a robot to show in different situations between the two methods used in the first and second experiment, indicating the feasibility of using emotions as an additional modality in SAR robots. After validating the feasibility of bringing emotions to SAR context based on the findings from the first two experiments, we created affective expressions based on Evaluation, Potency and Activity (EPA) dimensions of ACT with the help of LED lights on a rescue robot called Husky. We evaluated the effect of emotions on rescue workers’ situational awareness through an online Amazon Mechanical Turk Study with 151 participants (Chapter 5). Findings indicated that participants who saw Husky with affective expressions (conveyed through lights) had better perception accuracy of the situation happening in the disaster scene than participants who saw the videos of the Husky robot without any affective lights. In other words, Husky with affective lights improved participants’ situational awareness.
Investigating Dual Embodiment in Recurring Tasks with a New Social Robot: Designing the Mirrly Platform
(University of Waterloo, 2025-03-19) Yamini, Ali; Dautenhahn, Kerstin
In many contexts, including education, therapy, and everyday tasks, assistive robots have demonstrated considerable promise for augmenting human capabilities and providing supportive interactions. By designing and building a new tabletop social robot, Mirrly, as well as empirically examining how different robotic embodiments affect user engagement and task compliance, this thesis tries to contribute to this field. In light of advances in human-robot interaction (HRI) and child-robot interaction (CRI), I investigated a comprehensive set of mechanical, electronic, and software requirements. As a result of these requirements, Mirrly was developed, a low-cost, compact platform that could be deployed in schools, therapy centers, or personal homes and it is anthropomorphic enough for supporting social interactions with people. Following the design and implementation of Mirrly, I conducted a multi-session experiment to determine whether physical embodiment, virtual embodiment (mobile-based), or dual embodiment (both physical and virtual) promoted compliance with repetitive daily tasks, as relevant e.g. in clinical applications where patients need to comply with repetitive treatments. According to the results, physical presence is a strong motivator, leading to higher compliance and engagement, whereas dual embodiment enhanced participants' enjoyment (pleasure) of the interaction specifically. Interestingly, individual differences in the participant sample, such as personality traits and self-control, did not have a significant impact on adherence or user satisfaction. As at least within the short, relatively simple user tasks, these results emphasize the importance of design factors namely physical tangibility and interactive behaviors. As part of the thesis, a review of relevant HRI and CRI literature is conducted to contextualize Mirrly's design within the context of current robotics. Following a detailed description of utilized methodology, I present the experimental conditions, measures, and analytical methods for assessing compliance, engagement, and perceived enjoyment. Finally, I discuss the implications of the findings for building more adaptive, child-centered robots, especially in clinical, therapeutic and educational settings. Several future directions are also proposed, including extending task complexity, integrating advanced sensors for personalized feedback, and conducting longitudinal studies. As part of ongoing efforts in social and assistive robotics, this work introduces a novel robotic design. Moreover, in my study, I demonstrate that a robot with careful engineering, physical embodiment, and adaptability can significantly boost compliance. Consequently, this thesis lays a good foundation for future developments in CRI, highlighting how embodiment, anthropomorphism, and structured experimental design converge to support recurrent task compliance efficiently.
A Learning Social Referencing Disambiguation Framework for Domestic Service Robots
(University of Waterloo, 2023-08-24) Fan, Kaiwen; Dautenhahn, Kerstin; Nehaniv, Chrystopher
The successful integration of domestic service robots into home environments can bring significant services and convenience to the general population and possibly mitigate important societal issues, such as care provision for older adults. However, home environments are complex, dynamic, and object-rich. It is, therefore, very probable that service robots will encounter ambiguity while interacting with household items. Moreover, service robots need to have the capability to obtain knowledge and continuously learn from the environment to ensure high adaptability and persistent functional adequacy. This thesis presents a learning object disambiguation framework for domestic service robots that is inspired by the cognitive mechanism of social referencing and the human visual mental imagery perceptual experience. The framework allows the service robot to resolve various ambiguities in the object selection task and learn objects through bidirectional human robot interactions. The framework's technical details are explained in depth. We first describe the base framework, which consists of five functional components: the user command interface, fuzzy ambiguity determination, fuzzy human attention assessment, social referencing disambiguation, and short-term long-term memory object learning. We then explain our extended framework with the expansion of robot mental imagination and user objection absence detection to further enhance the robot's ability to handle novel objects and improve interaction robustness. To showcase and validate our robot framework, we developed a system validation study with human participants. The framework is implemented on our mobile robot manipulator, Fetch, for our testing scenarios. Our experiment is designed to measure the success of the framework objectively and to understand the human perception of the robot with the framework. The study design, implementation, results, and discussions are illustrated. Finally, we discuss the current limitations of the framework and summarize many valuable lessons learned in this research project. We conclude the thesis with an exploration of many promising and exciting future directions for the proposed framework. We believe this framework forms an important conceptual foundation for future service robots to become "lifelong learners" with human guidance.
Preference and Performance-Based Adaptive Task Planning in Human-Robot Collaboration
(University of Waterloo, 2024-04-23) Noormohammadi-Asl, Ali; Dautenhahn, Kerstin; Smith, Stephen L.
This thesis delves into a central challenge in human-robot collaboration (HRC): the adaptive task planning of robots to enhance team performance, fluency, and the human agent's perception of both the robot and the collaboration. This thesis tackles the challenge of proactive task planning and allocation in collaborative scenarios, involving a single human and a single robot working together to accomplish a task. Recognizing the existing gaps in the literature, our focus revolves around balancing human agents' leading/following preferences and their performance, with the aim of fostering collaboration while maintaining a high level of human perception of the robot. After an in-depth review of related work, we initiate our exploration with an online user study, in a simulation environment using a manipulator robot. This study is designed to evaluate the impact of the robot's planning strategy on participants' perception of the robot and collaboration. This study incorporates three distinct planning strategies: prioritizing the human's objectives, prioritizing the robot's objectives, and achieving a balance between both agents' objectives. The results guide our assessment of how the balancing strategy, in particular, can uphold both team performance and a high level of participants' perception of collaboration and the robot, in comparison to the other strategies. However, a limitation arises as the study employs fixed strategies, randomly assigned to participants, irrespective of their preference and performance. Building upon the results of the first user study, we address the limitations identified in the initial study by enabling the robot to estimate the human agent's leading/following preference. However, the human agent's preference is not the sole factor influencing the robot's decision-making process; the human agent's performance is also crucial for adjusting the team's overall performance, particularly in cases of the human agent's poor or suboptimal performance. Consequently, the robot also estimates the human agent's performance. Furthermore, the robot needs to be capable of updating the task state based on both agents' actions and mistakes. With an updated understanding of the human agent's performance, leading/following preference, and task state, the robot updates its plan for task allocation and scheduling to minimize collaboration costs. Next, we evaluate the adaptability of the task planning framework and algorithm in a simulation environment, demonstrating its effectiveness across various human performance and preference scenarios. Yet, recognizing the unique challenges posed by human participants, the complete evaluation of the algorithm's effectiveness requires real-world scenarios, considering uncertainties inherent in human behavior and decision-making. Subsequently, we tackle the challenges of implementing the task planning framework on a real robot, a mobile manipulator robot, within a carefully designed collaborative scenario. Providing details on the experimental setup and methodology, a system evaluation study highlights the robot's ability to adapt based on human behavior. Finally, we conduct a user study involving 48 participants, evaluating results from multiple perspectives, including participants' perception of the robot, tasks, and collaboration, participants' actions and performance, and the robot's actions and performance. Results from the study affirm the success of the task planning framework in achieving its objectives: enhancing team fluency by considering the human agents' preferences and performance while maintaining a high level of participants' perception of the robot and the human-robot collaboration. This thesis also explores participants' leading/following preferences in collaboration, revealing a dominant preference to lead the robot. This finding can assist robotics and autonomous systems designers in considering this factor in their designs. Additionally, we evaluated the influence of participants' leadership and followership styles on their collaboration, warranting further and more in-depth future studies. In summary, this thesis contributes a proactive task planning framework that takes into account both human leading/following preferences and performance, signifying an advancement in the field of human-robot collaboration. The validation through user studies offers valuable insights, laying the groundwork for future research and applications in the continually evolving domain of human-robot collaboration.
System Development and Evaluation of a Social Robot as a Public Speaking Rehearsal Coach
(University of Waterloo, 2023-12-11) Forghani, Delara; Dautenhahn, Kerstin; Nehaniv, Chrystopher
Skills are developed through practice and rehearsal. Mastery and leadership often arise from dedicated effort. Among the skills that demand ongoing attention and practice, public speaking holds a crucial place. It is a skill by which individuals are assessed in various facets of life, be it in academia or as they progress in their professional careers. Nonetheless, when it comes to advancing in this skill, it is notably more time and energy-efficient to receive feedback from an individual capable of providing a reliable analysis of our performance. Ideally, engaging a human public speaking coach would be the optimal choice. However, accessing a human coach may not always be feasible due to a variety of reasons. In this master's thesis, I delved into the journey of exploring the potential of using a social robot as a coach for public speaking rehearsals, with the aim of enhancing individuals' presentation skills. My MASc project begins by conducting a comparison between two interactive technologies: a voice assistant agent and a social humanoid robot, both considered potential artificial systems for enhancing presentation skills. This comparison explores the embodiment of these technologies and underscores the significance of incorporating social non-verbal behaviours in an agent. To carry out this comparison, we organized an online study through Amazon Mechanical Turk, involving a diverse sample of participants across different age groups. Participants were asked to share their perspectives on various aspects of the agents depicted as coaches for public speaking rehearsals. They were presented with a scenario involving a video showing a student's brief presentation rehearsal in front of both agents, followed by feedback provided by these agents. Our analysis encompassed both quantitative and qualitative methods. While participants generally displayed a preference for using socially interactive agents as coaches for public speaking rehearsals, the social robot outperformed the voice assistant agent on several metrics. Notably, it received higher rankings in terms of human nature attributes, perceived likeability, and perceived warmth. Building upon the findings of the first study, and considering the more favourable evaluation of the social robot, we selected the social robot as the coaching agent for public speaking rehearsals involving university students in a single in-person presentation rehearsal session. The system development involved analyzing participants' speech quality and (simulated) audience orientation and providing verbal feedback based on the calculated performance analysis. In the second user study, we sought to gauge participants' interactions with the robot, specifically their level of acceptance and their willingness to use the system in the future. We also administered questionnaires to assess participants' perceptions of the robot's behaviour during the session and the accuracy of its feedback. The results of this study highlighted strong agreement among participants on the use of the social robot as a public speaking rehearsal coach in the university, and their expressed intention to utilize it in the future. Furthermore, participants' perceptions of the robot's competence and usefulness were closely linked to the accuracy of its feedback and the perceived quality of its listening behaviour. Finally, we asked a human public speaking coach to evaluate the robot coach, and presented the results. Future work can improve on this work and provide an accessible public speaking coach that university students can benefit from.
The Unspoken Rules of Engagement: Understanding the Impact of Social Norm Violations in Human-Robot Interactions
(University of Waterloo, 2023-08-18) Lawrence, Steven; Dautenhahn, Kerstin; Hoey, Jesse
As robots increasingly permeate diverse domains such as healthcare, education, service industries, and domestic environments, the significance of understanding and navigating human-robot interactions intensifies. This thesis explores the role of social norms in human-robot interaction (HRI), specifically investigating the effects of norm-violating behaviours by robots on trust, discomfort, competence, enjoyment, and physiological state changes in human participants. To begin this research, we conducted a systematic review to gain a comprehensive understanding of existing literature on social norms in HRI and identify any potential research gaps we could fill. Our analysis revealed that social norms play a significant role in shaping user perceptions and experiences in HRI, with robots that adhere to social norms generally being perceived as more acceptable, trustworthy, and effective. Moreover, future research should involve direct interactions with robots rather than relying on online studies and simulations, and extend beyond the reliance solely on self-reported questionnaires by incorporating diverse assessment methods (e.g., behavioural measures and physiological data) to capture the intricacies of human-robot interactions. Our experiment used a 2x1 between-participant experimental design where participants were randomly assigned to one of two groups (i.e., experimental or control). The participants in the experimental group were exposed to social norm violations carried out by a robot during a competitive scavenger hunt game. In contrast, the participants in the control group did not experience any social norm violations by the robot. Throughout the experiment, we collected video footage of the participants for behavioural observations, and physiological data in the form of Galvanic Skin Response (GSR) and Photoplethysmography (PPG) to detect event-driven changes in Skin Conductance Response (SCR) and Heart Rate Variability (HRV). Additionally, we collected responses to self-reported questionnaires and open-ended questions targeting perceptions of trust, discomfort, competence, and overall enjoyment. By integrating diverse data streams we provide a comprehensive account of HRI dynamics in the context of social norm violations. Results reveal significant shifts in human perception and attitudes toward robots when social norms are violated. Notably, we found a decrease in participants' trust and overall enjoyment, an increase in discomfort, and physiological state changes that complement the findings on perceptual changes and further underscore the impact of these norm violations. These findings highlight the importance of adherence to social norms in designing and programming robots to integrate into human-centric environments successfully. This study offers insights into advancing the field of HRI and the design of socially compliant robots. It spotlights the significant influence of norm violations on human perception and physiological states, paving the way for improved understanding and encouraging more positively perceived interactions with robots.