The effects of integrating kinesthetic feedback, force feedback and non-speech audio feedback in human-computer interaction

Abstract

This research examines combining kinesthetic and force feedback in input devices with non-speech audio feedback (earcons) in interfaces for "everyday" computing tasks. A study involving human participants was designed to test the effects of this type of multimodel feedback in "point and select" computer tasks. This was a between-subject experiment (3 mouse-feedback groups) with repeated measures (2 x 2). The mouse-feedback groups were no feedback (using a normal mouse), kinesthetic feedback, and kinesthetic and force feedback. Repeat measures were sound condition (on and off) and task condition (single or duel). All participants performed a series of trials on a custom-built Feedback Calculator application that also recorded statistical data on each trial. Objective measures such as speed and accuracy as well as the NASA TLX workload measures along with Subjective Questionnaire responses were used for the analysis.

The results confirmed performance differences due to single and dual task conditions. Trial times increased (p < 0.001) and error rates increased (p < 0.001) for dual task conditions compared to single task conditions. However, results suggest that performance in the two task conditions were handled differently by each mouse-feedback group based on task difficulty. The results show that a mouse with kinesthetic and force feedback is useful in improving speed and accuracy in dual task conditions which required increased visual information processing when compared to single task conditions (p < 0.033). In single task conditions, the multi-feedback mouse was also useful in improving accuracy while maintaining the same speed as a normal mouse.

For single task conditions, auditory feedback did not significantly improve performance for any of the groups. However, in dual task conditions, participants using the normal mouse made use of auditory feedback to a greater level than participants using the kinesthetic mouse or kinesthetic and force feedback mouse. The results also indicate that using a kinesthetic mouse was not as effective as kinesthetic and force feedback mouse for both simple tasks and difficult tasks. Nonetheless, participants did not make use of the proactive auditory feedback provided by the earcons.

The workload analysis showed that there were significant differences between the feedback conditions on multiple factors. Participants using the kinesthetic and force feedback mouse had significantly higher physical demand ratings (p < 0.016). Further, participants rated the force feedback mouse as having less ease of movement (p < 0.001). Subjective measures indicated that participants found that trials with sound less mentally demanding (p < 0.001), less temporally demanding (p < 0.001), and took less effort (p < 0.001) to complete than trials without sound even though there were no significant differences in terms of actual performance (time and accuracy).

This research has examined one combination of multi-modal feedback that has not been studied before. The results were encouraging and suggest that feedback can be used to improve various aspects of human computer interaction. However, this research has found that the effects of feedback can be difficult to interpret because it influences performance, workload measures, and user preference measures differently. Further work in this area could contribute to a better understanding of the appropriate use of multi-modal feedback in a variety of different task situations.