Exploring Automated Trading: Modeling a Financial System with a Variable Degree of Automation, Display Design, and Evaluation

Abstract

Automation has become more pervasive and started to replace human operation in all information-processing stages (e.g., machine-learning based systems). Typically, automated systems have a variable degree of automation (DOA) that challenges modeling and design. Cognitive Work Analysis (CWA) is a modeling approach to support analysts in coping with the complexity of socio-technical domains and has shown success in providing implications for developing ecological displays that are effective in unanticipated situations. CWA allows space for modeling automated systems but has not been well developed to describe variable DOAs. This dissertation explores this problem by focusing on the case of automated trading which is underexplored in both human factors research and finance research. This dissertation starts with developing a modeling approach then focuses on design and evaluation.

Modeling. A DOA layering approach on the decision ladder was developed to serve as the theoretical foundation of this research. Two cases of automated trading – basket trading and trend following trading, each using a different DOA – were presented to facilitate the development of this approach. With this approach, the two most commonly used modeling tools in the CWA, the abstraction hierarchy, and the decision ladder each adopted an additional layer for representing human-administrated functions and automated functions. Also, the four information-processing stages in which the automation could take place were marked on the decision ladder to allow for a more detailed level mapping, which is unique in the CWA research. The DOA layering approach was demonstrated to extend the use case of CWA to include automated systems with a variable DOA and have important implications for ecological display design as well as automation design.

Design and evaluation. The experimental approach presented later in this dissertation further explored automation and display design implications of the DOA layering approach using AUTRASS (AUtomated TRAding System Simulation), a simulation developed as part of this research program. Two experimental studies on trend following trading are reported in which the design concepts were evaluated.

In Experiment 1, inspired by the CWA models, automation was designed as two configurations to represent distinct DOA situations. The moderate DOA configuration simulated a trading situation in which the participants performed a flexible trading task. The high DOA configuration represented a higher DOA situation where a trading algorithm that was unfamiliar to the participants traded in a similar market condition, and the participants monitored the automation and performed a fault detection task. Two types of displays were designed. Conventional displays were typical in information content and form to current trading displays and should support the basic use of the automation. Ecological displays were implemented from the CWA models to support monitoring for unanticipated situations for each DOA condition. Four scenario types were developed by combing the two DOA configurations with the two display types. Experiment 1 involving 24 participants was conducted to thoroughly examine the effectiveness of ecological displays with different DOAs. Based on the literature, the ecological displays were hypothesized to improve task performance and situation awareness and to trigger riskier actions without imposing higher workload. Results of Experiment 1showed that the ecological displays did not provide better support on either trading performance (moderate DOA) or fault detection performance (high DOA). Empirically, a trade-off of situation awareness and workload between the two DOA configurations seemed to exist without the influence of the ecological displays. Interestingly, the results of this study suggested a different pattern of risk preference compared to that in the literature. Specifically, the ecological displays imposed riskier financial trading decisions. The results of Experiment 1 provided implications for identifying system and contextual factors that could influence risk preference and demonstrated sufficient space for improving automation design.

Experiment 2 is a follow-up to the first experimental study with separately recruited 24 participants. The high DOA configuration previously used in Experiment 1 was improved with its traits inherited and its flaws in automation design eliminated (i.e., the improved-high DOA configuration was expected to be better supported by the ecological displays). An adaptive configuration was introduced to simulate adaptive automation in the automated trading setting. The conventional displays and the ecological displays continued to be used without any modifications, and similar hypotheses were examined with the two new DOA configurations. Results of Experiment 2 showed that with the improved automation design in the high DOA configuration, the ecological display significantly improved fault detection performance. A consistent pattern of risk preference was found in this study as in Experiment 1.

A comparison of the two experimental studies showed new opportunities to derive automation and display design from the DOA layering approach that can support fault detection performance in automated trading, and future research is warranted to explore the influence of ecological displays on risk preference. The development of AUTRASS also makes a unique contribution. The design of the different DOA configurations demonstrated the applicability of the DOA layering approach to guide automation design. Overall, the following conclusions were reached by conducting this dissertation research: CWA has been effective in characterizing the complexity in automated trading that is associated with the variable DOA, and it can further support the design of automation and the ecological displays. Ecological displays may foster risky operation with moderate DOA as well as performance improvement with high DOA. The applicability of the proposed approaches spanning modeling, design, and evaluation should go beyond the limit of automated trading to the brave new world of artificially intelligent automation.