Advancing ITR as a standard metric for real-time BCI performance assessment

Abstract

The current standard metric for assessing brain-computer interface (BCI) system performance is the information transfer rate (ITR). However, in our previous work using ITR to evaluate real-time BCI-controlled electric wheelchair performance, we obtained low ITR values while significantly outperforming prior studies in terms of task completion time. This led to the belief that ITR is an inconsistent metric for real-time applications, potentially indicating misleading results when comparing systems. The discrepancy in performance led to examining the limits of ITR and proposing an alternative metric, Jun’s Information Transfer Rate (JITR), aimed at addressing specific issues often overlooked in the current BCI system evaluations. In the literature, several key assumptions behind ITR are not met, including a memoryless system, independent choices, a constant update rate, and equally distributed choice probabilities. Common post-processing methods, such as weighted smoothing, feedback, and error correction, break these assumptions. As a result, ITR values become unreliable for comparison for real-time systems, motivating the use of JITR that maintains the basis of maximum channel bandwidth log2𝑁 while adding term in to penalized system delay and low accuracy. This approach affords opportunities to fine tune and optimize BCI configurations and predict task performance. To validate JITR, the update rate and weighted sum smoothing parameters were manipulated to find the configuration with highest transfer rate. Utility was tested in a simulated driving task, where performance was predicted using the JITR performance index. As a result, JITR was able to predict task completion time, if immediate pre-trial accuracy is given. JITR limitations, such as the need to model fatigue, learning effects, and false positives to improve precision and recall, and benefits are discussed, representing a step toward a reliable metric for assessing BCI systems.