| dc.description.abstract | Intermittent pneumatic compression (IPC) is used in the management of vascular disorders. IPC systems apply external pressure to the circumference of the leg to enhance blood velocity (BV). Although IPC device performance can be evaluated by the induced change in mean blood velocity (a representation of the device’s ability to vacate venous blood from the legs), commercial IPC systems are often designed with the main considerations of cost and size instead of performance. Previous studies have demonstrated that the application of cardiac-gated compression further enhanced blood velocity (BV) compared to fixed compression timing (CT). However, optimal CT, as a function of the cardiac cycle, is not constant across individuals and may change over time. Additionally, IPC is often being used over the course of days and weeks to promote blood flow. Consequently, at the start of a therapy session, the optimal CT for an individual is unknown. Learning the optimal CT for an individual will maximize BV and make the therapy session more efficient. Current CT modelling methods for IPC are limited to predictions for a single day and one heartbeat ahead, which is not sufficient for typical IPC usage since typical IPC sessions last for 60 minutes and IPC is used over the course of days and weeks.
In this thesis a data-driven predictive model to simulate the blood flow response to compression timing across two days was built. Furthermore, a deep reinforcement learning agent learns the CT that maximizes blood flow for each heartbeat in a custom-built simulated environment. Finally, a deep reinforcement learning agent optimizes compression timing with a human-in-the-loop.
In this thesis, six participants wore a custom IPC system and experienced random cardiac-gated CTs for 1.5 hours per day for two days. Six user-specific nonlinear autoregressive models with exogenous inputs (NARX) were implemented using an artificial neural network (ANN) to estimate the BV response to a CT. The BV response to a CT is estimated using past inputs of BV, electrocardiography (ECG), and CT. The NARX model is trained on the first session. This predictive model can be used online to estimate the optimal cardiac gated CT for a therapy session. The mean R2 for this model across participants on the second session was 0.74 ± 0.09 and the mean absolute error was approximately 3%, which is a reduction of only 11% compared to the first sessions, for both metrics. This study is the first to show that BV across IPC sessions can be predicted using a pre-trained model.
A deep reinforcement learning (DRL) algorithm learns the optimal compression timing to maximize its cumulative reward, mean beat-to-beat mean blood velocity (BBMV). In this work, participant-specific simulated lower limb environments were built, using the aforementioned NARX models, for six participants and show that DRL can optimize CT, across 19 CTs, in an IPC framework. We show that the DRL agent can adapt to changes in the physiological state unlike previous methods which employ fixed dynamics. The DRL agent can learn an optimal policy in 15 minutes ± 2 on average. The proposed DRL agent can be implemented in IPC systems to rapidly learn the optimal CT with a human-in-the-loop. This is particularly valuable as the ideal CT is unknown at the outset of a therapy session, and learning it maximizes the benefit of therapy.
It is shown that a DRL agent can optimize IPC CT with a human-in-the-loop within a 60-minute therapy session. This study included 10 participants. To observe learning within 60 minutes, the DRL agent optimizes across two cardiac gated CTs, diastole and systole. The DRL agent state is composed of previous heart rate and BBMV information. The utilized reward function incentivizes constant increases in BBMV to achieve agent convergence. The DRL agent converged to diastole CT at each heartbeat across 10 participants, selecting diastolic compression 83% ± 9 of the time. The mean difference in blood velocity between the DRL CT and cardiac-gated diastolic CT was -0.09 cm/s ± 0.09 (-4% ± 4). This DRL agent is the first of its kind to optimize IPC CT with a human-in-the-loop. | en |
