Neural Face: Generative Models for Continuous Digital Avatars
Loading...
Files
Date
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
University of Waterloo
Abstract
Digital avatars could enable more expressive virtual communication, personalized AI assistants, and new forms of telepresence. However, most existing avatar-generation systems focus on producing short video clips. A digital avatar that participates in an ongoing interaction must instead generate video continuously while preserving its identity, responding naturally to speech and facial motion, and remaining visually coherent over long periods of time.
This thesis studies continuous digital avatar generation. Given a target identity and a live driving stream, the system aims to render an animated version of the target that follows the driver's expressions and behavior without inheriting the driver's appearance. We develop a recurrent diffusion-based architecture that maintains visual state across time, allowing the avatar to generate extended sequences without repeatedly processing its entire history.
Our experiments reveal that continuous generation introduces a primary identity-stability failure mode. In the baseline autoregressive system, the avatar gradually loses the target's facial structure and converges toward the identity of the driving person, producing identity leakage in 93% of evaluated frames under our latent-space diagnostic. We introduce explicit identity anchoring, feature-space projection, and knowledge transfer from a pretrained diffusion transformer to better separate who the avatar is from how it moves. Under teacher-forced evaluation with clean historical context, these methods reduce observed identity leakage to 0%. However, identity drift can reappear during fully autoregressive generation as small visual errors accumulate over time. We also find that incorporating dense audio representations can disrupt the model's visual memory, highlighting an additional challenge for speech-driven avatars.
These results demonstrate progress toward digital avatars that can sustain a coherent visual presence throughout an interaction. They also identify long-term identity preservation, autoregressive stability, and audiovisual integration as key challenges for building persistent digital humans.