Contextual AI: Integrating Macro-Context with Transformer Architectures for Social Media Analysis, Federated Learning, and Recommender Systems

Abstract

Context is crucial for understanding the world and making informed decisions. While existing transformer architectures excel at contextualizing information locally, such as other words in a sentence, they fail to factor in broader, macro-level contexts. We identify available yet underutilized macro contexts in three use cases: online discussions, federated learning, and recommender systems. For each, we motivate the need to leverage existing macro context and propose context-aware solutions based on the transformer architecture.

In online discussion boards, the rich conversational and multimodal macro context in which a comment is made is often overlooked. This is especially pertinent in hate speech detection. Classical solutions that examine individual comments in isolation fail to account for this context, leading to ambiguity and misinterpretation. For instance, the comment ``Ew, that’s gross!'' has a different interpretation depending on whether it’s in response to food or sensitive issues like LGBTQ rights. Furthermore, images that accompany text can also provide crucial context. We propose mDT, a novel deep learning model architecture based on graph transformer networks, which incorporates this valuable context when evaluating the hatefulness of individual comments. Our experimental results demonstrate a 7\% F1 improvement over existing baselines that do not utilize this context, and a 21\% F1 improvement over previous graph-based methods.

Second, we tackle the context-agnostic paradigm of federated learning. The prevalent Federated Averaging (FedAvg) method statically averages model weights, failing to account for the crucial macro-level context of heterogeneous-agent environments, leading to a suboptimal, one-size-fits-all model. For example, autonomous driving agents exploring rural roads acquire different knowledge than those in urban settings, and this environmental context is lost in the process. We propose FedFormer, a novel federation strategy that leverages transformer attention to enable each agent to weigh and selectively incorporate insights from its peers in a context-dependent manner. In turn, FedFormer enables a more effective, efficient federation that respects and adapts to environmental diversity while preserving privacy. Our experiments across environments in MetaWorld, a set of heterogeneous robotic manipulation tasks, demonstrate improvements of 1.48x to 3.41x over FedAvg.

Finally, in recommender systems, the user’s intent can provide critical personalization context. Simple approaches rely on collaborative filtering, which only models implicit (micro-level) user preferences by extrapolating from historical data. Our solution, Flare, proposes a contextual recommender system that empowers users to steer recommendations via explicit natural language queries (e.g., ``Staplers'', ``Webcams''). Flare’s architecture fuses collaborative filtering signals with semantic representations of both the user’s explicit query and item descriptions, bridging the gap between long-term preferences and the context of the user's immediate goals. Our experiments using the Amazon Product Reviews datasets show a 1.7x and 2.53x increase in recall@1 and recall@10, respectively, compared to approaches that do not factor in user intent.