Symbolic Regression and Sequence Modelling with Conditional and Dynamic Language Models
Loading...
Date
2024-05-30
Authors
Valipour, Mojtaba
Journal Title
Journal ISSN
Volume Title
Publisher
University of Waterloo
Abstract
In an era where the boundaries of machine learning are continuously being pushed, this thesis presents two more advancements in the field of deep learning and artificial intelligence, with a focus on symbolic regression and dynamic training methodologies for neural networks. The first major contribution, SymbolicGPT, introduces a novel approach to symbolic regression using a transformer-based language model. This model significantly outperforms traditional methods by leveraging the strengths of probabilistic language models for improved accuracy and efficiency. The second theme of this thesis revolves around dynamic training methodologies, aimed at enhancing the adaptability and computational efficiency of neural networks under varying constraints. Within this framework, we introduce DyLoRA and SortedNet as key innovations. DyLoRA offers a dynamic, search-free low-rank adaptation technique, enabling models to adjust their complexity on-the-fly without extensive retraining. SortedNet proposes a generalized framework for embedding multiple neural network architectures within a single model, facilitating efficient model selection and adaptation. Extending SortedNet, SortedLLama applies these principles to large language models, demonstrating efficient dynamic inference capabilities.
Description
Keywords
Deep Learning, Natural Language Processing, Large Language Models, Symbolic Regression, Dynamic Inference, Modular Neural Networks, Anytime Inference, Low-Rank Adaptation