UWSpace is currently experiencing technical difficulties resulting from its recent migration to a new version of its software. These technical issues are not affecting the submission and browse features of the site. UWaterloo community members may continue submitting items to UWSpace. We apologize for the inconvenience, and are actively working to resolve these technical issues.
 

AfriBERTa: Towards Viable Multilingual Language Models for Low-resource Languages

Thumbnail Image

Files

Ogueji_Kelechi.pdf (1.32 MB)

Date

2022-08-29

Authors

Ogueji, Kelechi

Journal Title

Journal ISSN

Volume Title

Publisher

University of Waterloo

Abstract

There are over 7000 languages spoken on earth, but many of these languages suffer from a dearth of natural language processing (NLP) tools. Multilingual pretrained language models have been introduced to help alleviate this problem. However, the largest pretrained multilingual models were trained on only hundreds of languages. This is a small amount when compared to the number of spoken languages. While these models have displayed impressive performance on several languages, including those they were not pretrained on, there is a lot of ground to be covered. A lot of languages are often left out because pretrained language models are assumed to require a lot of training data, which the languages do not have. Furthermore, a major motivation behind these models is that such lower-resource languages benefit from joint training with higher-resource languages. In this thesis, we challenge both these assumptions and present the first attempt at training a multilingual language model on only low-resource languages. We show that it is possible to train competitive multilingual language models on less than one gigabyte of text data containing a selection of African languages. Our model, named AfriBERTa, covers 11 African languages, including the first language model for 4 of these languages. We evaluate this model on named entity recognition and text classification spanning 10 languages. Our evaluation results show that our model is very competitive with larger multilingual models - multilingual BERT and XLM-RoBERTa - on several languages. Results suggest that our “small data” approach based on similar languages may sometimes work better than joint training on large datasets with high- resource languages. Furthermore, we present a comprehensive discussion of the implications of our findings.

Description

Keywords

natural language processing, multilingual, language model, named entity recognition, pre-trained language model, text classification

LC Keywords

Citation

URI

http://hdl.handle.net/10012/18662

Collections

Theses
Computer Science