PERSONA: A Tool for Generating Algorithmic Personas for Reflective Annotations

Abstract

The domain of machine learning (ML) has grappled with the challenge of curating subjective datasets, where there can be many equally valid labels due to differences in perspectives and a significant technical gap remains in how we can effectively incorporate multiple subjective viewpoints into the labelling process. We contribute PERSONA, a dataset labelling tool that presents LLM-generated personas with diverse labelling per- spectives to encourage annotators to consider different human values during the dataset labelling process. We studied how interactions with these personas affect the annotator’s decision-making patterns. Based on a two-part user study, our evaluation shows that PERSONA enriches the labelling process by prompting the annotators to reflect on differ- ent viewpoints, showing the potential value of integrating LLMs in machine learning data generation pipelines.