Distilling Legal Language Models with Limited Data through Synthesis and Attribution

dc.contributor.authorArbabi, Alireza
dc.date.accessioned2026-11-20T19:56:00Z
dc.date.issued2026-11-20
dc.date.submitted2026-11-17
dc.description.abstractLarge language models are increasingly capable on specialized legal NLP tasks, but their direct use incurs recurring inference costs and may require proprietary or confidential data to be shared with external providers, creating practical barriers for sensitive legal applications. In this setting, knowledge distillation provides a natural approach for building small and specialized language models. However, conventional distillation methods are limited when only a sparse set of real task examples is available, a common challenge in the legal domain. To address this challenge, this thesis presents LegalSynth, a framework that uses synthetic data generation followed by attribution-based data selection as an alternative channel for transferring knowledge from the teacher to the student. Given a task description and a small set of real examples, LegalSynth first prompts a teacher model to infer the task structure from the available task information and then leverages this understanding to generate a large and diverse pool of synthetic training examples beyond the initial seed set. Because synthetic examples vary in quality and utility, LegalSynth then applies data attribution to filter the synthetic pool and retain the examples that are most useful for training, improving the performance of the student model. Experiments across four low-resource, multi-way legal classification tasks from LegalBench demonstrate the effectiveness of this framework, enabling Qwen3.5 models with up to 4B parameters to compete with, and in several settings outperform, frontier language models including GPT-5.6-Terra, Gemini-3.7-Flash, and Qwen3.5-397B.
dc.identifier.urihttps://hdl.handle.net/10012/24714
dc.language.isoen
dc.pendingfalse
dc.publisherUniversity of Waterlooen
dc.titleDistilling Legal Language Models with Limited Data through Synthesis and Attribution
dc.typeMaster Thesis
uws-etd.degreeMaster of Mathematics
uws-etd.degree.departmentDavid R. Cheriton School of Computer Science
uws-etd.degree.disciplineComputer Science
uws-etd.degree.grantorUniversity of Waterlooen
uws-etd.embargo.terms0
uws.contributor.advisorHe, Xi
uws.contributor.affiliation1Faculty of Mathematics
uws.peerReviewStatusUnrevieweden
uws.published.cityWaterlooen
uws.published.countryCanadaen
uws.published.provinceOntarioen
uws.scholarLevelGraduateen
uws.typeOfResourceTexten

Files

Original bundle

Now showing 1 - 1 of 1
Loading...
Thumbnail Image
Name:
Arbabi_Alireza.pdf
Size:
8.89 MB
Format:
Adobe Portable Document Format

License bundle

Now showing 1 - 1 of 1
Loading...
Thumbnail Image
Name:
license.txt
Size:
6.4 KB
Format:
Item-specific license agreed upon to submission
Description:

Collections