The complexity of legal language and limited accessibility to legal information pose significant challenges to justice delivery in Nepal. Traditional legal services remain inaccessible to many citizens due to language barriers, information fragmentation, and a critical shortage of legal expertise, particularly in rural areas.
Model Architecture and Training
We present NepLEGiT (Nepali Legal Expertise through Generative and Extractive Pre-trained Transformers), a specialized small language model (SLM) designed to democratize legal knowledge. The team pre-trains a decoder-based GPT-2 SLM from scratch on a curated corpus of ~4 million tokens of Nepali legal text.
The model comprises ~30 million parameters in a 6-layer, 6-head, 384-dimensional transformer. Training utilized warmup cosine-decay scheduling, gradient accumulation, and mixed-precision arithmetic.
Evaluation Results
On a held-out validation split, NepLEGiT attains a cross-entropy loss of 0.5684 and a perplexity of 1.8. The model also achieves a next-token prediction accuracy of 82.9%.
Researchers further evaluated continual masked-language-model pre-training of mBERT and MuRIL on the same corpus. mBERT achieved a perplexity of 2.35 (eval loss 0.8565), outperforming MuRIL (perplexity 6.07, eval loss 1.8026), providing a strong encoder baseline complementary to NepLEGiT's generative orientation.
Source: https://arxiv.org/abs/2609.16010
