Paper Information | SBC BOOKS

Fill in your paper information

Language (*)

Title (*)

Keywords (*)

Abstract (*)

The growing volume of digital documents in Brazilian courts has intensified challenges related to judicial inefficiency, including the slow identification of established precedents and the risk of inconsistent rulings in similar cases. To address this, this study presents a systematic empirical study on the effectiveness of different classes of BERT-based
embedding models for the semantic retrieval of legal documents, focusing on identifying relevant precedents for new complaints. A pipeline employing document chunking to handle long legal texts and using Elasticsearch for large-scale dense vector retrieval was implemented and evaluated. Using a corpus of legal complaints from the Maranhão State Court of Justice, three model categories were compared: (i) a general-purpose Portuguese model, (ii) three domain-specific models trained on Brazilian legal corpora, and (iii) a task-specific Sentence-BERT model fine-tuned for similarity tasks. Performance was assessed using a proxy-based protocol with precedent class labels as the ground truth, measured by Precision@k, MRR@15, and MAP@15. The results indicate that the task-specific SBERT-pt model achieved higher performance within the evaluated setting, consistently outperforming other tested models across the selected metrics. This model showed improvements in both immediate relevance (Precision@1 of 0.787) and overall ranking quality (MAP of 0.806). Although domain adaptation provided marginal benefits over the general-purpose baseline, task-specific finetuning for similarity appeared to be the most influential factor for retrieval quality in this scenario. These findings suggest that optimizing models for retrieval tasks can offer a promising direction for enhancing semantic searches in legal contexts. However, further studies using larger and more diverse datasets are needed to confirm the generalizability of these results and assess their impact on real-world judicial workflows.

Pages (*)

File Link

English Information

Title

Keywords

Abstract

The growing volume of digital documents in Brazilian courts has intensified challenges related to judicial inefficiency, including the slow identification of established precedents and the risk of inconsistent rulings in similar cases. To address this, this study presents a systematic empirical study on the effectiveness of different classes of BERT-based embedding models for the semantic retrieval of legal documents, focusing on identifying relevant precedents for new complaints. A pipeline employing document chunking to handle long legal texts and using Elasticsearch for large-scale dense vector retrieval was implemented and evaluated. Using a corpus of legal complaints from the Maranhão State Court of Justice, three model categories were compared: (i) a general-purpose Portuguese model, (ii) three domain-specific models trained on Brazilian legal corpora, and (iii) a task-specific Sentence-BERT model fine-tuned for similarity tasks. Performance was assessed using a proxy-based protocol with precedent class labels as the ground truth, measured by Precision@k, MRR@15, and MAP@15. The results indicate that the task-specific SBERT-pt model achieved higher performance within the evaluated setting, consistently outperforming other tested models across the selected metrics. This model showed improvements in both immediate relevance (Precision@1 of 0.787) and overall ranking quality (MAP of 0.806). Although domain adaptation provided marginal benefits over the general-purpose baseline, task-specific finetuning for similarity appeared to be the most influential factor for retrieval quality in this scenario. These findings suggest that optimizing models for retrieval tasks can offer a promising direction for enhancing semantic searches in legal contexts. However, further studies using larger and more diverse datasets are needed to confirm the generalizability of these results and assess their impact on real-world judicial workflows.

(*) To change the order drag the item to the new position.

Authors

#	Name
1	Adrielson Justino(adrielferreira28@gmail.com)
2	Antônio Fernando Lavareda Jacob(antoniojunior@professor.uema.br)
3	Fábio Lobato(fabio.lobato@ufopa.edu.br)

(*) To change the order drag the item to the new position.

Reference

#	Reference
1	BELTAGY, I., PETERS, M. E., AND COHAN, A. Longformer: The long-document transformer, 2020.
2	BRASIL. Lei nº 13.105, de 16 de março de 2015. Código de processo civil., 2015.
3	CARMO, F. A., SEREJO, F., JUNIOR, A. F. J., SANTANA, E. E., AND LOBATO, F. M. Embeddings jurídico: Representações orientadas à linguagem jurídica brasileira. In Anais do XI Workshop de Computação Aplicada em Governo Eletrônico, 2023.
4	DE SOUZA, C. M. F. AND DE SOUZA SALLES, S. Acesso à justiça em tempos de pandemia: a experiência do núcleo permanente de métodos consensuais de tratamento de conflitos do tjrj. Conhecimento & Diversidade, 2022.
5	DEVLIN, J., CHANG, M.-W., LEE, K., AND TOUTANOVA, K. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019.
6	HARISPE, S., RANWEZ, S., MONTMAIN, J., ET AL. Semantic similarity from natural language and ontology analysis, 2022.
7	HE, H. AND GARCIA, E. A. Learning from imbalanced data. IEEE Transactions on Knowledge and Data Engineering, 2009.
8	IZACARD, G., CARON, M., HOSSEINI, L., RIEDEL, S., BOJANOWSKI, P., JOULIN, A., AND GRAVE, E. Unsupervised dense information retrieval with contrastive learning, 2022.
9	KARPUKHIN, V., OGUZ, B., MIN, S., LEWIS, P. S., WU, L., EDUNOV, S., CHEN, D., AND YIH, W.-T. Dense passage retrieval for open-domain question answering. In EMNLP (1), 2020.
10	KU, L.-W., WU, T.-H., LEE, L.-Y., AND CHEN, H.-H. Construction of an evaluation corpus for opinion extraction. In NTCIR, 2005.
11	MAGALHÃES, R. A. AND FREITAS, F. O. A morosidade do poder judiciário e sua interferência nas relações contratuais. Revista Jurídica Cesumar-Mestrado, 2023.
12	MAIA FILHO, M. S. AND JUNQUILHO, T. A. Projeto victor: perspectivas de aplicação da inteligência artificial ao direito. Revista de Direitos e Garantias Fundamentais, 2018.
13	MARINATO, M. S., JUNIOR, A. F. J., LOBATO, F. M., AND CORTES, O. A. Classificação automática de petições iniciais usando classificadores combinados. In Anais do XVI Brazilian e-Science Workshop, 2022.
14	NI, C., WU, J., WANG, H., LU, W., AND ZHANG, C. Enhancing cloud-based large language model processing with elasticsearch and transformer models. In Proceedings of the International Conference on Image, Signal Processing, and Pattern Recognition (ISPP), 2024.
15	POLO, F. M., MENDONÇA, G. C. F., PARREIRA, K. C. J., GIANVECHIO, L., CORDEIRO, P., FERREIRA, J. B., DE LIMA, L. M. P., DO AMARAL MAIA, A. C., AND VICENTE, R. Legalnlp-natural language processing methods for the brazilian legal language. In Encontro Nacional de Inteligência Artificial e Computacional (ENIAC), 2021.
16	REIMERS, N. AND GUREVYCH, I. Sentence-bert: Sentence embeddings using siamese bert-networks.
17	RICARDO, B.-Y. AND BERTHIER, R.-N. Modern information retrieval: the concepts and technology behind search, 2011.
18	RUCKDESCHEL, M. Term-based and embedding-based similarity search in large unknown text datasets, 2020.
19	SANTOS, J. A., SOUZA, E., FILHO, C. J. B., ALBUQUERQUE, H. O., VITÓRIO, D., DE LUCENA, D. C. G., SILVA, N., AND DE CARVALHO, A. Hirs: A hybrid information retrieval system for legislative documents, 2024.
20	SCHÜTZE, H., MANNING, C. D., AND RAGHAVAN, P. Introduction to information retrieval, 2008.
21	SILVA, E. C., DE MEDEIROS, I. P., DE MENEZES, M. V., AND KAMIKAWACHI, D. S. L. Segmentation and summarization for extracting information about information technology equipment from government procurement notice. In Symposium on Knowledge Discovery, Mining and Learning (KDMiLe), 2024.
22	SILVA JUNIOR, D. D., OLIVEIRA, D. D., AND PAES, A. Evaluating text representations for unsupervised legal semantic textual similarity in brazilian portuguese. Discover Data, 2025.
23	SILVEIRA, R., PONTE, C., ALMEIDA, V., PINHEIRO, V., AND FURTADO, V. Legalbert-pt: A pretrained language model for the brazilian portuguese legal domain. In Brazilian Conference on Intelligent Systems, 2023.
24	SOUZA, F., NOGUEIRA, R., AND LOTUFO, R. Bertimbau: Pretrained bert models for brazilian portuguese. In Brazilian Conference on Intelligent Systems, 2020.
25	STEMLER, I. T. S. V. Identificação de precedentes judiciais por agrupamento utilizando processamento de linguagem natural, 2019.
26	TOFFOLI, J. A. D. AND GUSMÃO, B. G. Inteligência artificial na justiça. Brasília: CNJ, 2019.
27	WIRTH, R. AND HIPP, J. Crisp-dm: Towards a standard process model for data mining. In Proceedings of the 4th International Conference on the Practical Applications of Knowledge Discovery and Data Mining, 2000.
28	WORTSMAN, M., ILHARCO, G., KIM, J. W., LI, M., KORNBLITH, S., ROELOFS, R., LOPES, R. G., HAJISIRZI, H., FARHADI, A., NAMKOONG, H., ET AL. Robust fine-tuning of zero-shot models, 2022.

Paper Registration

Fill in your paper information

English Information

Authors

Reference