Paper Information | SBC BOOKS

Fill in your paper information

Language (*)

Title (*)

Keywords (*)

Abstract (*)

Várias métricas de avaliação para geração de texto foram propostas nos últimos anos. No entanto, muitas questões surgiram sobre o quão bem elas podem avaliar a acurácia e a qualidade do texto gerado. Neste trabalho, estudamos como algumas das métricas de geração de texto mais populares se comportam ao lidar com a tarefa de sumarização de texto no domínio jurídico em Português. Mais especificamente, avaliamos cinco métricas -- ROUGE, BERTScore, BARTScore, BLEURT e MoverScore --, usando um dataset contendo 892 acórdãos do Superior Tribunal de Justiça. Cada item do dataset é composto por um acórdão, que é o documento jurídico original, e uma ementa, que corresponde a um resumo manualmente gerado do documento jurídico original. Nosso estudo revelou que, para o domínio jurídico brasileiro, nenhuma das métricas avaliadas foi capaz de mensurar totalmente a qualidade dos resumos gerados manualmente quando comparados com seus documentos originais, e que, dentre as métricas avaliadas, ROUGE e BERTScore apresentaram os resultados mais promissores.

Pages (*)

File Link

English Information

Title

Keywords

Abstract

Various evaluation metrics for text generation have been proposed in recent years. However, many questions have emerged about how well they can evaluate the accuracy and quality of the text generated. In this work, we study how some of the most popular text generation metrics behave when dealing with the text summarization task in the Portuguese legal domain. More specifically, we evaluated five metrics -- ROUGE, BERTScore, BARTScore, BLEURT, and MoverScore --, using a dataset containing 892 rulings from the Brazilian Superior Court of Justice. Each item in the dataset is composed of a ruling, which is the original legal document, and a syllabus, which corresponds to a manually generated summary of the original legal document. Our study revealed that, for the Brazilian legal domain, none of the metrics evaluated were capable of fully measuring the quality of manually generated summaries when compared with their original documents, and that, among the evaluated metrics, ROUGE and BERTScore presented the most promising results.

(*) To change the order drag the item to the new position.

Authors

#	Name
1	Altigran da Silva(alti@icomp.ufam.edu.br)
2	Daniela Vianna(dvianna@icomp.ufam.edu.br)
3	Gustavo Feltrin(feltrin@icomp.ufam.edu.br)

(*) To change the order drag the item to the new position.

Reference

#	Reference
1	Fabbri, A. R., Kryściński, W., McCann, B., Xiong, C., Socher, R., and Radev, D. (2021). Summeval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391–409.
2	Farzindar, A. and Lapalme, G. (2004). LetSum, an automatic legal text summarizing system. In Jurix, pages 11–18.
3	Feijó, D. d. V. and Moreira, V. P. (2019). Summarizing legal rulings: Comparative experiments. In Proceedings of the International Conference on Recent Advances in Natural Language Processing, RANLP 2019, pages 313–322.
4	Feijó, D. d. V. and Moreira, V. P. (2023). Improving abstractive summarization of legal rulings through textual entailment. Artificial Intelligence and Law, 31(1):91–113.
5	Guimarães, J. A. C. (2004). Elaboração de ementas jurisprudenciais: elementos teórico-metodológicos, volume 9. Subsecretaria de Divulgação e Editoração da Secretaria de Pesquisa e Informação Jurídicas do Centro de Estudos Judiciários.
6	Jain, D., Borah, M. D., and Biswas, A. (2021). Summarization of legal documents: Where are we now and the way forward. Computer Science Review, 40:100388.
7	Kryściński, W., Keskar, N. S., McCann, B., Xiong, C., and Socher, R. (2019). Neural text summarization: A critical evaluation. arXiv preprint arXiv:1908.08960.
8	Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. (2020). BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 7871–7880.
9	Lin, C.-Y. (2004). ROUGE: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.
10	Liu, Y. (2019). Fine-tune BERT for extractive summarization. CoRR, abs/1903.10318.
11	Pandya, V. (2019). Automatic text summarization of legal cases: A hybrid approach. In 5th International Conference on Advances in Computer Science and Information Technology (ACSTY-2019).
12	Polsley, S., Jhunjhunwala, P., and Huang, R. (2016). Casesummarizer: A system for automated summarization of legal texts. In COLING 2016, 26th International Conference on Computational Linguistics, Proceedings of the Conference System Demonstrations, pages 258–262.
13	Pu, A., Chung, H. W., Parikh, A. P., Gehrmann, S., and Sellam, T. (2021). Learning compact metrics for MT. In Conference on Empirical Methods in Natural Language Processing.
14	Sellam, T., Das, D., and Parikh, A. P. (2020). BLEURT: learning robust metrics for text generation. CoRR.
15	Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. CoRR, abs/1706.03762.
16	Yuan, W., Neubig, G., and Liu, P. (2021). Bartscore: Evaluating generated text as text generation. In Advances in Neural Information Processing Systems, pages 27263–27277.
17	Zhang, J., Zhao, Y., Saleh, M., and Liu, P. J. (2020). PEGASUS: pre-training with extracted gap-sentences for abstractive summarization. In Proceedings of the 37th International Conference on Machine Learning, pages 11328–11339.
18	Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., and Artzi, Y. (2019). Bertscore: Evaluating text generation with BERT. CoRR.
19	Zhao, W., Peyrard, M., Liu, F., Gao, Y., Meyer, C. M., and Eger, S. (2019). Moverscore: Text generation evaluating with contextualized embeddings and earth mover distance. CoRR.

Paper Registration

Fill in your paper information

English Information

Authors

Reference