Traduttore, traditore: can machines outperform humans in translation accuracy?

Authors

DOI:

https://doi.org/10.15381/lengsoc.v23i2.26968

Keywords:

automatic translation, human translation, BLEU metric, translation accuracy, comparative analysis

Abstract

This study examines the efficacy and accuracy of Automatic Translation (AT) compared to Human Translation (HT), employing the Bilingual Evaluation Understudy (BLEU) metric for evaluation. The rapidly evolving field of language translation, especially in the context of machine learning and Artificial Intelligence (AI), necessitates a critical assessment of AT versus HT. We aim to compare the quality of machine-generated translations from Google Translate, DeepL, and ChatGPT 3.5 with HT in the English-to-Spanish language pair. The study employs the BLEU metric, comparing machine and human translations with a professional standard. Data from translation student exams are used for human-generated translations. Our findings indicate a higher structural correlation in machine-generated translations than previously reported, suggesting an increasing proficiency in AT. However, this study emphasises the need for continued evaluation as translation technologies evolve.

Author Biographies

  • Felipe von Hausen, Universidad de Las Américas, Concepción, Chile

    Is a translator of Spanish, German and English from the Universidad de Concepción, Chile. Teacher of German Language from the Universidad de Talca, Chile. He holds a Master’s in Higher Education and a Master’s in Applied Linguistics, both from the Catholic University of the Most Holy Conception, Chile. Academic at the Faculty of Communications and Arts, Universidad de las Américas, Concepción, Chile. His research area is experimental psycholinguistics, with a special emphasis on the processing of syntax and lexicon in L2.

  • Cristóbal Muñoz, Universidad de Concepción, Concepción, Chile

    Is a translator of Spanish, English and German from the Universidad de Concepción, Chile. Japanese Teacher, Master’s in Spanish Language Teaching from the Universidad de Barcelona, Spain, and Software Developer. Independent Translator and Teacher in Australia. His research areas are translation studies and L2 learning.

  • Carlos Contreras, Universidad de Concepción, Concepción, Chile

    Was a distinguished linguist and translator with a diverse academic background. He earned a Bachelor’s degree in Education, a degree in English Language Teaching, and a professional degree in Translation from the University of Concepción (UdeC), specialising in German, French, English, and Spanish. Dr. Contreras pursued advanced studies, obtaining a Master’s and PhD in Linguistics from UdeC. Additionally, he completed postgraduate studies in Italian Language and Culture, and had proficiency in Swedish. He taught Italian and significantly contributed to the field of translation education, particularly in German and French, shaping the curriculum of the Translation and Interpretation Programme at UdeC.

References

Abu-Zahra, M. J., & Shayeb, A. S. (2022). Do mobile translation apps enhance or hinder translation trainees’ linguistic competence: The case study of translation students at Birzeit University. Journal of Language and Linguistic Studies, 18(4).

Ahammad, S. H., Kalangi, R. R., Nagendram, S., Inthiyaz, S., Priya, P. P., Faragallah, O. S., Mohammad, A., Mahmoud, M. A., & Rashed, A. N. Z. (2024). Improved neural machine translation using Natural Language Processing (NLP). Multimedia Tools and Applications, 83(13), 39335-39348.

Ali, M. A. (2020). Quality and machine translation: An evaluation of online machine translation of English into Arabic texts. Open Journal of Modern Linguistics, 10(5), 524-548.

Brownlee, J. (2017, november 20). A Gentle Introduction to Calculating the BLEU Score for Text in Python [Blog post]. Machine Learning Mastery.

Callison-Burch, C., Koehn, P., & Osborne, M. (2006, june). Improved statistical machine translation using paraphrases. In Proceedings of the Human Language Technology Conference of the NAACL, Main Conference (pp. 17-24).

Chesterman, A. (2009). The name and nature of translator studies. HERMES-Journal of Language and Communication in Business, (42), 13-22.

Colman, T., Fonteyne, M., Daems, J., & Macken, L. (2021). It’s all in the eyes: An eye tracking experiment to assess the readability of machine translated literature. 31st Meeting of Computational Linguistics in The Netherlands (CLIN 31).

DeepL Translator. (2017). DeepL Translator.

De García, J., & Pérez, R. (2011). Reflexiones y recomendaciones sobre buenas prácticas en la traducción científica y técnica. Tecnología y Desarrollo, 9, 1-9.

Dew, K. N., Turner, A. M., Choi, Y. K., Bosold, A., & Kirchhoff, K. (2018). Development of machine translation technology for assisting health communication: A systematic review. Journal of Biomedical Informatics, 85, 56-67.

Dodos, A. (2017). Phrase-based translation model.

Google Translate. (2018). GNMT en acción.

Gouadec, D. (2010). Quality in translation. In Handbook of Translation Studies (pp. 270-275). John Benjamins.

Gouws, S., & Dehghani, M. (2018, august 15). Moving Beyond Translation with the Universal Transformer. Google AI Blog.

Hartley, T. (2007). MT Evaluation – challenges and techniques.

Hernández, R., & Mendoza, C. (2018). Metodología de la investigación: Las rutas cuantitativa, cualitativa y mixta. McGraw Hill Education.

Hernández-Sampieri, R., Fernández Collado, C., & Baptista Lucio, P. (2014). Metodología de la investigación (6.ᵃ ed.). McGraw Hill.

Inghilleri, M. (2005). The sociology of Bourdieu and the construction of the ‘object’ in translation and interpreting studies. The translator, 11(2), 125-145.

Khoong, E. C., & Rodriguez, J. A. (2022). A research agenda for using machine translation in clinical medicine. Journal of General Internal Medicine, 37(5), 1275-1277.

Lavie, A. (2011). Statistical MT with Syntax and Morphology: with Syntax and Morphology: Challenges and Some Solutions.

Lee, J. (2022). Comparing student self-assessment and teacher assessment in Korean-English consecutive interpreting: Focus on fidelity and language. INContext: Studies in Translation and Interculturalism, 2(3).

Łoboda, K., & Mastela, O. (2023). Machine translation and culture-bound texts in translator education: a pilot study. In G. Massey, M. Piotrowska and M. Marczak (Eds.), (Re-)profiling T&I education: meeting evolution with innovation (pp. 503–525).

Madnani, N. (2011). iBLEU: Interactively Debugging & Scoring Statistical Machine Translation Systems. Proceedings of the Fifth IEEE International Conference on Semantic Computing.

Madnani, N. (2011). Interfaz de la sección de visualización en iBLEU.

Mikros, G., & Boumparis, D. (2022). Cross-linguistic authorship attribution and author profiling. Is machine translation the solution?

Munday, J. (2011). Introducing translation studies. Routledge.

O'Brien, S. (2012). Translation as human–computer interaction. Translation spaces, 1(1), 101-122.

Papineni, K., Roukos, S., Ward, T., & Zhu, W. (2002). BLEU: a method for automatic evaluation of machine translation. Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, Philadelphia, Pennsylvania, United States.

Pérez, C. (2013). Post-edición en el contexto de la Traducción Controlada.

Popović, M. (2020). Relations between comprehensibility and adequacy errors in machine translation output. Association for Computational Linguistics (ACL).

Snover, M., Dorr, B., Schwartz, R., Micciulla, L. and Makhoul, J. (2006). A Study of Translation Edit Rate with Targeted Human Annotation. In Proceedings of Association for Machine Translation in the Americas.

Tantuğ, A., & Oflazer, K. (2007). BLEU+: a tool for fine-grained BLEU computation. Proceedings of the Sixth International Language Resources and Evaluation, Marrakech, Morocco.

Tantuğ, A. (2007). Interfaz predeterminada de BLEU+.

Uszkoreit, J. (2017). Transformer: A Novel Neural Network Architecture for Language Understanding.

Van Gerven, M. (2017). Computational foundations of natural intelligence. Frontiers in Computational Neuroscience, 11.

Venuti, L. (2017). The translator's invisibility: A history of translation. Routledge.

Vieira, L. N., O’Hagan, M., & O’Sullivan, C. (2021). Understanding the societal impacts of machine translation: a critical review of the literature on medical and legal use cases. Information, Communication & Society, 24(11), 1515-1532.

Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cap, Y., Gao, Q., Macherey, K., Klingner, J., Shah, A., Johnson, M., Liu, X., Kaiser, L., Gows, S., Kato, Y., Kudo, T., Kazawa, H.,... & Dean, J. (2016). Google's neural machine translation system: Bridging the gap between human and machine translation. ArXiv, 1.

Zappatore, M., & Ruggieri, G. (2023). Adopting machine translation in the healthcare sector: A methodological multi-criteria review. Computer Speech & Language, 84, 1-46. https://doi.org/10.1016/j.csl.2023.101582

Downloads

Published

2024-12-30

Issue

Vol. 23 No. 2 (2024)

Section

Dossier sobre inteligencia artificial, lenguaje y discurso digital

License

Copyright (c) 2024 Felipe von Hausen, Cristóbal Muñoz y Carlos Contreras

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

AUTHORS RETAIN THEIR RIGHTS

a. Authors retain their trade mark rights and patent, and also on any process or procedure described in the article.

b. Authors can submit to the journal Lengua y Sociedad, papers disseminated as pre-print in repositories. This should be made known in the cover letter.

c. Authors retain their right to share, copy, distribute, perform and publicly communicate their article (eg, to place their article in an institutional repository or publish it in a book), with an acknowledgment of its initial publication in the journal Lengua y Sociedad.

d. Authors retain theirs right to make a subsequent publication of their work, to use the article or any part thereof (eg a compilation of his papers, lecture notes, thesis, or a book), always indicating its initial publication in the journal Lengua y Sociedad (the originator of the work, journal, volume, number and date).

How to Cite

von Hausen, F., Muñoz, C., & Contreras, C. (2024). Traduttore, traditore: can machines outperform humans in translation accuracy?. Lengua Y Sociedad, 23(2), 791-805. https://doi.org/10.15381/lengsoc.v23i2.26968