Morphological Error Analysis of Machine Translation Output
A Case Study on Kurdish Texts Translated by OpenAI ChatGPT
DOI:
https://doi.org/10.24086/cuejhss.v10n1y2026.pp120-129الكلمات المفتاحية:
Classification and Identification of Errors، Kurdish (Sorani)Language، Machine translation، Morphological error analysis، OpenAI ChatGPTالملخص
Evaluation of machine translation output is an effective step to develop the quality of any automated translation project. This step has been taken by different researchers in a variety of methods, most of them lead to holistic findings without targeting the main causes of deficiencies. The current study is an attempt to address morphological errors that affect the quality of an automatic translation system that has recently been used as the most popular platform (OpenAI ChatGPT) among Kurdish individuals and institutions to translate texts from English into Kurdish (Sorani) and vice versa. To target the aims of the study, 30 sentences from different text types defined by Reiss (2000) were selected among a 100-sentence corpus as the data of the study and translated by a practiced human translator, taking the role of reference translation and the same source sentences translated by the understudy ChatGPT system. Based on the integrated model of error categorization proposed by Popović and Arcan (2015) and the multidimensional quality metrics model developed by Lommel et al. (2014), problems related to lexical choice were by far the most common and followed by inflectional, syntax-morphology, and derivational errors, respectively. At the same time, weak semantic discrimination related to choosing the right equivalents has greatly affected the quality of the Kurdish output of the understudy system.
التنزيلات
المراجع
Aronoff, M., & Fudeman, K. (2011). What is Morphology. 2nd ed. United Kingdom: Blackwell Publishing.
Banerjee, S., & Lavie, A. (2005). METEOR: An automatic metric for ChatGPTevaluation with improved correlation with human judgments. In: Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization. Pennsylvania: Association for Computational Linguistics. pp. 65-72.
Burlot, F., & Yvon, F. (2017). Evaluating the morphological competence of machine translation systems. In: Proceedings of the 2nd Conference on Statistical Machine Translation (WCHATGPT2007). Copenhagen, Denmark: Association for Computational Linguistics. pp. 43-55. DOI: https://doi.org/10.18653/v1/W17-4705
Callison-Burch, C., Fordyce, C., Koehn, P., Monz, C., & Schroeder, J. (2007). (Meta-) evaluation of machine translation. In: Proceedings of the Second Workshop on Statistical Machine Translation. Denmark: Association for Computational Linguistics. DOI: https://doi.org/10.3115/1626355.1626373
Costa, Â., Ling, W., Luís, T., Correia, R., & Coheur, L. (2015). A linguistically motivated taxonomy for machine translation error analysis. Machine Translation, 29(2), 127-161. DOI: https://doi.org/10.1007/s10590-015-9169-0
El Kholy, A., & Habash, N. (2011). Automatic error analysis for morphologically rich languages. In: Proceedings of the Machine Translation Summit XIII. Xiamen, China: Association for Computational Linguistics.
El Marouani, M.E., Boudaa, T., & Enneya, N. (2020). Statistical error analysis of machine translation: The case of Arabic. Computación y Sistemas, 24(3), 1053-1061. DOI: https://doi.org/10.13053/cys-24-3-3289
Farr´us, M., Costa-Juss`a, MR., Mari˜no, JB., & Fonollosa, JAR. (2010). Linguistic-based evaluation criteria to identify statistical machine translation errors. In: Proceedings of the 14th Annual Conference of the European Association for Machine Translation (EACHATGPT2010). France: Saint-Raphal. pp. 167-173.
Federico, M., Negri, M., Bentivogli, L., & Turchi, M. (2014). Assessing the impact of translation errors on machine translation quality with mixed-effects models. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP 2014). Doha, Qatar: Association for Computational Linguistics. pp. 1643-1653. DOI: https://doi.org/10.3115/v1/D14-1172
Koehn, P. (2005). Europarl: A parallel corpus for statistical machine translation. In: Conference Proceedings: The Tenth Machine Translation Summit. India: AAMT. pp. 79-86.
Llitjos, AF., Carbonell, JG., & Lavie, A. (2005). A Framework for Interactive and Automatic Refinement of Transfer-Based Machine Translation. In Proceedings of EACHATGPT10th Annual Conference. pp. 30-31.
Lommel, A., Uszkoreit, H., & Burchardt, A. (2014). Using a new analytic measure for the annotation and analysis of MT errors on real data. In: Proceedings of the 17th Annual conference of the European Association for Machine Translation pp. 455-463. DOI: https://doi.org/10.5565/rev/tradumatica.77
Massoudi, A., Zaidan, M.N., & Qasim Agha, A. (2024). The adoption of technology acceptance model in E-commerce with artificial intelligence as a mediator. GECONTEC International Journal of Knowledge Management and Technology, 12(2), 20-36.
Massoudi, A.H., Fatah, S.J., & Jami, M.S. (2024). The role of artificial intelligence application in strategic marketing decision-making process. Cihan University Erbil Journal of Humanities and Social Sciences, 8(1), 34-39. DOI: https://doi.org/10.24086/cuejhss.v8n1y2024.pp34-39
Papineni, K., Roukos, S., Ward, T., & Zhu, WJ. (2002). BLEU: A method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. Vol. 2. Stroudsburg, PA, USA: Association for Computational Linguistics, ACL. pp. 311-318. DOI: https://doi.org/10.3115/1073083.1073135
Popovic, M. (2018). Error classification and analysis for machine translation quality assessment. In: Moorkens, J., Castilho, S., Gaspari, F., & Doherty, S., (eds.), Translation Quality Assessment. Vol. 1. Cham: Springer International Publishing. pp. 129-158.
Popović, M., & Arcan, M. (2015). Identifying Main Obstacles for Statistical Machine Translation of Morphologically Rich South Slavic Languages. In: Proceedings of the 18th Annual Conference of the European Association for Machine Translation. pp. 97-104.
Popović, M., & Ney, H. (2007). Word error rates: Decomposition over POS classes and applications for error analysis. In: Proceedings of the 2nd Workshop on Statistical Machine Translation (WCHATGPT2007). Prague: Czech Republic. pp. 48-55. DOI: https://doi.org/10.3115/1626355.1626362
Popović, M., & Ney, H. (2011). Towards automatic error analysis of machine translation output. Computational Linguistics, 37(4), 657-688. DOI: https://doi.org/10.1162/COLI_a_00072
Posner, R. (1975). P. H. Matthews, morphology. An introduction to the theory of word-structure. (Cambridge text-books in linguistics, 1.) London: Cambridge University Press, 1974. pp. 243. index. Journal of Linguistics, 11(2), 270-277. DOI: https://doi.org/10.1017/S0022226700004588
Rasouli, F. (2018). Assessment of machine translation output: A comparative study between human and automatic models. Cihan University-Erbil Scientific Journal, 2, 119-141. DOI: https://doi.org/10.24086/cuesj.si.2018.n1a9
Rasouli, F., Seyyedi, K., & Azaldin, A.H. (2026). A text typological assessment of automatic translation: A case study of English into Kurdish translated texts by open AI GPT and google translate. Cihan University-Erbil Journal of Humanities and Social Sciences, 10(1), 13-18. DOI: https://doi.org/10.24086/cuejhss.v10n1y2026.pp13-18
Rasouli, F., Soleimanzadeh, S., & Seyyedi, K. (2024). Acceptability of Google Translate machine translation system in translation from English into Kurdish. Cihan University-Erbil Journal of Humanities and Social Sciences, 8(1), 7-14. DOI: https://doi.org/10.24086/cuejhss.v8n1y2024.pp7-14
Reiss, K. (2000). In: Rhodes, E.F., Translator. Translation Criticism-Potentials and Limitations: Categories and Criteria for Translation Quality Assessment. Rome: St. Jerome.
Snover, M., Dorr, B., Schwartz, R., Micciulla, L., & Makhoul, J. (2006). A study of translation edit rate with targeted human annotation. In: Proceedings of the 7th Conference of the Association for Machine Translation in the Americas (AMTA 2006). Boston, MA: Association for Machine Translation in the Americas. pp. 223-231.
Spencer, A., & Zwicky, A.M., (eds.). (1998). The Handbook of Morphology. United States: Blackwell.
Van Der Wees, M., Bissaza, A., & Monz, C. (2015). Five Shades of Noise: Analyzing Machine Translation Errors in User-Generated Text. In: Proceedings of the Workshop on Noisy User-Generated Text. pp. 28-37. Available from: https:// www.researchgate.net/publication/301449172 [Last accessed on 2026 Jan 10]. DOI: https://doi.org/10.18653/v1/W15-4304
Vilar, D., Xu, J., D’Haro, LF., & Ney, H. (2006). Error Analysis of Machine Translation Output. In: International Conference on Language Resources and Evaluation, Genoa, Italy. pp. 697-702. DOI: https://doi.org/10.63317/453fcbexxngk
التنزيلات
منشور
كيفية الاقتباس
إصدار
القسم
الرخصة
الحقوق الفكرية (c) 2026 Fereydoon Rasouli
هذا العمل مرخص بموجب Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
يوافق المؤلفون الذين ينشرون مع هذه المجلة على الشروط التالية:
1. يحتفظ المؤلفون بحقوق الطبع والنشر ويمنحون حق النشر في المجلة الأولى مع العمل المرخص له في وقت واحد بموجب ترخيص Creative Commons Attribution [CC BY-NC-ND 4.0] الذي يسمح للآخرين بمشاركة العمل مع إقرار بتأليف المصنف والنشر الأولي في هذه المجلة.
2. يمكن للمؤلفين الدخول في ترتيبات تعاقدية منفصلة إضافية للتوزيع غير الحصري للنسخة المنشورة من المجلة عن العمل (على سبيل المثال ، نشرها في مستودع مؤسسي أو نشرها في كتاب) ، مع إقرارها الأولي نشر في هذه المجلة.
3. يُسمح للمؤلفين بتشجيعهم على نشر أعمالهم عبر الإنترنت (على سبيل المثال ، في المستودعات المؤسسية أو على موقع الويب الخاص بهم) قبل وأثناء عملية التقديم ، حيث يمكن أن تؤدي إلى تبادلات مثمرة ، وكذلك ذكر أقدم وأكبر للأعمال المنشورة ( انظر تأثير الوصول المفتوح).
