The study aims at performing an error analysis as well as providing an evaluation of the quality of neural machine translation (NMT) represented by Google Translate when translating relative clauses. The study uses two test suites are composed of sentences that contain relative clauses. The first test suite composes of 108 pair sentences that are translated from English to Arabic whereas the second composes of 72 Arabic sentences that are translated into English. Errors annotation is performed by 6 professional annotators. The study presents a list of the annotated errors divided into accuracy and fluency errors that occur based on MQM. Manual evaluation is also performed by the six professionals along with a BLEU automatic evaluation using the Tilde Me platform. The results show that fluency errors are more frequent than accuracy errors. They also show that the frequency of errors and MT quality when translating from English into Arabic is lower than the frequency of errors and MT quality when translating from Arabic into English is also presented. Based on the performed error analysis and both manual and automatic evaluation, it is pointed out that the gap between MT and professional human translation is still large.
fluency, accuracy, error analysis, evaluation, test suite.
This work is licensed under a Creative Commons Attribution 4.0 International License.
JSS publishes Open Access articles under the Creative Commons Attribution (CC BY) license. If author(s) submit their article for consideration by JSS, they agree to have the CC BY license applied to their work, which means that it may be reused in any form provided that the author (s) and the journal are properly cited. Under this license, author(s) also preserve the right of reusing the content of their article provided that they cite the JSS.