I find these types of "match human performance" claims to be ridiculous, especially when it comes to Chinese -> English translations. Translation is both an art and a science, requiring nuanced understanding of the languages, cultures, and context. It also demands quite a bit of creativity. No translation tool I've tried has come even close to matching human performance of a good human translator, including microsoft's tools. AI will need to reach the point where its understanding of language, culture, context, and creative ability matches that of humans to truly be capable of "human performance" in translation.
After reading the paper, my takeaway is that humans aren't really very good at translation either. None of the methods scores higher than 70% in the evaluation and that includes several different human translations (whose performance varies greatly depending on how they were sourced). So while matching the quality of the average human translator is a great milestone, there's still lots of room to improve.
Most humans who attempt to translate are not actually translators in the proper sense. Specifically, they are not fully competent in both the source and target language and usually have no formal training in translation. Translation is hard, but there are competent people out there who do it extremely well. Sadly, as an industry it's not taken as seriously as it should be, and most of the people who are actually doing translations do not have the appropriate skill set.
I can offer some insight on this, as I've dealt with translations of papers before (Chinese to English). There have been countless times where I've been asked to check the translation of a paper and ended up retranslating it completely. I was often curious as to why the original translation was bad - it was typically either of the following two reasons: 1. The so-called translator was just an overworked research assistant who knows a bit of English. 2. The translation was outsourced to a dodgy company that spends more on marketing than on their translators, usually hiring college grads who did English-related majors.
Some people without a specific cultural understanding happen to translate the content perfectly well, too. Experience and constant mastering of translational skills is what counts. If experienced and skilled translators are cooperating in developing process, it could be more than possible to get to a human performance.
