Assessing Translation Ability through Vocabulary Ability Assessment

1 Introduction

Translation ability is known as one of the most difficult language abilities to measure [Stansfield et al., 1992; Stubbe, 2014]. Such difficulty is caused by a variety of language proficiencies that comprise translation ability, for example, reading skills in the source language and writing skills in the target language. A typical method of measuring translation ability involves translation tests, in which a translator is asked to translate given sentences and professional evaluators are asked to grade the translation results. This method precisely assesses translation ability; however, hiring the professional evaluators is expensive and imposes a heavy burden on the translators.

⇤This work was performed for the National Institute of Information and Communications Technology (NICT) when the first author was employed by NICT. He currently works for the National Institute of Advanced Industrial Science and Technology (AIST).

There is a growing need for easy-to-use methods for measuring translation ability due to the recent expansion of crowdsourcing platforms that allow the hiring of nonprofessional translators. Because there is no guarantee that all non-professional translators on a crowdsourcing platform have sufficient translation skills, requesters who are willing to post translation jobs need to assess the translators' ability in order to obtain the services of high-quality translations and make good use of their budget.

In contrast with translation ability, vocabulary ability is considered a convenient measure for assessing language skills [Nation and Beglar, 2007; Meara, 2010]. For example, we can evaluate learners' vocabulary ability by using a simple test with which learners are asked to select the correct meaning from multiple choices. Such vocabulary tests have already been applied to select highly skilled second language learners [Nation, 2006] and for efficient student class placement [Read and Chapelle, 2001].

In this paper, we propose a convenient method for measuring translation ability. Our key idea is to use the results of low-cost vocabulary tests for translation ability assessment so that we do not need to rely on professional evaluators. Our method first obtains information about translators by making them take a translation test. In this test, the translators are asked to translate several sentences as well as to tell if they know the meaning of each word in the source language sentences. The second question is designed as a self-report vocabulary test and we use the results of the vocabulary test, called vocabulary information of translators, to estimate translation ability. Additionally, we can consider a case in which we can afford to hire professional evaluators to evaluate a small number of translations obtained from the translation test. Our method can be incorporated with such groundtruth information to estimate translation ability more precisely.

We introduce a probabilistic model for estimating translation ability. Our model is built based on an assumption that a translator with high translation ability is likely to have high vocabulary ability, and a translator with high vocabulary ability is likely to know the meaning of difficult words. Given the vocabulary information of translators, word difficulties and vocabulary and translation abilities are estimated using the maximum a posteriori (MAP) inference. Word features are incorporated for estimating word difficulties, therefore, when a new source language sentence is given, our model can estimate the probability of a translator providing an acceptable translation for the sentence. Because we apply the self-report vocabulary test, one may be concerned that over-confident or unreliable translators are likely to answer that they know the meaning of most of the words. Our model incorporates a parameter representing unreliability of a translator to address such cases.

We conducted two experiments indicating that the proposed method accurately estimates translation ability. The first experiment showed that the proposed method selected good translators in a realistic setting of crowdsourcing translation. The results will immediately lead to an application to reduce the cost of crowdsourcing translation. The second experiment showed that estimated vocabulary and translation ability correlate, which validates our method.

The contributions of this paper are as follows:

• ÷ This is the first study addressing easy translation ability measurement using vocabulary measurement.
• ÷ From investigating the obtained model, we argue that translation ability correlates with vocabulary ability.
• ÷ Our method significantly reduces the cost in selecting good translators in a realistic setting of crowdsourcing translation.

2 Problem setting

Let S be a given set of source language sentences for translation tests. Each source sentence s 2 S is defined as an enumeration of words s = ¶ws1↪ ws2↪ · · · ♦. Each word w in a source language is associated with an N-dimensional feature vector w 2 RN and each sentence s is associated with an M-dimensional feature vector s 2 RM .

When a translator k translates a test sentence, she is also asked to tell if she knows the meaning of each word in the sentence. If k answers that she knows the meaning of word w, then we set ykw = 1; otherwise, ykw = 0. We assume that ykw is given for all k 2 K↪ s 2 Sk, and w 2 s.

Let the set of translators be K. Each translator k 2 K takes a translation test and provides translations for a set of source language sentences Sk. The quality of a translation given by translator k for source language sentence s is then graded by a professional evaluator. Let zks = 1 if the translation is judged as acceptable and zks = 0 otherwise. We call zks the translation quality label. We consider two settings: semi-supervised and unsupervised. In the semi-supervised setting, we assume that some test translations is graded by professional evaluators while none are evaluated in the unsupervised setting, that is, zks is not given for all k 2 K and s 2 Sk.

Given ¶s♦s2S, ¶w♦w2s↪s2S, ¶ykw♦k2K↪s2Sk↪w2s, and ¶zks♦k↪s , our goal was to build a translation quality predictor Pr (zks0 = 1♣s0 ↪ ¶w♦w2s0 ↪ ¶ykw♦w2s0 ), where s0 ◁2 S is a new source language sentence. When translator k provides her vocabulary knowledge for every word w 2 s0 , the predictor predicts whether the quality of translation given by k for s0 will be acceptable or not, even when k does not actually translate the sentence. Notations are listed in Table 1.

3 Vocabulary Knowledge-based Translation Quality Predictor (VKTQP)

3.1 Translation AbilityModel

Our model called Vocabulary Knowledge-based Translation Quality Predictor (VKTQP) is composed of two probabilistic models: translation ability model and vocabulary ability model. For building the translation ability model, we assume that translation quality varies according to translation ability of a translator, and translation difficulty of a source language sentence. The Rasch model [Rasch, 1960; Baker and Kim, 2004] holds the key idea of prediction regarding estimating ability and difficulty. This is a simple and standard model for representing human ability and task difficulty in education and psychology. We model the translation ability of translator k as k and translation difficulty of source language sentence s as ⌫s.

Using the Rasch model, the probability that translator k acceptably translates source language sentence s is given as

where σ denotes the logistic sigmoid function, i.e., for a real number a, σ(a) ⌘ 1 exp(-a)+1 . Being binary, zks is predicted to be 1 if translation ability k outperforms difficulty ⌫s; otherwise, 0. Note that the Rasch model is a special case of logistic regression.

We further model sentence difficulty ⌫s as the inner product of sentence feature vector s and weight parameter ⌧ , that is, ⌫s = ⌧ >s. Incorporating sentence feature vectors allows our model to estimate the probability of a translator providing an acceptable translation for a new sentence. The translation ability model is illustrated on the left side of Figure 1.

3.2 Vocabulary AbilityModel

We assume that the probability of a translator knowing the meaning of a word is dependent on the vocabulary ability of the translator and difficulty of the word. We model the vocabulary ability of translator k as ✓k and difficulty of word w as θw. Because we apply the self-report vocabulary test, one may be concerned that over-confident or unreliable translators are likely to answer that they know the meaning of most of the words. Thus, we introduce a parameter φk to model the unreliability of translator k. We assume that, if translator k provides an acceptable translation for sentence s, the probability of k answering that she knows the meaning of word w 2 s depends on vocabulary ability ✓k. Otherwise, we model the probability by using φk. We apply the Rasch model and represent the probabilities, namely,

where w 2 s.

We also incorporate the word feature vectors into the vocabulary ability model by modeling the difficulty of w as θw = ⇡>w, where ⇡ denotes a weight parameter. The vocabulary ability model is illustrated on the right side of Figure 1.

Overall, the probability described with the VKTQP model

4 Parameter Estimation

4.1 Priors

We now define the priors on the parameters listed in Table 2. Since we have many parameters, priors are important not only for preventing overfitting but also for describing relations between parameters.

First, for preventing overfitting, we define the priors for ⇡, ⌧ , ✓k, and φk. In Table 2, N denotes the probability density function of the Gaussian distribution. These priors are set to have 0 means so that they can prevent overfitting by regularizing parameters towards 0.

Second, we define two versions of the VKTQP models called VKTQP-I and VKTQP-D by changing the prior for VKTQP-D Pr [ k] ⌘N (✓k↪ ⇠-1 )

translation ability k. These models have a direct effect on the prediction of translation quality (Table 2). The VKTQP-I model simply uses the Gaussian priors with a 0 mean for preventing overfitting for each parameter. Unlike VKTQP-I, the prior for k in VKTQP-D is set to ✓k. This prior enforces the assumption that translation ability will correlate with vocabulary ability by regularizing translation ability k towards vocabulary ability ✓k.

4.2 Parameter Estimation of Supervised Setting

We use the MAP inference for estimating the model parameters. The posterior distribution is written as

where the function "Priors" returns the products of the priors of each parameter defined in the previous section.

Instead of maximizing (5), we minimize its negative log of (5) for the MAP inference. Although we only consider the unsupervised and semi-supervised settings, it would be worth mentioning that when we observe zks for all k and s 2 Sk (i.e., in the supervised setting), the negative log of the posterior is convex with respect to all parameters. This means that we can obtain the optimal parameters. This convexity is directly derived from that of the negative log of the posterior function of the L2-normalized logistic regression.

4.3 Parameter Estimation in Unsupervised and Semi-supervised Settings

In the unsupervised and semi-supervised settings, the set of translation qualities ¶zks♦ contain unobserved variables. Thus, we use the expectation-maximization (EM) algorithm [Dempster and Rubin, 1977] for the MAP inference of the parameters. Once the unsupervised setting is derived, the semi-supervised setting can be easily derived by fixing the observed part of ¶zks♦ to the observed values.

For simplifying the equations, we introduce the following notations. ↵kw denotes the probability that k knows w given that zks is acceptable, i.e., ↵kw ⌘ Pr [ykw = 1 ♣ zks = 1] = σ (✓k -θw); βkw denotes the probability that k knows w given that zks is not acceptable, i.e., βkw ⌘ Pr [ykw = 1 ♣ zks = 0] = σ (φk -θw); and γks denotes the probability of obtaining an acceptable quality when k translates s, i.e., γks ⌘ Pr [zks = 1] = σ ( k - ⌫s).

In the E-step, we evaluate the following formula to update zks.

After the E-step, we update zks with new z0ks , which is defined as z0ks ⌘ Pr ⇥zks = 1 ♣ ¶ykw♦w2s ⇤.

In the M-step, we maximize the Q function, which has the same form as the likelihood function L in (5), except that zks is replaced with the updated value z0ks . Once z0ks is replaced and fixed, the negative log of the Q function is convex with respect to parameters, as is the case with the negative log likelihood. Thus, the Q function can be globally optimized in the M-step.

5 Evaluation

5.1 Dataset

We targeted English-to-Japanese translation for our experiments. We used the Japanese-English Bilingual Corpus of Wikipedia's Kyoto Articles 1 . We randomly selected 104 English sentences that have more than ten words and their corresponding translations given in Japanese.

For collecting a dataset, we used Lancers 2 crowdsourcing service, one of the largest crowdsourcing services in Japan. Most of the translators working in this service were assumed to be native Japanese speakers.

We hired 55 translators and obtained 1,498 English-toJapanese translations. The translation cost was 10 JPY (approximately 0.1 USD) for each sentence. One sentence was translated by 14▷4 translators on average. One translator translated 27▷2 sentences on average.

Before translators engaged in the English-to-Japanese translation tasks, we asked the translators to report the words unfamiliar to them by clicking on them. Then, the translators started their translation tasks. The use of dictionaries was allowed (but not required) during the tasks.

The translation quality used for the evaluation was manually judged by two annotators, who were native speakers of the target language (Japanese) and fluent in the source language (English). The translation quality was judged using a 5-point scale, where 5 is acceptable and the other 4 points are unacceptable translation mistakes. The kappa coefficient 3 on the acceptability of the translations between the two annotators was 0▷619, which is a "significant agreement" [Landis and Koch, 1977; Mihalcea and Chklovski, 2004].

1 http://alaginrc.nict.go.jp/WikiCorpus/index E.html

2 http://www.lancers.jp

3 We used the Python NLTK library for calculating the kappa coefficient. http://nltk.org/

5.2 Features

Throughout the experiments, all our VKTQP models used the same sentence features designed to allow us to easily apply our method to other source languages. Table 3 lists the sentence features.

The word features use three corpora and one difficulty index (Table 4). These features were reported to be helpful for accurately predicting whether a learner knows a word [Ehara et al., 2012; 2013; 2014].

5.3 Experiments

Table 5 lists the models that we compared in our experiments. VKTQP-I and VKTQP-D are our models.

We used five-fold nested cross validation throughout the experiments. All the models have hyper-parameters. We conducted grid-search over the validation sets for tuning the hyper-parameters. The hyper-parameters of all the models were chosen from 10-3▷0 , 10-2▷4 , 10-1▷8 , 10-1▷2 , 10-0▷6 , 100▷0 , 100▷6 , 101▷2 , 101▷8 , 102▷4 , and 103▷0 . After tuning the hyper-parameters over the validation set, we measured the performance on the test set, which is disjoint from the validation and training sets.

5.4 Cost Reduction Rate: EvaluationMeasure for Assigning Tasks

As a realistic evaluation measure in crowdsourcing translation, we define a measure called the cost reduction rate (CRR). Cost indicates the number of translation requests of Table 6: Cost reduction rate of each model against training size. In each column, bold value denotes best model and underlined value denotes second best. Asterisks denote that best model significantly outperformed second best model. (**:p < ▷01, *:p < ▷10, Wilcoxon test was used).

4 http://www.statmt.org/moses/?n=moses.baseline

5 https://catalog.ldc.upenn.edu/LDC2006T13

6 http://corpus.byu.edu/coca/

7 http://www.alc.co.jp/vocgram/article/svl/

translators to translate a new sentence. The CRR shows a decrease in the number of requests by the predictor's assignments compared to random assignments.

Suppose we have a new sentence to translate, a set of available translators, and a predictor. First, all the available translators are candidate translators. The predictor ranks the translators by its predicted translation quality. We select the best translator from the predictor's ranking and request translation of the sentence from him/her. Then, we evaluate the translator's translation quality. If the quality is not acceptable, we discount the translator from the group of candidates and request the next translator to translate the same sentence. We repeat these steps until we encounter a translator who returns an acceptable translation. The number of translators until then is defined as #TR.

The CRR is defined in (6). Larger CRR values mean larger reduction or better performance. The numerator of (6) is the number of requests by the predictor until an acceptable translation is obtained. The denominator of (6) is the expected number of requests by random predictors. In other words, the denominator is the average number of requests until an acceptable translation is obtained when choosing translators randomly.

For example, suppose only 1 in 3 translators can translate a given source sentence acceptably. If we randomly take translators from the bag of 3 without replacement, 1◁3 ⇤ (1 + 2 + 3) = 2 translators are necessary to obtain one good translation on average. If we encounter a good translator first, the CRR is 1 - 1◁2 = 1◁2, which halves the cost. If second, the CRR is 1 - 2◁2 = 0, which provides no improvement. If third, the CRR is 1 - 3◁2 = -1◁2, which increases the cost.

If no translator can translate the sentence acceptably, we define CRR as 0. If all translators can translate the sentence acceptably, CRR also becomes 0. This means that no improvement can be made by any predictor.

5.5 Evaluation based on Cost Reduction Rate

Table 6 shows the CRR values of each model. Each column shows the size of the training data. The column with 0 training data shows the experiments in an unsupervised setting, where no translation quality label was provided. We used 5fold nested cross-validation for this evaluation as well.

In Table 6, in an unsupervised setting where the training size is 0, we can easily see that VKTQP-D reduced cost by 32▷7%. This is our main result. The reason that translation ability was difficult to estimate in previous studies lies in the high cost for obtaining translation quality labels. The reduction in cost shows that VKTQP-D estimated translation ability accurately through vocabulary ability using no translation quality labels. The estimation was so accurate that it would actually reduce the cost of crowdsourcing translation by selecting good translators.

The next notable observation is that VKTQP-D consistently outperformed VKTQP-I. As explained earlier, VKTQP-D differs from VKTQP-I in that with VKTQPD, it is assumed that translation ability is close to vocabulary ability. Thus, this observation demonstrates that this assumption is true and vocabulary tests are helpful for predicting a translator's translation ability. This difference between VKTQP-Ds and VKTQP-Is was statistically significant where the training size was 0, 10, and 55.

The SVM+AVGWF constantly outperformed SVM and LR+AVGWF constantly outperformed LR. The former, i.e., those with "AvgWF" in their names, use vocabulary features while the latter do not. This result shows that the use of vocabulary features improves predictive performance even if the same algorithms, i.e., SVM and logistic regression, are used.

5.6 Analysis of Abilities

The previous section supports the assumption that making translation ability close to vocabulary ability ✓ achieves better performance. This section further verifies this assumption by directly observing the translation ability and vocabulary ability ✓.

The values of translation ability are plotted against those of vocabulary ability ✓. In each sub-figure of Figure 2, each dot represents each translator. The dots were regressed by linear regression. The regression results are shown in the caption of each sub-figure.

In Figure 2, the upper figures, namely a) and b), show the results when the ability parameters were estimated without any translation quality labels, whereas c) and d) show those when the ability parameters were estimated using all 896 translation quality labels.

When the training size (TS) was 0 in Figure 2, we can clearly see that, in a), there is no correlation between translation ability and vocabulary ability ✓. In contrast, in b), there is a clear correlation between translation ability and vocabulary ability ✓. This is because VKTQP-D used in b) is designed to make translation ability close to vocabulary ability, whereas VKTQP-I is not designed to do so. Note that VKTQP-D greatly outperformed VKTQP-I in this unsupervised setting, as stated in the previous section.

An interesting observation is shown in the lower figures of Figure 2, when the TS was 896. Although VKTQP-I is not designed to make translation ability and vocabulary ability ✓ close, these two abilities estimated with VKTQP-I in c) clearly correlated. In fact, this correlation is statistically significant. These results verify the assumption that translation ability and vocabulary ability naturally correlate. Since this is directly assumed with VKTQP-D and this assumption is actually true, VKTQP-D outperforms VKTQP-I.

When the TS was 896, the dots of VKTQP-D were more scattered in d) compared to those in b) while the dots in d) still significantly correlated. The reason of this is presumably due to the noise in the translation quality labels.

6 RelatedWork

The studies [Malakoff, 1992] and [Stansfield et al., 1992] are seminal in translation ability measurement. They involved measuring translation ability in realistic job-related settings in Spanish-to-English translation. While they focused on professional translations or translation by bilingual people, they did not address non-professional translations by secondlanguage learners.

Stubbe [2014] recently addressed the relationship between translation ability and vocabulary ability at the word level. However, he did not address sentence-level translation nor propose a method for measuring translation ability by using vocabulary ability. Gao et al. [2015] recently addressed reducing costs in crowdsourcing translations. However, they measured translation ability by actually letting translators translate some sentences for a trial. Thus, they did not mention a method for reducing the cost of measuring translation ability. Moreover, they did not address vocabulary measurement since it was not their focus.

Previous related studies focused on estimating the quality of given translations, while we aimed to predict translation quality without translations. Quality Estimation (QE) is an example of these studies, which aims to estimate the quality of given translations. Quality estimation is typically formulated as a supervised regression problem [de Souza et al., 2014]. It also involves crowdsourcing for collecting supervision [Zaidan and Callison-Burch, 2009; 2010] and has been extensively studied, e.g., [Callison-Burch et al., 2012; Bojar et al., 2013]. Quality estimation mainly targets machinetranslated texts. In contrast, application of QE to human translation is very recent [Specia and Shah, 2014]. Although in that study they used the verb "predict", their approach, unlike ours, requires written translations for estimating quality. For clarity, we use "predict" only when translations are not given and "estimate" only when they are.

Regarding our use of the self-reporting approach for measuring vocabulary, many educational experts support selfreport vocabulary measurement, as surveyed by Nation [2006]. Typical multiple-choice measurement, in which learners are asked to choose the correct meaning of a word from a set of multiple choices, depends heavily on the creation of "non-correct" choices. Therefore, those who support self-report vocabulary measurement claim that it is more reliable than multiple-choice measurement [Meara, 2010].

7 Conclusion

Translation ability is difficult and costly to measure compared to vocabulary ability, though there is a great need to measure it. This paper is the first to propose a method for measuring translation ability through the measurement of vocabulary ability and requires little or no translation.

We quantitatively evaluated our method in a realistic setting where the measured translation ability was used to select good translators for crowdsourcing translation. Our method exhibited significantly high accuracy in selecting translators who produce acceptable translations.

Future work includes support for other types of information, such as editing information, to measure translators' translation ability.

Acknowledgments

This work was supported by JSPS KAKENHI Grant Number 26730115 and 15K16059.