A Comparison of Human and Machine Translation of Health Promotion Materials for Public Health Practice: Time, Costs, and Quality

Abstract

Context

Most local public health departments serve limited English proficiency groups but lack sufficient resources to translate the health promotion materials that they produce into different languages. Machine translation (MT) with human postediting could fill this gap and work toward decreasing health disparities among non–English speakers.

Objectives

(1) To identify the time and costs associated with human translation (HT) of public health documents, (2) determine the time necessary for human postediting of MT, and (3) compare the quality of postedited MT and HT.

Design

A quality comparison of 25 MT and HT documents was performed with public health translators. The public health professionals involved were queried about the workflow, costs, and time for HT of 11 English public health documents over a 20-month period. Three recently translated documents of similar size and topic were then machine translated, the time for human postediting was recorded, and a blind quality analysis was performed.

Setting

Seattle/King County, Washington.

Participants

Public health professionals.

Main Outcome Measures

(1) Estimated times for various HT tasks; (2) observed postediting times for MT documents; (3) actual costs for HT; and (4) comparison of quality ratings for HT and MT.

Results

Human translation via local health department methods took 17 hours to 6 days. While HT postediting words per minute ranged from 1.58 to 5.88, MT plus human postediting words per minute ranged from 10 to 30. The cost of HT ranged from $130 to $1220; MT required no additional costs. A quality comparison by bilingual public health professionals showed that MT and HT were equivalently preferred.

Conclusions

MT with human postediting can reduce the time and costs of translating public health materials while maintaining quality similar to HT. In conjunction with postediting, MT could greatly improve the availability of multilingual public health materials.

The Need for Machine Translation in Public Health

As of 2011, there were an estimated 25.2 million people in the United States with limited English proficiency, defined as “persons who do not speak English as their primary language and who have a limited ability to read, write, speak or understand English.”^1,2 Title VI of the Civil Rights Act³ prohibits discrimination on the basis of race, color, or national origin, and Executive Order 13166⁴ expands this to include requirements that federally funded agencies provide “meaningful access” to their services. Despite this legal protection, obtaining language-congruent health information may be very difficult for non–English-speaking groups in the United States.^5–7

Language barriers can contribute to major health disparities even when socioeconomic status is accounted for.⁸ Immigrant populations tend to seek care later,⁹ have reduced quality of care,¹⁰ and fail to seek out certain kinds of care, such as mental health,¹¹ prenatal care,¹² and chronic care.^13–16 These disparities in health care can have short- and long-term consequences on mortality, morbidity, and quality of life.¹⁷

An important barrier to producing multilingual health materials is the time and costs required for language translation, given that there are an estimated 300 different languages spoken across the United States.¹⁸ One of the primary services of more than 3000 local health departments in the United States is to inform and educate the public¹⁹; however, health departments have limited funds for producing quality health promotion materials in English and even less funding to translate them into other languages.

Freely available online machine translation (MT) tools, such as Google Translate and Microsoft Translator, use statistical translation models trained from a large amount of text data in the relevant languages. These tools could be used to reduce translation costs. The quality of MT tools has improved steadily over the last few years, but these tools are not error free. Moreover, MT tools are often better at translating generic text, such as news articles, than translating domain-specific text containing specialized vocabulary. Thus, there is concern by many^20–23 that the quality of MT alone might not be sufficient. Evidence indicates that MT combined with human postediting (ie, the subsequent correction of MT output by human readers fluent in the source and target languages) produces a quality product at a lower cost.²⁴ In fact, the use of MT and human postediting has become a de facto standard for many commercial translation vendors.²⁵ Therefore, a possible solution to addressing financial constraints, and ensuring access to health information for the public, would be for public health departments to translate materials using MT software and then use native speakers, ideally with a background in public health, to postedit the documents for clarity and accuracy.

There has been very little published research on the use of MT in a health context. In a study of MT and Clinicaltrials.gov, Rosemblat et al²⁶ referred to the importance of MT for improving access to timely health information for the public. To our knowledge, our previous research is the only prior feasibility study of the use of MT in a public health setting.²⁷ In our pilot study, health promotion materials in use by public health departments were translated by a free MT program, then postedited by humans, and finally compared with fully human-translated documents. We showed that public health materials translated using MT with postediting were equivalent in quality to manual translations. However, it remained unclear whether MT could reduce the time and costs of producing translations for public health.

We report here on a study comparing the quality of postedited MT with manual translations of health promotion materials. In addition, we estimate the time and costs of producing commercially translated health materials and compare these numbers with the time and costs of producing materials using MT with human posteditors.

Methods

We conducted 2 phases of data collection to compare the time, costs, and quality associated with the translation of public health documents through MT versus human translation (HT). This study received exempt status from the University of Washington Institutional Review Board.

Phase 1: Postediting and quality rating

We collected 25 health promotion documents from a local health department, which the health department had previously translated from English to Spanish by the traditional process using hired translation vendors and internal bilingual staff. We then translated the English version of each document into Spanish using online MT software (Google Translate [translate.google.com]). Google Translate is free and translations are produced within seconds upon uploading the text.

We recruited native Spanish-speaking public health professionals (n = 12) and divided them into 2 groups: (1) posteditors, who postedited each machine-translated document; and (2) quality raters, who blindly compared the quality of the postedited MT and the corresponding HT.

Posteditors were presented with the original English document and the machine-translated document and instructed to correct for grammar and accuracy but not to make stylistic changes based on personal preference. The postediting sessions took place in a laboratory setting using a proprietary tool developed specifically for this process. Each postediting session consisted of 2 to 5 machine-translated documents in total, with each session designed around word count and not total documents. Each document was postedited independently by 2 posteditors. The time to postedit each document was recorded. Each posteditor received training on 1 document, the times for which were not retained for study purposes.

Quality ranking

For ease of analysis, we randomly selected 1 of the 2 postedited translations for each document, since the content of the documents was the same. We matched each of these with the human-translated documents collected from the health department. Two quality raters then compared the quality of the 2 translations. They were blindly presented with the machine/human-translated document pairs along with their English originals. Quality raters were instructed to review each translated document and indicate whether they preferred one translation to the other or felt they were equivalent. We also asked quality raters to briefly state the reasons for their preferences (Figure). The quality ranking analysis consisted of a total of 25 matched pairs of documents. The results of Phase 1 are included as Supplementary Digital Content (available at: http://links.lww.com/JPHMP/A52) to this article.

FIGURE.

Quality Comparison of HT Versus MT + Human Postediting

Phase 2: Time and task analysis of human translation

In previous research on translation workflow analysis in public health departments,²⁸ researchers conducted 34 semistructured qualitative interviews with employees involved in the translation process at Public Health–Seattle & King County and the Washington State Department of Health. On the basis of these interviews, researchers identified and mapped the discrete tasks involved in creating translated public health materials and verified these findings with staff through focus groups. The interviews also included questions about the type of document and the cost for the vendor translation. In most cases, a monolingual staff member would prepare the English document for translation (eg, removing slang, writing instructions to the translation vendor) and then send the document to a commercial translation vendor. This vendor would translate the document and perform an initial quality review. Upon receipt of the document, the monolingual staff member would typically (time allowing) send the translated document for final quality review (by a bilingual internal staff, a bilingual community provider, or another vendor).

To estimate the time spent on HT at the health department, we created a spreadsheet on the basis of our prior research on translation workflow, listing the primary tasks involved in the translation process and defining their start and end points (eg, “start—when you begin filling out the translation request form; end—when you receive a cost estimate from the translation vendor”). The translation process was defined as beginning the moment an English language document was deemed ready for translation and ending when the translation was deemed ready for public release. We asked health department employees to record the start and end times of each task for the translation projects they were involved in. Employees in 3 different programs tracked a total of 11 translation projects between February 2011 and October 2012.

Using the start and end times recorded by employees, we calculated hours and minutes for each task in each project using a 24-hour scale, including weekends and holidays (eg, if someone sent a document to a vendor at 5 PM on a Friday and received the translation back at 5 PM the next Monday, the time calculated for that task would be 72 hours). The rationale for a 24-hour time scale is that the project could not move to the next step until the task was completed. Therefore, in most cases, the physical task required fewer hours than calculated by the 24-hour scale. We rounded the start and end times provided to the nearest quarter hour. The times listed in Table 1 include the total time for all languages.

TABLE 1.

Types and Titles of 11 Human-Translated Documents at Public Health–Seattle & King County

Document No.	Document Title	Topic	Document Type
1	“Unusual Rise in Heroin Overdose Deaths Over Weekend”	Health alert regarding heroin overdoses	Press release and flyer
2	“King County Closed for All Species of Shellfish Harvesting”	Shellfish harvesting closure due to paralytic shellfish poisoning risk	Press release
3	“PrEP Q&A: Using HIV Drugs to Prevent HIV Infection”	HIV preexposure prophylaxis	Fact sheet
4	“Disaster Buddies!”	Emergency preparedness	Children’s coloring book
5	“HIV and STD Testing”	HIV and STD testing	Web page
6	“Refugee Communication Survey”	Cell phone use among refugees	Survey and information sheet
7	“Farmers Market Temporary Food Service Enforcement Procedure”	Enforcement of food permit rules	Fact sheet
8	“Electricity for Your Restaurant Will Not be Available June 5th/6th”	Safety and legal requirements during planned electrical outage	Fact sheet
9	“Planned Power Outage: Keeping Your Food Safe to Eat”	Food safety tips during planned electrical outage	Flyer
10	“Why HIV Testing Is Important for YOU!”	HIV testing	Fact sheet
11	“Prevent the Spread of Whooping Cough (Pertussis)”	Free pertussis vaccine clinic	Flyer

Abbreviation: STD, sexually transmitted disease.

On the basis of our qualitative workflow analysis, we grouped smaller individual tasks into larger “phases” and then calculated the time spent for the entire phase using the start time of the first task and the end time of the last task. The phases are as follows:

• Pretranslation activities: starting when the employees deemed the English text ready for translation and ending when the text was sent to the vendor for translation. The individual tasks include obtaining cost estimates from vendors if necessary, returning the signed estimate to the vendor, and, if required, filling out a requisition form.

• Translation activities: from the beginning to the end of the initial manual translation.

• Posttranslation activities: starting when the staff began contacting potential quality assurance reviewers and ending when the quality assurance reviewers returned their edits.^*

When internal staff conducted postediting activities (as opposed to commercial translation vendors or community partners conducting this step), researchers asked these staff to estimate how much time they spent postediting the document, to get the most accurate estimate of actual time spent. We then calculated costs by dividing the total cost of the vendor translation portion by the number of English words in the original document, not adjusting for inflation. Words per minute (WPM) are calculated using the total time and number of English words.

We chose 3 of the 11 documents from the time and task analysis, based on the type of document, language, and length (<1500 words) to translate into Spanish using MT. Posteditors conducted timed postediting of the translations using the same process as described in Phase 1. Quality raters were then recruited to blindly rate the postedited MTs against their Spanish HT counterparts, in terms of preference as described in Phase 1. Each of these preferences was then tallied using a voting-style system where each reviewer’s preference would account for 1 “vote” for that type of translation. If reviewers declared the translations equivalent, it counted as 1 vote for each.

Results

Health department HT time analysis: Phase 2 results

As shown in Table 2, it took a minimum of 2 days, and up to 35 days, for a document to go through the entire translation process, with an average time of 15 days. These times included pretranslation activity and quality assurance activities. These are tasks that would remain if MT were used. The manual portion of the translation process, which was performed by commercial translation vendors contracted by the public health departments in this analysis, took anywhere from 17 hours (rush orders) to 6 days, with an average of about 3 days. These numbers reflect the total time for all translation activities required to complete a given translated document. This does not imply that the vendor or posteditor was working on the document continuously for that long. Since the document cannot be publicly distributed until completed, however, assessing the total time is relevant, especially in an emergency or crisis situation. The costs of HT, based on invoices, ranged from $130 up to $1220. The costs varied, depending on the word count and the number of languages involved (Table 2). The estimated times for postediting the human-translated documents ranged from 1 to 12 hours and WPM ranged from 1.58 to 5.88, with an average time of 3.30 WPM.

TABLE 2.

Time Analysis of Current Health Promotion Document Translation Process at Public Health–Seattle & King County

Document No.	English Word Count	Languages	Pretranslation Activitiesa	Translation Timeb	Posttranslation Activitiesc	Total Hours (Days)	Words per Minute (Total)	Total Costd (Cost per Word)
1 (rush)	706	Spanish, Chinese, Vietnamese	24.5	24	92	140.5 (5)	0.08	$313 ($0.44)
2 (rush)	292	Spanish, Chinese, Vietnamese, Khmer, Lao	138.25	25.75	426	590 (19)	0.008	$304 ($1.20)
3e	1141	Spanish	119.25	42.75	96	258 (33)	0.07	$228 ($1.04)
4	530	Spanish, Chinese, Vietnamese	46.5	140.25	69	255.75 (35)	0.03	$492 ($0.93)
5	2136	Spanish	17.25	145	215.75	378 (19)	0.09	$430 ($0.20)
6 (rush)	706	Russian, Arabic, Farsi, Amharic, Tigrinya, Somali, Burmese, Karen, Nepali	99.5	138.5	237.75	475.75 (20)	0.02	$1220 ($0.20)
7	719	Spanish	22.5	99	333.5	455 (13)	0.03	$144 ($1.73)
8 (rush)	450	Chinese	1.5	17.75	4.25	23.5 (2)	0.32	$155 ($0.20)
9	150	Chinese, Vietnamese	45.75	25.5	112.5	183.75 (9)	0.01	$130 ($0.34)
10	1022	Swahili	16.25	141.75	102	260 (11)	0.07	$309 ($0.87)
11	81	Spanish, Russian, Vietnamese, Chinese, Khmer, Amharic	24.75	72.75	N/Af	97.5 (4)	0.01	$300 ($0.30)
Average	721.18		50.55	79.36	153.53	238.43 (15.45)	0.06	$365.91 ($0.67)

^aObtaining cost estimates, returning signed estimates, requisitions; 24-hour time.

^bTime at vendor.

^cIncludes postediting by the vendor, postediting quality assurance, and when applicable, final editing by the internal staff. Total time from initial request to delivery. Actual time estimates for postediting task in Table 2.

^dIncludes charges for rush and all languages.

^eQuality of this document was poor, so payment was withheld.

^fNo postediting; the document was received from the vendor and distributed as is.

The calculated postediting times following MT were 2 to 10 times faster than the estimates of the postediting times following HT (Table 3). Postediting of MT documents took between 28 and 85 minutes and WPM ranged from 10.32 to 13.40, with an average time of 11.76 WPM.

TABLE 3.

Phase 2 Comparison of HT (Estimated) and MT (Actual) Postediting Times

Document No.	Word Count (English)	HT QA/Postediting, min	HT QA/Postediting WPM	MT Postediting, min	MT Postediting WPM	Reviewer Preferences
1 (rush)	706	120	5.88	61.10	11.55	MT/HTa
2 (rush)	292	120	2.43	28.27	10.32	HT/MT
3	1141	720	1.58	85.14	13.40	HT/ =
Average	713	320	3.30	58.17	11.76
Total MT votes						3
Total HT votes						4

Abbreviations: HT, human translation; MT, machine translation; QA, quality assurance; WPM, words per minute.

^aMT refers to a reviewer preference for the machine-translated version; HT refers to a reviewer preference for the human-translated version; and “=” refers to a reviewer preference for neither version. In our analysis, this means 1 “vote” is added to the tally for both MT and HT.

Abbreviations: HT, human translation; MT, machine translation.

Quality analysis: Phase 1 results

Supplemental Digital Content Table 2 (available at http://links.lww.com/JPHMP/A52) presents the results of Phase 1. The quality analysis for Phase 2 is given in Table 3. The quality analysis was performed only with Spanish documents.

Reviewer preferences were combined from both Phase 1 and Phase 2, which resulted in 36 “votes” for HT and 37 “votes” for MT. These numbers are almost exactly the same, with a 1-vote preference for MT.

Discussion

This study presents an analysis of time, cost, and quality comparisons of machine-translated/human postedited public health documents with those translated completely by humans. It is clear from other research that using MT alone will not guarantee that a given translation will communicate the intended message.²⁹ In quality comparisons of HT versus MT, however, MT with human postediting has comparable quality (as judged by bilingual public health translators) with human-translated documents. In addition, MT results in average postediting times of approximately 12 WPM whereas HT of the same documents results in much slower average postediting times of approximately 3 WPM (Table 3).

Businesses have begun to harness the power of MT for the increasingly global economy.³⁰ They recognize that society is progressively more mobile and more international and that multiple languages must be accommodated in every aspect of public-facing communication.³¹ While MT is not the only solution to this problem,³² it is one of the most effective and least costly. By moving toward the model of MT with human postediting, the goals of ensuring access and lowering costs while also maintaining the quality and accuracy that are vital to public health communication can be achieved. The most obvious cost savings with MT arise from eliminating the cost of paying an outside vendor or in-house translator for the initial HT. The cost savings generated from eliminating the cost of translation services could result in more documents being translated on a yearly basis. On the basis of costs estimated in this study, these documents cost an average of $365 each to translate. Postediting a machine-translated document took an average of 1 hour. By using MT for this initial translation and assuming an average hourly wage of $18,^† 20 documents could be postedited for the cost of 1 vendor-translated document. Since access to language-specific health information has been shown to reduce health disparities,³³ increasing the number of translated documents would greatly benefit limited English proficiency populations.

There are limitations to this study. The cost analysis is minimal and not intended to represent a complete cost-effectiveness analysis but rather a preliminary indication of cost reductions that can be achieved through MT with postediting. The number and type of documents analyzed may not be completely representative of the yearly output of public health departments. The time for postediting the human-translated documents was estimated on the basis of self-reports rather than observed. Another limitation is that the postediting times for machine-translated documents were determined in a controlled setting, whereas in the field, these documents will be postedited as part of a normal workflow. The quality analysis was performed with public health professionals involved in translation activities; however, they were not performed using the target audiences.

Conclusion

By eliminating the time taken to perform HT, and using local resources to postedit documents, MT can both lower costs and provide additional job demand for bilingual public health staff, without compromising translation quality. The savings in time and money may allow for an increased number of documents to be translated in a year while freeing up translators to focus on postediting tasks, thus reducing the time it takes to bring vital public health messages to limited English proficiency groups.