Modeling workflow to design machine translation applications for public health practice

Abstract

Objective

Provide a detailed understanding of the information workflow processes related to translating health promotion materials for limited English proficiency individuals in order to inform the design of context-driven machine translation (MT) tools for public health (PH).

Materials and Methods

We applied a cognitive work analysis framework to investigate the translation information workflow processes of two large health departments in Washington State. Researchers conducted interviews, performed a task analysis, and validated results with PH professionals to model translation workflow and identify functional requirements for a translation system for PH.

Results

The study resulted in a detailed description of work related to translation of PH materials, an information workflow diagram, and a description of attitudes towards MT technology. We identified a number of themes that hold design implications for incorporating MT in PH translation practice. A PH translation tool prototype was designed based on these findings.

Discussion

This study underscores the importance of understanding the work context and information workflow for which systems will be designed. Based on themes and translation information workflow processes, we identified key design guidelines for incorporating MT into PH translation work. Primary amongst these is that MT should be followed by human review for translations to be of high quality and for the technology to be adopted into practice.

Counclusion

The time and costs of creating multilingual health promotion materials are barriers to translation. PH personnel were interested in MT's potential to improve access to low-cost translated PH materials, but expressed concerns about ensuring quality. We outline design considerations and a potential machine translation tool to best fit MT systems into PH practice.

1. Introduction

Advances in machine translation (MT) technology have greatly increased its potential for improving access to multilingual health materials for limited English proficiency populations (LEP) in the US. However, publicly available MT systems (e.g., Google Translate, Bing Translator) traditionally perform poorly in the domain of public health (PH), an arena where clear and accurate communication of health messages to broad populations is crucial.[1,2,3] The need for multilingual health promotion materials is clear. According to the American Community Survey, 381 different languages are spoken in the US.[4] Nearly 20% of the population over 5 years of age speaks a language other than English at home, and 44% of those are categorized as LEP,[4] which is defined as having a primary language other than English and a limited ability to read, write or understand English.[5] Minorities with LEP are at a higher risk for health disparities than English-speaking minorities, and have less access to health education, less preventive screening, and report poorer health statuses than English-speaking minority groups.[6-8] Federal regulations require that individuals have equal access to health information and thus all states have enacted laws to ensure that language barriers do not inhibit access to health and social services.[9] Despite these legal requirements, the cost and resources necessary to translate materials into other languages limits PH professionals' ability to produce them. Recent advances in MT technology have greatly increased the potential of using MT to facilitate and accelerate the production of multilingual health materials. However, a crucial prerequisite for the adoption of new technology is its smooth integration into existing workflow practices. For this reason, our team explored current PH translation processes to better understand how MT technology could successfully be applied to PH settings. Workflow studies were conducted using a cognitive work analysis framework to identify functional requirements for engineering software designed to improve the production of multilingual health education materials for PH practice.[10,11] Given the substantial financial constraints, competing needs, and diverse services of the more than 3,000 local health departments in the United States, as well as the local rather than central funding streams, our goal was to design a system with freely available software tools that could be easily adopted in a variety of PH settings without expensive subscription costs and a need for negotiating ongoing contracts. These studies provided the foundation for designing a prototype collaborative machine translation tool called PHAST for use in PH practice.

1.1 Machine translation and its potential use in public health settings

Over the last decade, MT technology has been greatly improved; it is now used by language providers and various companies, as well as by governmental and non-profit organizations. With statistical MT - which is presently the most promising approach - models are trained automatically using large bodies of parallel text or text in the source language paired with its translation in the target language.[12] A more detailed introduction to statistical MT technology can be found in Cancedda et al.[13]

Statistical MT has been shown to be more powerful than older approaches like rule-based MT and translation memories, which rely on sophisticated linguistic analyses or on large databases of stored examples that need to match the input text. Therefore, the best statistical MT systems typically outperform rule-based or example-based MT systems, even on specialized technical text (see e.g., [14]). However, it is generally recognized in the MT user community that MT engines still produce flawed output yet are often sufficient for text classification or gisting (creating a rough translation) tasks. More importantly, MT is most often used in combination with post-editing by a human reader, i.e., first-pass machine translations are manually corrected in a second pass by a human post-editor. This process has been shown to be significantly faster and less costly than having human translators create translations in an entirely manual process.[15] In the health domain, MT has been used for years by organizations such as the Pan-American Health Organization and the Canadian-UN Global Network for Public Health Intelligence.[16,17] By contrast, MT has only recently been introduced into local and regional health care and PH settings in the US.[18,19] Despite the growing need for multilingual health materials and the potential of MT to facilitate production of, and access to, these materials, there exists widespread skepticism among professionals about MT's ability to produce quality translations of health information.[20] There are many reasons for this. Firstly broad range of subject matters, diverse audiences, and criticality of clear communication of health messages make the PH domain particularly challenging with respect to MT. In addition, some concerns may derive from the perceived difficulty of language in public health documents. Work on natural language processing (NLP) methods for the related problem in biomedical or clinical text processing (see [21-24] for overviews) has shown that extensive adaptation of NLP tools is required in order to produce usable results; these include, among others, customized tools for part-of-speech tagging, named-entity recognition, event detection, and relation extraction.[25-28]

In our past work we have studied the linguistic characteristics of typical PH documents intended for translation in a variety of public health departments.[29,30] These included web pages, flyers, and hand-outs on a variety of PH topics (e.g., vaccinations, specific diseases, food safety, etc.). These materials are typically intended for the general public and are required to be written at an 8th grade reading level. In practice, this is not always the case; documents often include specialized medical terms (e.g., “yersiniosis”), acronyms (“MRSA”), and other complexities that might be difficult to process for some human readers. At the syntactic level, information is often condensed into bulleted lists, enumerations, or multi-noun compounds (such as “waste clearance request form”). Nevertheless, the language of PH documents is not nearly as complex, technical, or ambiguous as that of biomedical literature or clinical texts, and is closer to everyday language.

In a detailed analysis of machine translations of 25 randomly selected public health documents into Spanish, we found that a generic MT engine (Google Translate) produced high-accuracy outputs: on a scale from 1 (worst) to 5 (best), the semantic adequacy of the translations (i.e., the correct translation of the content) was rated 4.19 on average; the grammatical fluency (correct syntax, word forms, etc.) was rated 3.73 on average.[29] Thus, we can expect current generic MT systems to provide a reasonable starting point for applications in the health domain.

1.2 Tailoring a MT system to public health practice

A second, equally important problem involves actually incorporating MT systems into PH practice. Information systems created for PH practice are often underutilized. This is due in part to information systems in PH being designed without a clear understanding of the context of PH work and information workflow.[31-33] Information systems that are poorly aligned with workflow are not well adopted and can produce negative results.[31,34-36] The goal of this study is to investigate current PH translation information workflow and processes in order to identify the functional requirements for an MT tool that can effectively be incorporated into PH workflow and practice.

We utilized principles from the Cognitive Work Analysis (CWA) framework to understand the goals, tasks, processes, and constraints associated with creating translations for LEP populations.[37] CWA is a valuable framework for understanding the context of complex work environments and the individuals who perform that work. We utilized this framework to investigate two large health departments in Washington State (WA) with the aim of identifying how MT systems can be designed to fit into current PH workflows through context-driven design of new technologies.

CWA is a framework derived from cognitive engineering in which work is explicitly analyzed to inform the design of technologies to support human work.[37] Through interviews, artifacts informal observations and participant feedback we used CWA principles to take into consideration the context in which the work is done, the environment, and the goals and values of the individuals performing the work. CWA has been used to inform the design of clinical physician order entry (CPOE) systems [38] and other clinical information systems.[39,40] Because information systems in PH are often underutilized, we hoped to improve utilization by better understanding the context in which PH translation work occurred. To our knowledge, CWA has not previously been applied to PH practice.

The studies reported here were designed to provide a better understanding of the information workflow involved in the creation and generation of multilingual health materials in PH, with the goal of informing the design of MT systems that meet the needs and work practices of PH agencies. The results of these studies led to the identification of functional requirements for the development of a prototype MT tool (PHAST) for PH as described in more detail below.

2. Materials and methods

We performed case studies of two health departments using qualitative techniques.

2.1 Site selection & setting

We investigated the two largest health departments in WA: Public Health - Seattle & King County (PHSKC) and the Washington State Department of Health (WA DOH). A prior health department survey revealed that these two health departments are responsible for producing the greatest number of translations in WA.[41] Initial interviews with key informants identified the divisions within each health department principally responsible for developing and translating health promotion materials.

Located in Seattle, PHSKC serves 1.9 million residents. PHSKC is one of the largest metropolitan health departments in the nation, with 1,500 employees and a budget of $318 million.[42] In King County, WA, 300,000 residents over the age of 5 speak a language other than English at home; 199,000 or 10.9% of the population have limited English proficiency [43].

WA DOH, located in Tumwater, WA, serves the 6.8 million residents of WA. In addition, there are seven satellite offices located throughout the state. WA DOH has 1,500 employees and an annual budget of $600 million.[44] In 2010, the total number of people classified as LEP in WA was about 512,000, or 7.6% of the total WA population. The top five non-English languages spoken were Spanish, Chinese, Vietnamese, Korean, and Russian.[45]

2.2 Semi-structured interviews

Over a 12-month period, semi-structured interviews were conducted with key health department personnel involved in the translation of health promotion materials (factsheets, press releases, health education pamphlets, etc.). Of The 34 interviewees, 10 were bilingual, of whom nine were bilingual in Spanish.

Interviews were performed primarily onsite. Three researchers (MB, KC, and CB) conducted 1hour, semi-structured interviews with key informants from PHSKC (n=21) and WA DOH (n=13). At the time of the study, all three interviewers had a Master's degree in Public Health and one also had a Master's degree in Social Work. All interviewers were trained and experienced in conducting qualitative studies. All three interviewers were bilingual. The principle interviewer (MB) was a native Japanese speaker who is fluent in English and speaks basic Spanish. The other two were native English speakers; one is fluent in French, the other speaks basic Spanish.

Consistent with the CWA framework, the interviewees were asked questions regarding the social, physical, and political contexts of translation work; information processes, tasks, resources, and key decisions involved in the translation of health promotion materials; barriers and facilitators to translation activities; and attitudes towards using MT to produce multilingual materials.[37] We recruited staff from work units within the departments that frequently performed translations. Interviews were recorded using a Sony digital recorder (model #ICD-U60) and field notes were taken. Artifacts such as forms and guidelines used in the translation process were collected.

Interviews were conducted until saturation was reached (i.e., the point at which no additional information regarding translation processes is gathered).[46] The University of Washington (UW) Institutional Review Board (IRB) approved all study procedures.

2.3 Interview data analysis

Interviews were transcribed verbatim by an experienced member of the research team or by professional transcriptionists. Two researchers analyzed the field notes and transcripts using Braun and Clarke's method of thematic analysis [47] to identify broad categories and the initial coding scheme. The researchers independently coded two transcripts to create a preliminary code list using qualitative data analysis software (ATLAS.ti, Version 5.7). The researchers then reviewed each other's coding and built consensus whenever disagreements arose. Throughout the coding process, researchers updated and refined the code book and transcripts if new codes were added.

2.4 Workflow model development & member checking

Analysis of the interview transcripts and artifacts directed the initial task analysis and development of the information workflow models.[48] Workflow diagrams provide a summary of the tasks involved, the order in which work is carried out, and what steps must be completed to proceed to the next task. By better understanding and documenting workflow, we can identify task redundancies and inefficiencies in the current information work processes that could be improved through a well-designed information system. Workflow diagrams help managers anticipate alterations in workflow and staff roles which may be met with some employee resistance during the process of information system implementation.

Observations are a common method of documenting workflow; however, because of the distributed and episodic nature of the tasks related to translation, performing direct observation to investigate translation workflow was determined to be impractical. Instead, we used detailed interviews in conjunction with document artifacts to outline the tasks involved in machine translation. We sought feedback from participants via focus groups to confirm our workflow and task analysis findings. The process of task analysis is described in greater detail in Turner et al., 2013.[48]

Briefly, we applied sequential process analysis to develop our information workflow models.[49] First, key processes were identified. Specific categories of codes included translation tasks, staff role, triggers, modes, and points of information transfer. Two researchers created separate diagrams which were combined to ensure reconciliation of differences. The specific tasks involved in each process, as well as the order of tasks, were used to generate preliminary information workflow models.

We conducted two 2-hour focus groups with staff members at PHSKC and at WA DOH to verify translation processes, task flow, and workflow diagrams. The four focus group sessions had 7-8 participants, all of whom were involved in some aspect of the translation process. Participants were employed as health educators, communication specialists, program managers, and graphic designers. Focus groups were audio recorded and transcribed. Based on focus group input, we made minor modifications to the sequence and frequency of tasks, as well as clarifications regarding who conducts each task.

3. Results

We conducted a total of 34 interviews (Table 1). Two researchers reviewed and coded 36 hours of interview transcripts and 8 hours offocus group transcripts.

Table 1.

Participants and roles

Role	Number of Participants	Health Department
		WA DOH	PHSKC
Manager	9	5	4
Health Educator	6	3	3
Environmental Health Investigator	5	2	3
Health Communication Specialist	1	0	1
Administrative Staff	3	1	2
Medical Interpreter/Translator	1	0	1
Translator/Outreach Coordinator	1	0	1
Administrator	1	0	1
Graphic Designer	1	1	0
Public Information Officer	4	2	2
Public Health Researcher	1	0	1
Public Health Nurse	1	0	1
Total:	34	13	21

3.1 Characteristics of information workflow processes

Based on the content analysis of the interview and focus group transcripts, translation processes were divided into four phases (Figure 1). Differences in workflow between divisions or departments are noted.

Figure 1.

Four phases of translation workflow

3.1.1 Material development phase

In the material development phase, English documents are prepared for translation. This phase includes tasks such as making decisions regarding target audience, language(s), resources, and the best format for communicating key messages, and also often involves reducing the reading level, removing colloquialisms, and ensuring use of culturally appropriate terms, phrases, concepts and graphics. PH materials commonly translated included flyers (e.g., “Planned Power Outage: Keeping Your Food Safe to Eat”), factsheets (e.g., “Why HIV testing is important for you”) and press releases (e.g., “Unusual Rise in Heroin Overdose Deaths Over Weekend”). A more comprehensive list of materials translated at participating health departments is included in a recent article in the Journal ofPublic Health Management and Practice.[48]

3.1.2 Translation phase

Each translation project involves a balance between proven best practices and time/resource constraints. Key tasks involved in the translation phase include the selection of a translator and the actual translation of the English source document. Translation methods include hiring a commercial vendor, assigning the task to a bilingual in-house staff member, or using a community partner. Choosing a translation method can be a complex process and depends on the target language and audience, urgency (of subject matter?), scope of project, and budget.

The variety of languages represented by bilingual staff is limited even in large health departments. As a result, PHSKC and WA DOH staff primarily use translation vendors. In general, the vendor translates the document and a second translator in the company reviews it for quality. When creating high-profile materials such as brochures and posters, the vendor may format the document at additional cost. The vendor returns the document in Microsoft Word or PDF format. At our request, a PHSKC staff member questioned the ten vendors most commonly used for translations. Of the eight vendors that responded, none said they used any form of machine translation in their translations.

In addition to using vendors, PHSKC and WA DOH staff members may rely on bilingual staff, community partners, or interpreters to conduct translations. This is not the preferred approach but can happen particularly when short turnaround times are necessary. The English document is sent with instructions to the selected translator, who then translates the content. Depending on the availability of bilingual reviewers, some materials may receive additional review, and revisions are made. If indicated, graphics are added and further reviews are conducted until the project manager approves the final product.

3.1.3 Quality assurance phase

The quality assurance (QA) phase involves additional quality control after the initial translation is complete in order to ensure the accuracy of translation. Two main QA methods are used depending on the urgency, budget, availability of reviewers, and project size (Figure 1). The staff first sends the translated document electronically to the reviewer, who sends comments and edits back to the originator. Staff members review the document and decide whether to incorporate the edits themselves or send the corrections to the initial translation vendor for editing. If a staff member has the capacity (i.e., familiarity with the target language and ability to maintain fonts), then he or she performs the final editing. Translations are occasionally sent back to the vendor for retranslation.

3.1.4 Post-translation phase

The post-translation phase consists of administrative tasks related to payment and distribution. Once the translation is ready for distribution, it is either sent to a printing company for mass production, or distributed via the Internet. Vendors are paid using an internal procurement system. Upon payment, the project is considered complete.

3.2 Contrasting PHSKC and WA DOH workflows

Although the sequence of tasks to create translated materials is fairly similar for both health departments, there were some variations in workflow. First, the amount of audience testing and quality review conducted varied between health departments (see “Varying Quality Standards”). Additionally, graphic designers played a bigger role in the overall translation processes at WA DOH compared to PHSKC. WA DOH employs multiple in-house graphic designers with extensive experience in formatting and designing translated materials (which high-profile materials often require), as well as serving as a liaison to translation vendors and internal staff.

Themes

Analysis of the conducted interviews led to the emergence of themes related to barriers, facilitators, and machine translation.

Barriers

Factors that either discouraged staff from creating translations or made translation processes difficult were defined as barriers. Major barriers included cost, time constraints, lack of clear translation guidelines, and lack of awareness or commitment to translation.

3.3.1.1 Cost

Cost was the most frequently reported barrier to translating health promotion materials. Most programs at WA DOH and PHKSC did not have designated funds for translation. Recent budget cuts further limited available funds. The lack of funds affected whether the material was translated, the number of languages the material was translated into, and the option of performing quality-enhancing activities such as content customization, material testing, and incorporating appropriate graphic design work.

3.3.1.2 Intermittent nature of translation work

In most programs, translation work occurred at irregular intervals, and was often triggered by unpredictable events such as critical PH incidents (e.g., a major snow storm) or requests from field staff. Consequently, there is no established time to perform translations, resulting in translation often being considered a “side job” that bilingual employees perform on an ad hoc basis in addition to their primary duties. The need for an urgent PH response often prevented programs from translating materials into more than one language; creating high-quality translated materials took time, with reported turnaround from the vendor ranging from one to several weeks.

3.3.1.3 Lack of clear translation guidelines and policies

Uncertainty around when to translate materials was a barrier. Some interviewees attributed this to a lack of clear departmental policies on when to translate. This lack of clear guidelines on when and what to translate prevented them from producing translations altogether. Others indicated that a lack of data on language needs among the target populations was a barrier to making a decision to translate. (Of note, PHSKC did have translation guidelines, but most staff members were unaware of them.)

There was a lack of clarity around what constitutes a translation “need.” Unless translation was mandated by a grant or legal requirements, programs often decided whether to translate materials on a case-by-case basis.

Although not directly stated, an implicit theme was a lack of standardized processes for performing translation (such as when to translate, who should translate and how to ensure quality) between health department divisions. The lack of standard practices and clear guidelines contributed to inefficiencies in creating translations.

3.3.1.4 Lack of awareness or commitment to translation

There were varying levels of commitment to translation across the health departments. Political will by the upper management was perceived as key to overcoming financial and resource barriers. Programs with a strong commitment to providing translated materials considered translation a priority and advocated for translated materials even when resources were scarce. Staff members from these programs were more aware of the need for translations and reported encountering fewer barriers.

3.3.2 Facilitators

Facilitators were defined as factors that make the work of translation easier. Key facilitators are outlined below.

3.3.2.1 Skilled and experienced translation staff

Creating high-quality translated health education materials is aided by multiple staff members with a unique set of skills, including fluency in another language, background in health education and promotion, and project coordination skills. Translations are aided by the availability of bilingual staff representing a wide variety of languages.

Programs with clear translation policies and with staff experienced in coordinating translation projects had smoother translation operations, produced translations more frequently, and were able to complete translation projects in a timelier manner, even undertight deadlines.

3.3.2.2 Communication with translators and QA reviewers

Prior to starting translation work and throughout the translation process, staff members often provide detailed instructions to translators and QA reviewers, communicating requirements about reading level, preferred terminologies, and the avoidance of literal translation. Staff members have in-depth conversations with the translators and QA reviewers to resolve discrepancies. This open and iterative communication was considered the key to optimizing translation quality.

3.3.3 Quality

Staff members implied that translation quality was among the most significant, yet most difficult, element of their translation activities. Two key themes related to translation quality included the importance of QA and constraints on QA processes.

3.3.3.1 The importance of QA

Staff members frequently voiced concern that their limited language ability prevented them from adequately evaluating translation quality. Many staff members discussed the complexity of language and described a sense of anxiety around translation quality, which was ascribed to fears about causing offence, losing credibility, and being held responsible for any negative consequences of translations. These concerns were often expressed through stories of disseminating translated materials that contained errors or were culturally inappropriate. Poor quality translations were perceived as a sign of disrespect and hurt the credibility and validity of the intended health message. Stories about errors occurring with translations were common. For example, in a Spanish language factsheet about breast cancer, “breast cancer” was translated as “lactating cancer”. In a press release advising users to be even more careful about street drugs, the term “contaminated cocaine” was translated to “poor quality cocaine.” A notice about an emergency hotline was translated to “a line that is hot”.

3.3.3.2 Constraints on QA Process/Compromised QA Process

Four main constraints were repeatedly mentioned with regard to the QA process: 1) a lack of access to a QA reviewer with the necessary domain and translation expertise; 2) concern about “over-burdening” bilingual staff who could serve as QA reviewers; 3) resolving differences in opinion about word choice; and 4) differences in quality standards.

3.3.3.2.1 Difficulty finding qualified translators quality assurance reviewers

Relationships with a trusted translator or QA reviewer appeared to be the foundation of many programs' translation systems; however, it was often difficult to find translators with the domain expertise needed to translate or review specific content areas, especially in languages other than Spanish. Much is required of trained translators, who must ensure accuracy, adequacy, formatting, and appropriate reading level, tone, and dialect. Additionally, reviewers must be able to tailor translations to the community and have content expertise.

3.3.3.2.2 Burden on bilingual staff

Many staff members shared concerns about asking bilingual staff, whether internal or external, to conduct or review translations. Progarms that utilized bilingual staff from community-based organizations reported similar concerns.

3.3.3.2.3 Challenge with the vetting process

In any language, certain words or phrases could be written in a variety of ways. Often times, peer review results in different suggestions for changes to be made to the same document. Because the staff managing the project often does not speak the language, deciding how to resolve word choice disagreements can be a challenge.

3.3.3.2.4 Varying quality standards

QA is a critical translation activity. However, there was a range of opinions regarding what makes for a high-quality translation. For some staff members, a high-quality translation was one that clearly communicates the key messages, even if it was not perfect. For others, quality translations were linguistically accurate and culturally appropriate in terms of tone and subtleties of language.

A variety of factors influenced quality standards. One was document profile, which included two important aspects of document status: 1) longevity and 2) extent of dissemination. High-profile documents (e.g., county-wide advertising campaigns) last longer and require greater scrutiny. Low-profile documents (e.g., personal letters sent to individuals) required different quality standards, prompting comments such as, “as long as the point gets across, I'm okay with some typos,” compared to high-profile projects, where staff described being concerned about the potential for even small errors to offend the targeted community.

Programs with more frequent community contact often produced low-profile documents and distributed translations to their target audience in-person or followed up after (other means of?) distribution. They were able to clarify any points of confusion directly with the target audience and did not perform as much quality review. By contrast, programs producing high-profile documents had less community contact and lacked this opportunity for clarification. Even if they received feedback about errors early in a document's distribution, they would have difficulty making major changes because of the work's high-profile nature.

3.3.4 Attitudes towards automation

We gauged interviewees' knowledge, attitudes, and beliefs toward machine translation. More than half of the interviewees at PHSKC and WA DOH had used online MT software (such as Google Translate or Bing Translator) in the past. Most had used MT software only a handful of times. About half of those who had used MT had done so for personal use, mostly to get the “gist” of translated information. The remainder experimented with it in their work, either to translate a word or a sentence, or to get a second opinion on word choice. A small number of bilingual interviewees at both health departments reported using MT on a regular basis to check words and phrases. Two of these interviewees reported using MT to create an initial translation that they would then post-edit. Regardless of use, the majority of interviewees described the quality of MT translations as relatively poor.

While the negative impression of MT was prevalent, results were mixed in terms of interviewees' attitudes about the idea of using MT to translate PH documents. About half of the participants expressed an open - albeit cautious - attitude, explaining that MT could be viable if the quality was improved and the translation was post-edited by a native speaker.

A number of interviewees felt that MT could be useful for translating documents that have specific content, length, or urgency. Some thought that the use of MT would be appropriate when translating simple, basic information, rather than materials with complex information and technical or medical terminologies. Some also suggested that MT could be helpful when translating a word or sentence in short documents or for emergency materials that required quick.dissemination.

Several health department staff expressed concern that inaccurate translations of health messages could negatively impact health. Concerns expressed about the use of MT for translation included: 1) worry that MT may give literal translations and fail to reflect the nuances of language; 2) concern about the accuracy and fluency of the translations produced; 3) fear of damaging relationships with the community due to poor quality translations; 4) concern that translation using MT would not meet policy or program requirements or standards; and 5) a belief that conducting MT with post-translation review might be more work than translating a document from scratch.

4. Discussion

4.1 Understanding information workflow

Our findings identified key translation processes and barriers that are affecting the creation of multilingual health materials in PH. Interviews with key personnel from the two largest health departments in WA revealed that current translation processes are diverse across departments and lack standardization. The time and costs of creating translations in light of competing priorities and limited health department resources serve as barriers to producing language-appropriate materials for LEP groups.

Quality assurance is a principle concern for all divisions of both health departments. However, for lower-profile documents such as letters and notes, where the author had closer ties to the individual and the community, attention to quality assurance was less important than for higher profile documents because there was more trust that any confusion caused by translations could.be.readily.corrected.

4.2 Implications for information systems

We applied CWA to assist in determining the necessary functions and tasks of a translation system that would complement the work environment and the values and goals of the PH professionals involved in translation work. CWA has been used in multiple work settings including aviation, the petrochemical industry, vehicle safety, and health care environments.[50-60] Most applications of CWA have focused on interface design.[60] This study applied CWA in order to investigate the contextual environment in which translation was being performed to understand how to design a machine translation tool that would fit well into current PH practice. The CWA framework was useful for understanding the roles, goals, and tasks that need to be incorporated into the design of an information tool to assist with PH translation work. Given the importance of human quality assurance to ensure accurate translations that fit local and cultural norms, we designed a system that would provide the low-cost efficiency of freely available online machine translation systems with PH bilingual expertise.

The results of these studies have several implications for the design of MT systems for assisting in the creation and dissemination of multilingual PH materials. First, although some participants reported using MT for individual words or phrases, there were strong reservations about the use of MT in PH. Health department personnel will need to see evidence that a quality end product can result from the use of MT. Second, potential users of the technology will need to have a clear idea of how MT can fit into their current translation workflow.

The primary design consideration for a system that integrates MT into PH translation workflows is that MT cannot be used in isolation to produce multilingual health materials. Instead, MT always needs to be followed by human review and corrections (so-called post-editing) before the final product is released to the public. This approach, called MT plus post-editing, is widely used in the commercial sector. In previous studies, we have compared the time requirements and quality of manual translation versus MT plus post-editing for PH materials. [1] We found that blinded reviewers rate the quality of the resulting translations as equivalent, and that MT plus post-editing results in a much faster and more cost-efficient process.

Another key consideration was cost. Health departments reported having limited to no budgets for translations. Any translation tool must be easily accessible at low cost. For this reason, we focused our studies on the use of freely available, non-proprietary translation systems such as Google Translate and Microsoft Translator. In a previous study we followed translation tasks at several Washington health departments for 20 months and calculated the time and costs of producing 1-2 page translations using commercial translation vendors. [48] The average cost of translating a 1-2 page document was $365 using manual translation, versus less than $25 using MT plus post-editing by public health staff. A blinded comparison of the documents produced manually to those produced using MT plus post-editing, found that the two translations were essentially equivalent. Thus MT-plus post editing could offer substantial cost savings while at the same time providing quality translations.

In some domains, such as business, proprietary translation systems may perform better than the publicly available systems used for our studies. However, to our knowledge there are no proprietary systems specific to the PH domain, and subscription and contract costs for proprietarytranslation systems can be expensive.

Additional design considerations resulting from our study findings are listed in Table 3 and are linked to the themes in Table 2.

Table 3.

Design considerations

Theme	Design requirements
9.* Attitudes towards MT	MT always needs to be followed by human review and corrections (post-editing).
1. Cost	The system should be inexpensive.
2. Time constraints	The system should be easy to use, and should require a minimum level of maintenance and technical expertise.
2.2 Bilingual staff needs to “fit” translation in	The system should allow collaboration across different programs and health departments to take advantage of complementary staff language expertise.
2.3 Time to create quality translations	The system should be able to accommodate work in progress. One should be able to save the document to work on later.
6.2 Importance of having trusted QA reviewers	The system should allow communication between the requester and the post-editors (quality assurance translators), and provide contact information and ability to provide feedback from users.
7.1 Uncertainty about translation quality	The system should allow for comments, notations, marking disagreements about how to translate/post-edit certain terms, and provide support for resolving conflicting opinions.
2.1 Urgency	The requester should be able to note the level of urgency of the translated document.
8.1 Document profile	The requester should be able to note the intended visibility and permanence of the document. Key attributes of translations include the target audience, dialects, reading level, domain terminology, and formatting. Information about these factors should be stored with the documents.
6.1 Trusted translation partners	Background, qualifications and contact information regarding translators should be available, including information regarding language expertise, years of experience, certifications, and user ratings.
3. Lack of clear guidelines/standardized practices and policies	The system should reflect and incorporate standard processes and translation guidelines, as well as best practices.
9.2 Concerns about offending public	Systems should identify, track, and archive MT errors for continued training and improvement of MT of PH information.

^*Numbers correspond with themes listed in Table 2

Table 2.

Themes and Selected Quotations

BARRIERS DOMAIN
Theme/Sub-themes	Quotations
1. Cost
1.1 Translation cost	“We need to do more translation work, but right now, with the way things are, it's hard to realistically think where's the money going to comefrom...” (Transcript 14 24:101).
1.2 Cost of enhancing the document	“Interviewer: What's preventing you [from doing] more translation? Interviewee: It's notso much the translations as all the testing and the graphics and so forth. It's mostly time and money” (Transcript 24 10: 204).
2. Time constraints
2.1 Urgency	“Currently, there's this very marked increase in the number of syphilis cases among gay men, and there's an urgency in getting that information out there, and limited staff, and if we had more staff and less urgency, we might do more in Spanish that we're going to do” (Transcript 16: 30:88).
2.2 Bilingual staff needs to “fit” translation in	“What makes it hard to create most translated materials is the responsibilities and the timelines put in by our supervisors on our own job” (P8:342). (A quote from a bilingual employee who frequently conducts translations).
2.3 Time to create quality translations	“A reality of [translation] is that it's a process, it's not instantaneous. It takes a while” (Transcript 18 40:56).
3. Lack of clear guidelines/standardized practices and policies
3.1 Uncertainty around when to translate materials	“[T]here should be an agency policy and procedure... At the moment, there's absolutely nothing official to mandate that any program gets anything translated” (Transcript 28 16:261).
3.2 Lack of standardized translation processes	“And maybe it's lack of awareness but it's also a kind of confusion about whether to do it, how to do it, when to do” (PHSKC FG).
4. Lack of awareness or commitment to translation
4.1 Lack of awareness and political will	“It's always been the attitude of a lot of the management of this department that if we can do it [translation] great, if we can't there's no obligation” (Transcript 22 18:84).
FACILITATORS DOMAIN
5. Staff translation skills and experience
5.1 Highly experienced staff	“[M]ost of the time I've had xxx who had been my project manager... she worked hard on the translation manual. She was our translation guru and she handled a lot of it. (Transcript 2 29:90).
6. Trusted translators and QA reviewers
6.1 Trusted translation partners	“[T]here's the relationship with the person that coordinates the translations, I feel like I've developed a rapport over time... I feel comfortable that they've asked me questions, clarifying questions, which is something I feel really good about in the translation process” (Transcript 3 38:94).
6.2 Importance of having trusted QA reviewers	“My sense is that what we got [from a translation vendor] was an accurate translation, but it was not a culturally sensitive translation for this particular population. With a language like Spanish, we have enough contact I think within that community that we can get that kind of feedback and we can feel more confident about the things we put out” (Transcript 16 30:32).
QUALITY DOMAIN
7. Importance of QA
7.1 Uncertainty about translation quality	“It can be really tricky cause you're really trusting somebody else, did they get the concept you hope that they get, and that that's the right word choice for it” (Transcript 3 38:36).
7.2 Concerns about negative consequences of poor translations	“Because if you translate something really poorly, it damages your credibility with that population” (Transcript 22 18:155).
8. Constraints on QA
8.1 Difficulty finding qualified translators	“We have found in these major translations services that they're not perfect, especially when we're dealing with the nuancing of messaging... You may have these simple sentences and you try to rewrite it three or four different ways so they really understand, but we're not assured... that they really get that until we go get them proofed and they get nuanced a little more. But we've had some major mishaps in the past” (Transcript 32 15:154-155).
8.2 Burden on bilingual staff	“I hear that it really is not appropriate to pursue [bilingual staff], to approach them with something and, say will you check this or translate it again just cause there's so many other things that they need to be doing” (Transcript 1 26:293).
8.3 Varying quality standards	“From a working group... it was OK for me to use the half-translations that were happening. But for use statewide, on a broad population base, it became not OK” (Transcript 32:98).
ATTITUDES TOWARDS AUTOMATION DOMAIN
9.1 The importance of human post-editing	“I think [machine translation] can help in the process, but it should never be relied on solely” (Transcript 31:378).
9.2 Concerns about offending the public	“Anyway that's what makes everyone nervous about machine translation because we know we'll be raked over the coals every time we get it wrong and we just worry with a machine doing it, the chances of there being an error would be high” (Transcript 2 29:164).

4.3 Description of the PHAST system

Based on these requirements, we have begun to develop a translation management system (the Public Health Automated System for Translation, or PHAST) intended for use by staff members in local and regional PH departments. The system is an online platform that supports a complete MT-assisted translation pipeline. The pipeline reflects the information workflow analysis findings. As the first step, an English-language source document is uploaded by a requester. The document is automatically translated into the desired target language upon uploading. The resulting translation can then be post-edited by any user registered in the system who has expertise in the target language. Once post-editing has been completed, the finished document can be downloaded.

Within the limits of its initial deployment, the system is inexpensive and easy to use. It has been implemented as a secure, centrally hosted web-based platform that can be accessed via any modern web browser; no specialized software installation or maintenance is required on the health departments' side MT is provided by the Microsoft Translator API; use of this functionality is currently free up to a limit of 2 million characters per month. The user interface and functionalities of the system were specifically designed for users without an in-depth technology background or experience in MT/post-editing.

The current design allows employees from different geographically distributed health departments to utilize the system in a collaborative fashion. For example, a health department needing a translation into Russian can upload a document, and a staff member of a different department with expertise in Russian can post-edit the MT version of that document. Departments can thus take advantage of complementary language expertise and, over time, build a shared repository of translated documents in a wide variety of languages This feature is further supported by a survey of Region X health departments that found the vast majority did not share promotion materials. [20]

Users who commit to post-editing a machine-translated document can save their work and resume it at a later stage. They can also release the document to a different user to finish the post-editing. However, to avoid real-time conflicts, multiple users cannot edit a document simultaneously. The necessary communication between different post-editors (e g , about how to translate a specific term) is facilitated by a “notice board” that is associated with each document and that collects and archives all comments and messages related to the document.

The need for a document profile is handled by a field that allows requesters to enter meta-data about the uploaded document, such as the intended audience, the desired reading level of the translation, etc. A similar field is available to post-editors when registering with the system to list any pertinent information about themselves (e g , years of experience, certifications obtained, etc.). This information is stored in their profile, thus addressing the need for trusted translation partners.

Standardization of the translation process is facilitated by the design of the translation pipeline and post-editing interface. For example, documents are not marked as downloadable until they have been completely post-edited. In order to reach this status, every sentence needs to have been explicitly saved as “ok” by the post-editor. Thus, the system enforces a thorough sentence-by-sentence post-editing procedure. Of course, additional translation guidelines and standards need to be set by each individual agency. Finally, post-edits can be stored and downloaded along with the machine translations and original documents to serve as new training data for machine translation engines. For example, the Microsoft Translator Hub offers a way to build domain-specific, customized translation models that could be used in this context.

The PHAST system is currently in its beta version and is being field tested with local health departments in WA.

4.4 Improving MT technology for PH applications

In addition to designing translation management systems for our intended audience, continued work is needed to improve core MT technology for PH information. As stated previously, the broad range of subject matters, diverse audiences and criticality of clear communication of health messages make the PH domain particularly challenging with respect to MT. A more detailed study of individual MT errors found that the most important problem that can be attributed to the domain and subject matter of the documents is word sense errors.[30] Such errors involve the translation of a word as an expression that is not appropriate to the context, e.g., the Spanish translation of “drugs” as “drogas” in the sense of narcotic drugs rather than “medicamentos” (medicine). Although word sense errors are not the most frequent errors overall, they are among the most disruptive error types.[30, 61]

While current machine translation performance on public health documents is reasonably good, it could be further improved by customization and domain adaptation. Our team has previously identified common errors created through MT and is currently developing algorithms to automatically identify and correct those errors.[61] To our knowledge, MT systems customized for the PH domain are currently not available either commercially or in the open-source community. One possible way of customizing a generic system is to collect parallel text from the desired domain and use it to update the translation models. This is a procedure that is enabled by tools such as the Microsoft Translator Hub, and could be used in our PHAST system. Initial experiments using a data set of roughly 10,000 words of parallel data have shown slight improvements in translation performance as measured by automatic evaluation standards. An alternative way of customization that we are actively pursuing is to create specialized modules for improving the disambiguation of multiple alternative translation options for content words.[62]

4.5 Study limitations

This case study was an exploratory investigation of translation processes and information workflow at two large health departments in WA. Although these departments produce the majority of translated materials in WA, the results may not be similar for smaller local health departments or other departments across the country. PH work has a dynamic nature and includes staff changes, budget changes, and policy changes. This not only makes assessments of workflow challenging, but also needs consideration when designing systems for PH. It is important to maintain contact with health department personnel over time and to document and incorporate changing contexts where pertinent.

Conclusion

Our results indicate that there is a tremendous need for low-cost, efficient methods of translating a variety of PH materials into multiple languages for the growing LEP population. Improved access to quality translations is clearly a shared value of the health department personnel we interviewed. However, the barriers of time and cost prevent health departments from meeting this goal.

Despite concerns expressed, we believe that novel informatics solutions such as MT could greatly assist in increasing the production of low-cost, high-quality, multilingual PH materials. However, the necessary prerequisites are quality assurance and the design of MT-assisted translation management systems that are tailored to fit current PH workflow and resources.

The success of such systems will require clear guidelines outlining standardized procedures, sharing of resources across programs and departments, and quality assurance processes to ensure the production of accurate and culturally appropriate health materials for non-English speakers.

Highlights.

• Participatory design was used to model translation processes and inform the design of a public health translation tool.

• Time and language limitations are barriers to providing multilingual materials for limited English proficiency groups.

• Translation work is distributed, episodic and involves multiple quality assurance steps.

• A collaborative machine translation system may provide an efficient way to create public health multilingual materials.