RO1 Funding for Mixed Methods Research: Lessons learned from the Mixed-Method Analysis of Japanese Depression Project

Abstract

Mixed methods research has made significant in-roads in the effort to examine complex health related phenomenon. However, little has been published on the funding of mixed methods research projects. This paper addresses that gap by presenting an example of an NIMH funded project using a mixed methods QUAL-QUAN triangulation design entitled “The Mixed-Method Analysis of Japanese Depression.” We present the Cultural Determinants of Health Seeking model that framed the study, the specific aims, the quantitative and qualitative data sources informing the study, and overview of the mixing of the two studies. Finally, we examine reviewer's comments and our insights related to writing mixed method proposal successful for achieving RO1 level funding.

Introduction

Mixed methods research has made significant in-roads in the effort to examine complex health related phenomena. This fact is illustrated by development of mixed methods research and mixed methodological journals, the publication of a variety of studies using these methodologies, and ongoing inquiry into the development and application of methods. However, little has been published on the funding of mixed methods research projects. For U.S. academic centers and researchers, the gold standard of funding success remains the acquisition of National Institute of Health (NIH) funding. The NIH is the research arm of the U.S. Office of Health and Human Services and has 27 institutes and centers, generally focused on research related to specific diseases. In recent years however, NIH has recognized the richness and strength of multidisciplinary and transdisciplinary research models. The time has come for new research methods that use these new approaches, and mixed methods research can be used to address these multi-level health problems. However, there are as yet too few examples of such research funding, or discussions about what makes a mixed methods proposal fundable through the NIH.

This paper addresses that gap by presenting an example of an NIMH funded project entitled “The Mixed-Method Analysis of Japanese Depression.” This project was enthusiastically embraced by the NIH, and was funded on the first submission; a relatively uncommon achievement. We received special attention and awards from both the Office of Behavioral and Social Science and the Office of Women's Health research for this study. We believe that these awards were garnered because the reviewers and agencies recognized that mixed qualitative and quantitative research methods are an innovation. In recent years, the NIH has actively sought projects that use new methods and technologies to provide insights into health related processes.

In the following, we present a discussion of how mixed method research can be used for research about health disparities. Next, we present the theoretical framework for the exemplar NIH study described in this paper, and then describe the specific aims and pertinent methodological considerations of our research. Finally, we present some examples and analysis from the grant reviews that can help our emerging science to be more successful in gaining funding. It is our hope that this example will begin a dialogue and facilitate future successes of additional mixed methods research proposals submitted for NIH funding.

In the arena of health disparities research, there has been a cry for mixed method and multi-level research both outside and within NIH. In a review of research and publication on health disparities for women, Read and Gorman concluded that mixed method and/or multilevel research is an essential next step to both elucidate and eventually remedy health inequities for women and other minority groups (Read and Gorman, 2010). In 2009, the NIH hosted a Health Disparities research summit, and this resulted in a special issue of the Journal of Public Health. In general, this summit concluded that multilevel and mixed method research was needed to understand these complex problems (Dankwa-Mullan et al., 2010; Ruffin, 2010). The strategic vision of the Office of Behavioral and Social Sciences Research (OBSSR) at the NIH has as its foundation a call for collaboration among diverse disciplines, suggesting that this is the best way to address the complex, multidimensional issues that challenge the public's health. NIH has reorganized in response to this need for innovative research. One such reorganization is the Common Fund, which was enacted as part of the 2006 NIH Reform Act to support cross-cutting, trans-NIH programs that require participation by at least two NIH Institutes or Centers. The Common Fund was created to supports short term initiatives that work across the institutes and programs at NIH, which are known collectively as the “NIH Roadmap for Medical Research.” One such program is Interdisciplinary Research (IR). The IR program is working to affect the research culture of academe, and supports this change by efforts to encourage the biological and behavioral and social sciences to engage in trans-disciplinary research together. However, as promising as these efforts sound, the barriers to this type of research are substantial, and multi-level research is complex.

Since this new call for health disparities research requires multiple levels of data, from the genetic and cellular to the individual, to the group to the social level, it inherently “mixed methods research.” Conceptually, mixed methods research combine methods in order to increase the breadth, depth and scope of knowledge using different methods for different inquiry components. Mixing method allows data from different sources or levels to inform each other, strengthening the reliability of findings about any given phenomenon (Coyle & Williams, 2000; Creswell, 2003; Stewart, Makwarimba, Barnfather, Letourneau, and Neufeld, 2008; Tashakkori & Teddlie, 2003, 2010). Quantitative (QUAN) methods broaden knowledge by using numerical data across large samples. However, limitations of this method are that the findings are necessarily limited to a few variables, and divergences among the populations, process details, and situated meanings of the phenomenon are not available. Qualitative (QUAL) methods provides rich data that deepens knowledge about processes, antecedents, consequences, and meanings, capturing the vicissitudes of participant's lives. However, limitations of this method include whether the findings are transferrable to the population or to other groups.

Some of the major purposes of mixed methods research include corroboration, elaboration and development. Corroboration is also known as methodological triangulation, and refers to the use of multiple methods to test the consistency of findings across methods (Creswell & Plano Clark, 2007). Elaboration refers to the use of data from one source to clarify or aid in the interpretation of data gathered using another method. Development refers to the use of results from one method to either design instruments or to select data analysis strategies for using another method. Mixed-method research designs include both QUAL and QUAN methods collected concurrently, sequentially, or in an iterative fashion. Data can be mixed by transforming one type of data into another form, and then merging them together into a single dataset (Creswell, 2003; Tashakkori & Teddlie, 2003, 2010).

All research methods (including mixed methods) have strengths, weaknesses and tradeoffs. While linked qualitative-quantitative (QUAL-QUAN) methods allows researchers to address deficits in any given method with the strengths of a complementary method, determining how and when to link data can be a major challenge for the researcher. The combining QUAL and QUAN data can be labor intensive, and can lack theoretical direction. Researchers may hold underlying assumptions about the nature of truth that can make integration difficult. QUAL research methods are inductive, beginning with observations of reality, then looking for patterns that might become theories, while QUAN research methods are deductive, beginning with theory testing. Clarity about these assumptions, and the theoretical rationale for use of each method can help the researcher determine how and when to link data. The study presented here uses parallel data gathering, and shares the assumptions of the integrated mixed method study design, with a conceptualization that all of the data is “research evidence” about the nature of the phenomenon, and equal emphasis given to QUAL and QUAN data forms. This study design facilitates corroboration and elaboration in both data analysis and interpretation phases of the study, illuminating deeper meanings and processes, an aim that is essential for health disparities research. Mixing of the research data in this current study occurs at the end of the project.

Background and Specific Aims of the Study

Research literature and our research with Japanese women in the U.S. and in Japan have yielded consistent findings revealing how culture shapes symptom experience and interpretation, and how these interact with social structure to affect help-seekingⁱ (Ayu et al., 2010; Groleau, Young, & Kirmayer, 2006; Guarnaccia, 1997; Kirmayer, 1989; Kirmayer, 2001; Kleinman, 1995; Mezzich et al., 1999; Nakayama, Seko, Takatsuki, Miura, & Miyatake, 2010). This knowledge makes it imperative that multiple methods of data collection and analysis, as well as an interdisciplinary team of researchers, examine this in more careful detail. While an ethnographic qualitative study would potentially inform our knowledge of the definitions of distress and the related processes of help seeking, the generalizability of these findings can be enhanced by large and quantitative analyses that can link symptom patterns and help seeking behaviors. Keeping in mind that quantitative data alone, without ethnographic and psychiatric data, limits understanding of the meanings of these for the people experiencing the phenomenon. Therefore, we assembled a team of researchers representing nursing, family medicine, psychological anthropology, transcultural psychiatry and social psychology, and developed a proposal that used mixed methods data collection to explicate how culture and social networks interacted to shape help-seeking within large samples of Japanese people, in the hopes that these findings and methods might be used with other cultural subgroups in the U.S.

In the mid-1990s, a Japanese expatriate woman in Detroit tragically became the focus of the evening news when, during a postpartum psychosis, she drowned her baby (Fetters, 1997). The intercultural dialog between the Japanese community and the clinical community that followed highlighted the critical need to understand how symptoms interact with social and cultural factors to affect help seeking for Asians. Asian Americans are the fastest-growing ethnic group in the United States (U.S. Census Bureau, 2001; US Department of Health and Human Services, 2001). The Asian population grew faster than the total U.S. population between 1990 and 2000 (U.S. Census Bureau, 2001). Michigan has a large Japanese expatriate population due to the presence of the Japanese auto industry around Detroit. The majority of Japanese living there work in the automotive industry and related businesses, and they frequently bring their families with them. There are between 6-8,000 Japanese families affiliated with this industry within Greater Detroit alone, with a population in 2008 of over 10,000 (Consulate General of Detroit, 2008), and the Japanese population in surrounding counties ranges from 4.2 to 8.3% of the total population (U.S. Census Bureau, 2001).

Characteristics of the Asian population that may put them at risk for mental illness include separation from their extended families; intergenerational conflict, family system and role relationship changes; acculturation conflict related to strong ethnic identification; lack of English language proficiency; employment problems; and discrimination. While it is true that any given immigrant or ethnic population varies along a continuum of less to more acculturation to the host culture(s), we believe that the starting point for understanding how culture affects symptoms, illness explanations and help-seeking is to focus on less acculturated groups. This is important for two reasons. First, less-acculturated populations are most vulnerable to the risk factors outlined above. Secondly, we do not yet understand how less-acculturated populations experience symptoms or what help-seeking options they access or employ.

The prevalence of mental health conditions among Asian populations is not well understood. Studies examining the incidence of depression have found rates among Asians living in the U.S. as high as 25% (Cho, Nam, & Suh, 1998; D. Saint Arnault, 2004; Shibusawa & Mui, 2001) and women are predominately affected. Most of the prevalence data on depression in Japan is available only from Japanese government data published in Japanese. One such summary of studies is published by the Japan Committee for Prevention and Treatment of Depression (JCPTD). Lifetime prevalence rates for 2004 ranged from 4-18.5%, and one-year prevalence rates were 4-5%. These figures are similar to lifetime prevalence rates of 4.4-14.9% and one-year prevalence rates of 2.6-8.6% in the U.S. (JCPTD, 2010).

Despite the overlapping risk factors and prevalence rates, there is convergent evidence that Asians underutilize mental health services regardless of the service type or their regional population density (Sue, 1999; US Department of Health and Human Services, 2001). The consequences of underutilization include high illness prevalence and high illness severity (Sue, 1999; US Department of Health and Human Services, 2001). Studies examining factors that may cause this underutilization of mental health services have found the following; the tendency to endorse somatic rather than emotional and interpersonal problems (Kirmayer, 1986, 1991; Kirmayer, Dao, & Smith, 1998); stigmatization of psychiatric disorders and the desire to avoid shame and loss of face (F. Cheung & Lin, 1997; Gaines, 1998; D. M. Saint Arnault, 2002, 2009; Vogel, Wade, & Ascheman, 2009); differences in language, culture, and ethnicity that present barriers to health care access (Abe-Kim et al., 2007; Loue, 1998; Mezzich, Kleinman, Fabrega, & Parron, 1996; Sue, 1999); and a culturally-based lack of understanding of western-defined psychiatric disorders (Christensen, 2001; Groleau, et al., 2006; Guarnaccia, 1997; Kirmayer, 1991; Lee, 2002; Mezzich, et al., 1999). These studies have also found that Asians may favor help seeking through primary care and other physically oriented self-help, combined with social support. Physically-oriented self-help includes the use of natural remedies and professional help such as herbalists, acupuncturists, massage, exercise, chiropractors (De Wester, 1996; Kirmayer, 2001; Lock, 1987; Maeno, Kizawa, Ueno, Nakata, & Sato, 2002; Simon, VonKorff, Piccinelli, Fullerton, & Ormel, 1999; Waza, Graham, Zyzanski, & Inoue, 1999).

Since most research that studies culture and depression presumes that depression is a universal phenomenon, we began with a flexible definition of distress that included physical, emotional and interpersonal experiences identified by the subject as abnormal and significant, and that prompted subject attempts to ameliorate, mitigate, or eliminate them. Because there was so much that was unknown about the phenomena that interact in the help-seeking trajectory, we chose to use mixed methods to illuminate interactions among variables and processes. Our study used a three-phase approach: we used a survey sample and then an expansion subsample. Finally, we quantified the QUAL codes and imported them into a database for the women in the expansion subsample. Our analysis design was simultaneous, separate and iterative (Creswell, Fetters, & Ivankova, 2004 ; Greene, Caracelli, & Graham, 1989; Tashakkori & Newman, 2010, in press). The purpose of the expansion model of mixed method design is to increase the breadth or depth of the generated knowledge base by using different methods for different inquiry components. The first phase of this study sought to our knowledge about help seeking by using survey quantitative (QUAN) methods to gather personal, social and cultural data related to help-seeking data with a community-based sample. The second phase of this study sought to deepen our understanding of the processes by which these (and possibly other) variables operate in the life course of women by using ethnographic (QUAL) interviewing, followed by standardized psychiatric evaluations. Phase 1 and 2 data were analyzed simultaneously but separately. This simultaneous data analysis was iterative, but primarily for the purposes of interpretation, rather than for a redesign of any data collection strategies. We did not include in our plan revisions of either the QUAL or QUAN data collection methods based on analysis. Finally, we imported the QUAL codes into a database of the expansion subsample to examine differences based on demographics and symptom profiles.

The overall purpose of this proposed research was to investigate how the symptom and illness experience, cultural interpretations and evaluations, and social structural factors interact to influence help seeking. We used Japanese language translations of QUAL and QUAN instruments developed or refined from previous research to study distress and depression in a relatively non-acculturated population of first generation Japanese women. Our purpose was to examine the influence of culture on distress and help seeking for 250 Japanese women sampled from the general Japanese community and a Japanese primary care site. The specific aims of this study were to: 1) Examine the demographics, symptom experiences, and sociocultural factors that explain help-seeking strategies used by Japanese women recruited from community-based, primary care and mental health treatment sites; 2) Describe the ethnographic and psychiatric profiles of an expansion sub-sample of highly distressed women; and 3) Compare ethnographic and psychiatric profiles. Analyses from both phases of this study are complete, and findings are being interpreted and disseminated.

Theoretical framework of the study

Culture affects all aspects of distress, including the perception of it, the explanations for it, and the behavioral options to relieve it. Specifically, studies of depression in diverse non-western groups have revealed a tendency to experience negative, depression-like emotions as symbolic and holistically interrelated with somatic sensations and interpersonal disharmony. The phrase “idioms of distress,” has been used in medical anthropology to describe the culturally specific experience of psychosocial and physical suffering. Our proposal addressed the intersection of the macro-level force of culture with individual level phenomenon of perception, as well as help seeking for that distress. Because these phenomena occur in complex and interacting ways, we made the case that multilevel, multi-method evaluation was necessary.

Cultural Determinants of Help-Seeking Model

The PI developed the Cultural Determinants of Help-Seeking (CDHS) middle-range theory to guide this study. This theory has been published in its entirety elsewhere (XXXX, 2009) (see Figure 1). In brief, this model operationalizes cultural concepts derived from previous research, social psychology, and medical and psychiatric anthropology. The CDHS model proposes that the processes of perception and interpretation of distress, as well as the actions taken to relieve it, are filtered through cultural models. Culture guides internal perception. Further, when a person perceives a physical or emotional sensation, culture guides their interpretation about whether those perceptions are normal or abnormal. When a person deems the perception to be both significant as well as abnormal, these symptoms begin the process of contemplation about what further steps are necessary to alleviate them. Culture continues to guide additional interpretation and evaluation. First, a person must determine what the potential cause of the symptom might be. This causal interpretation is related to, but not the same as, an evaluation about the social significance of the symptoms. The evaluation of the social significance of symptoms prompts a person to examine the potential for support within their social network. Next, a person must determine how to communicate their distress, and the type of help that they need and/or can expect. Throughout this dynamic process, individuals make choices based on cultural beliefs, values and social structure. For example, a person may not deem certain symptoms to be abnormal. Even if the symptoms are determined to be of concern, a person may determine that they are mild, and only warrant further monitoring. Sometimes, a person may determine that their symptoms are caused by their own behaviors or ways of thinking, and institute internal adjustments. Alternatively, a person may see their own symptoms as caused by personal habits, and alter their diet, exercise and the like. Sometimes a person may decide that their symptoms suggest that they have committed a social failure, and decide to hide them. Of course, a person may also find themself in a social context where closeness and support are unavailable, or decide that asking for help creates too many social complications that they decide to keep quiet. Finally, a person may decide that their condition is not amenable to help, or that it is simply too shameful to acknowledge their needs.

Figure 1.

Cultural Determinants of Help-Seeking Model (CDHS) and Study Variables

Research Design and Methods

Study design

The overall purpose of this mixed methods QUAL-QUAN design was to examine the relationship between distress experiences, cultural interpretations, social structures and help seeking for less acculturated random sample of 250 Japanese women (Creswell, et al., 2004 ; Greene, et al., 1989; Qureshi, 1992; Tashakkori & Newman, 2010, in press). For this study, we chose an expansion design proposed by Greene, et al (Greene, Caracelli, & Graham, 1989). Specifically, each variable of the study had more or less developed knowledge bases that could be applied to immigrant help seeking. We used QUAN measures to examine the relationships among known variables that had appropriate measures, and QUAL methods to expand our knowledge aspects of those same variables that were unknown, including how these processes operated in the women's lives.

Because we were trying to understand the factors that were related to help seeking, we examined a community-dwelling sample for the survey. However, since we were interested in the detailed processes that made up women's experiences, interpretations and help seeking for distress, we selected an expansion subsample of highly distressed women. In order to select them, we evaluated the survey data to identify women who met threshold levels of a mental health screen and a symptom checklist. Women from this highly distressed expansion sub-sample of women were invited to participate in the second phase of the study, which included ethnographic and psychiatric interviews. Because of the stigmatized nature of the research topic, we advertised our study in local Japanese language newspapers, and included a letter that highlighted the numerous sponsors of our study, including the Japanese Business society, the local Japanese Saturday schools and the Consulate General for Japan. All data collection was carried out in Japanese, and all interviews were transcribed in Japanese and then translated into English.

Figure 2 depicts the flow of the study. This figure shows how both our QUAN and QUAL instruments emerged from preliminary studies, and how the data sources relate to each other. Analysis activities are depicted with shadows, and outcomes of various analyses are noted in outlined boxes.

Figure 2.

Study Design and Analysis Flow

All of the theoretical variables were examined with QUAL and QUAN data sources. However, for the QUAL analysis, additional information was provided for the reviewer that laid out the relationships between the theoretical domains of interest in the study, their QUAL analyses coding categories, data sources, and examples of QUAL (summarized in Table 1 below). For example, our analyses of the social aspects of help seeking included QUAN measures of the perceptions of the availability of social support (Perceived Resources Questionnaire), the values and beliefs related to social exchange (Asian Values Scale) and the use of social forms of support (Help Seeking Questionnaire). In addition, the ethnographic interview examined social network dimensions using the grand tour of the social network.

Table 1.

Data collection procedures according to type of data

Theoretical Domain	Analyses Categories	QUAN Data Source	QUAL Data Source	Example QUAL Data Analysis Statements
Social structure and social exchange	Level of intimacy	Perceived Resources Questionnaire (PRQ)	Grand tour of social network	Relationship qualifiers about closeness and distance
	Kinds of exchange	Social Help Seeking (HSQ)		Type of information, emotions and favors that are exchanged
	Barriers to exchange	Barriers to Care (HSQ)		Forces that inhibit exchange or help seeking
	Beliefs and values about exchange	Asian Values Scale (AVS)		Statements about customs, norms, expectations or potential consequences of social exchange
Symptom experiences	Specific symptoms	Center for Epidemiological Studies Depression CES-D Composite Symptom Checklist CSC	MINI: Western defined psychiatric diagnoses; categorical data	Presence of symptoms consistent with major mental illnesses including anxiety and somatoform, mood, and substance use disorders
			Life-line: Subjectively experienced distress defined in culturally relevant terms	All subjectively experienced physical, emotional and interpersonal experiences experienced during their distress periods
	Level of severity			The extent to which the symptom or distress caused concern or loss of function
Cultural interpretations and evaluations	Causal interpretations	Symptom Interpretation Questionnaire (SIQ)		Statements in which symptoms or distress are referred to in a cause and effect hypothesis, such as “because,” “related to” or “came from”
	Social significance	Beliefs about Mental Illness (BMI)		Why they specifically got the symptom, what the symptom says about them as a person, and/or social consequences or changes that occurred because of the symptom
Help-seeking behaviors	Sources	Help Seeking Questionnaire (HSQ)	Life-line	Social, self-help, non-medical professional, medical and psychological sources
	Types			Herbal and home remedies, emotional support, advice, exchange of favors, treatments, medication, counseling,
	Effectiveness			Description of specific relief obtained from both source and type of treatments and remedies

Sampling

Adult women 20 years and over who are able to complete questionnaires in Japanese were included in the study (based on our understanding that in Japanese culture, there is a turning adult ceremony that socially marks when a person becomes an adult at 20 years of age)(R. Y. Cheung & Park, 2010). This study focused on adult Japanese women because women are much more likely to experience depression than men (at rates estimated to be as high as 2:1)(US Department of Health and Human Services, 2001). We focused on first generation Japanese women because they have the lowest health care utilization and are at highest risk for distress (Abe-Kim, et al., 2007; D. Takeuchi et al., 2007 ; D. T. Takeuchi, Hong, Gile, & Alegría, 2007; Williams, 2002). This strategy was used to gather the “purest” Japanese experiences, interpretations, social structures and help seeking strategies.

This study recruited a survey sample using interval sampling with random start. The interview sample was a nested sub-sample who met criterion as being “highly-distressed” (see Figure 1). The random start interval sampling design was used to maximize the potential heterogeneity of demographics of the women, thereby increasing the likelihood that these findings might be generalizable to the rest of the less-acculturated population of Japanese women in this and other regions of the U.S. The steps for our sampling procedure were to estimate the number of women possible to sample, dividing that by the total sample number, and using the quotient as our sampling interval. Because we knew that Asian people tend to use multiple forms of help for distress, we sampled from the general community and primary-care sites to maximize the diversity of symptoms and help seeking strategies. The primary care recruitment site is the Japanese Family Health Program. This program has been under the direction of XXXX (second author and co-investigator on this research project) and began providing care for their first Japanese patient in July 1994. The primary mission program for this program is to provide culturally sensitive comprehensive health care services to the over 8,000 Japanese-speaking people within the Ann Arbor Detroit region, and to serve as a regional referral center to meet the needs of the Japanese-speaking population in the Midwest United States (Mitka, 2000). The community-based recruitment sites included Japanese-specific schools and women's clubs. The staff at both the primary-care clinic and the community based agencies compiled lists of all of the women for who they had addresses in the last six months. These staff then used random start interval sampling to select the sample and mailed survey packets to the selected women. Research staff members were blind to who received the packets.

Measures

Figure 1 depicts how we operationalized the model into the mixed method study variables. We used both qualitative and quantitative methods to capture these variables. The QUAN measures for Aim 1 included demographics, symptom experiences [the Center for Epidemiological Studies-Depression scale, Korean version, Revised (CES-D-K-R) and the Composite Symptom Checklist (CSC)] and the sociocultural factors that might explain help- seeking strategies [the Asian Values Scale (AVS), the Symptom Interpretation Questionnaire (SIQ), the Beliefs about Mental Illness Scale (BMI), Personal Resources Questionnaire (PRQ2000) and the Help Seeking Questionnaire (HSQ)]. The instruments used to measure the variables in Aim 1 are shown in Table 2.

Title of Measure	Measurement	References	Reliability (α)
Demographics	Age, marital status, visa status, the length of time abroad, employment
Center for Epidemiologic Studies-Depression scale, Korean version, Revised (CESD-K-R)	Measures mood, somatic complaints, interactions with others, and motor functioning; CESD-K-R reverses the wording of the positive affect items	Cho and Kim (1998); Cho et al. (1998); Noh, Kasper, and Chen (1998); Radloff (1977)	.71-.89
Composite Symptoms Checklist (CSC)	130-item multidimensional symptom measure from standardized somatic, affective, and interpersonal checklists	Kitayama, Markus, and Kurokawa (2000); Pennebaker (1982); Yasuda, Lubin, Kim, and Van Whitlock (2003)
The Symptom Interpretation Questionnaire (SIQ)	Causal attribution of commonly occurring physical symptoms	Robbins and Kirmayer (1991)	.86, .71, and .81
Beliefs about Mental illness Scale (BMI)	Culturally related beliefs about aspects of mental illness	Hirai and Clum (2000)	.81-.91
The Asian Values Scale (AVS)	Adherence to Asian cultural values	Kim,Atkinson, and Yang (1999)	.80-.82
Personal Resource Questionnaire (PRQ20000)	Perceived social support	Weinert and Brandt (1987)	.87-.93
Help-Seeking Questionnaire (HSQ)	Recent life events, barriers or beliefs they inhibited help seeking, specific sources of help	Lin, Goering, Offord, Campbell, and Boyle (1996)

The second aim of this study was to develop ethnographic profiles of the women in the interview phase of the study. In order to explore the variables in the model, we developed the Clinical Ethnographic Interview (EI) (XXXX, 2010). This interview was developed from the Cultural Formulation Appendix I for the Diagnostic and Statistical Manual (DSM) IV (Lewis-Fernandez & Diez, 2002; Mezzich, et al., 1999; Mezzich, et al., 1996), and was validated by the PI in a series of pilot tests. The construction of the interview was carried out using the DSM topic list. In addition, based on the pilot studies, the PI-designed questions about social network, the lifetime history of distresses, and the use of a body map to understand culture and distress in women's lives (for details of this instrument, see XXXX, under review). The interview began with a grand tour of the participant's social network, asking the woman to draw her social network, and the interview discussed the intimacy, hierarchical relationships, shared group affiliations, and reciprocal exchange within these relationships.

The next question asked the woman to depict how she had felt in the last two weeks on a blank body map, which helped us understand the symptoms concept of the CDHS model. Next, the woman was asked to use a sheet that consisted of a blank line across the paper. We asked her to situate major life events along this line as the high points and low points of her life (see Figure 3). We asked the woman to explain her symptoms and their meaning at the most recent low point. We referred to the most recent low point throughout the rest of the interview. The ethnographic interview took about 1-1 ½ hours to complete.

Figure 3.

Life-Line

We also aimed to identify the psychiatric profiles of the highly distressed women in the interviews. We used the Mini International Neuropsychiatric interview (M.I.N.I.) (Sheehan et al., 1998). This interview was developed using both the DSM and the International Classification of Diseases, and designed for international use. It was translated into 34 languages, including Japanese. It was validated using the SCID and the CIDI in two studies, and had Cohen's kappa's of between .45 and .75 (Sheehan, et al., 1998). The screening version of the MINI took approximately 30 minutes to administer. Because we were not confident that the Japanese women in the sample would fit the western defined criterion for all of the mental illnesses, we asked all of the questions on the MINI.

Translation

All research was conducted in Japanese; therefore, the translation and pilot testing of these newly translated instruments required many months of work. The translation procedures are reported elsewhere (XXXXX, under review). However, two women translated every instrument, and we met to examine them for differences. Next, reconciled version of the instrument went to the third member of the team for back translation. Finally, all instruments went to a bilingual expert for review. After all instruments were completed, the entire package was pilot tested with seven women, and their suggested revisions were incorporated. Interviews were transcribed verbatim and then reviewed by the interviewer for accuracy. Translations were completed with a detailed notation system. The interviewer reviewed all translations to ensure accuracy of the meaning of the English.

Data Analysis

The first aim of this study was to examine the demographics, symptom experiences, and sociocultural factors that explain help-seeking strategies, proposing conventional statistical methods to accomplish this aim. In addition, we planned to complete descriptive statistics for demographic variables, the variables derived based on distress screens (CSC and CES-D-K-R, total and subscales) and all model predictors for each cultural group. Categorical variables were summarized into two-way contingency tables and cross tabulating the categorical variables. Continuous variables were described using measures of central tendency and spread. We proposed testing help-seeking mediation and moderation models for symptom cluster groups, while controlling for key demographic variables. To evaluate help-seeking profiles in detail, we proposed to determine the frequencies of Professional Medical (PM), Physical Self-Help (PSH), Traditional Help (TH), Non-Medical Professional (NMH), Professional Psychological (PP) and Social (S) help-seeking strategies reported in the help-seeking questionnaire (HSQ). Help-seeking type sums were used as outcome and mediator and moderator models were proposed for the six (6) help-seeking strategy types. The potential mediator/moderators are the theoretical constructs and measures: Internal experience (CSC and CES-D-K-R); Evaluation of social significance (BMI); Interpretation of cause (SIQ); Group Solidarity (AVS); Hierarchical distances (Dem); Perception of intimacy and social resources (PRQ2000); and Reciprocity Rules and Norms (AVS). By evaluating the relationships among the variables, we tested the PI's Cultural Determinants of Help-Seeking (CDHS) model.

The second aim of the study was to describe the ethnographic and psychiatric profiles of sub-samples of highly distressed women (20 women from each of the recruitment sites). In order to do this, we proposed to examine the ethnographic and psychiatric interview data. We used Analytic Ethnography (AE) as our qualitative analysis strategy because it allowed us the most flexible framework to analyze data using our theoretical model (Lofland, 1995). AE retains the qualitative dimensions of concern in grounded theory (GT), including the coding rigor as well as emergent analysis. However, unlike grounded theory, which relies on theoretical sampling and theoretical saturation, AE provided us with the ability to select a predefined sample. Selecting a sample size had several advantages; allowing us to use estimations of transcription, translation and analysis time, as well as the ability to provide estimates of statistical power for the QUAN aim (see more discussion of method and sampling below). AE also allowed us to qualitatively evaluate the propositions of our model without violating the philosophical assumptions of the research design. During analysis, the AE approach was used to examine the type, structure, process, cause and consequence, frequency, magnitude and agency of the model variables, as well as those concepts specified in the theoretical framework. However, AE is a flexible procedure that allowed us to identify and code heretofore unknown dimensions of the help-seeking process.

The conceptual model facilitated the analysis of the ethnographic data. For example, as noted in Table 1, for the analysis of the social network data, we used some guides to help discover specific examples of key social network and help seeking data, such as levels of intimacy and barriers to exchange. Using sub-concepts such as the level of intimacy, we made sure to look for interview quotations that had qualifiers such as, “we can talk about anything”; “we are very close”; “we keep a kind of distance”; or “she is a superficial acquaintance.”

The lifeline provided us with data regarding both the cultural interpretations of distress, as well as help-seeking actions to relieve it. Our analysis identified the specific symptoms first throughout the narrative. We identified symptoms as all those physical, emotional and interpersonal experiences the participants had during the “low” periods identified in the interview; the perceived level of severity; causal statements; and the social significance of these symptoms in terms of how having that distress changed their relationships with others or how their lives and relationships have altered since they had that distress. The psychiatric profiles included life-time history, current psychiatric diagnoses and assessment of current levels of functioning.

The third aim was the mixed method aim, in which we proposed to compare ethnographic and psychiatric profiles within and across the three groups. We proposed merging ethnographic codes and psychiatric diagnosis data into the larger quantitative dataset. The code categories (listed in Table 1) of the ethnographic analysis were imported into the statistical database, and we used descriptive statistics to examine whether these differ by recruitment group, demographic groups, and by psychiatric diagnosis group. The variables that we coded in the ethnographic analysis included intimacy, exchange type, barriers to exchange, beliefs about exchange, level of severity, causal statements, social significance, help-seeking sources, help-seeking types and their perception of the effectiveness of their help-seeking. We also proposed analyzing how CSC symptom cluster groups and CES-D scores were comparable with or differed from psychiatric diagnoses group for those who entered the interview phase. We proposed to import qualitatively generated codes into the quantitative dataset, and use descriptive statistics to explore patterns between these and help-seeking types.

Assessment of Factors that Contributed to Funding

This study was warmly received by two of the three reviewers. However, evaluation of the reviews for a mixed methods proposal requires examination of not only what they liked, but also why this was fundable. Here, we attempt to break apart the reviewer comments into elements that can be replicated in future studies. There were three main areas that prompted the reviewers to embrace this research project: a research question that clearly required the use of mixed methods; a theoretical framework that allowed a coherent explanation of the data sources, rationale and analysis; and a clear explanation of a feasible study design.

Research questions that require mixed methods to answer

As with any study, the research question must be compelling, and the chosen methods must be clearly appropriate to the research question. In this proposal, we carefully crafted our argument that there was a gap in the literature that mixed methods could fill and we used both QUAL and QUAN research to make the case. In addition, after each identified gap, we demonstrated how both QUAL and QUAN data could lend an answer to the question, why they should both be used, and how each data source informed the other. For example, each variable or phenomenon of the study had some aspects that had previously been studied, and for which we could use a QUAN data source. That same variable or phenomenon also had much that was as yet unknown, and we showed that QUAL data sources could provide additional information about critical aspects of the phenomenon. Using both QUAN and QUAL specific aims, as well as a specific aim that linked the two, we provided the reviewers with a clear idea of how the data would be gathered, analyzed and merged. These specific aims, along with the theoretical framework, provided a meshwork that allowed complex phenomena, and diverse methods, to integrate together as a comprehensive whole.

Clear theoretical framework

The problem of health disparities is complex, and help seeking is a process that includes numerous interacting variables. Theoretical frameworks help reviewers (and the researcher!) organize and understand theoretical concepts drawn from diverse disciplines, remember what affects what, and anticipate how data should integrate together in the end to answer the research questions. Therefore, we used our theoretical framework to organize ALL aspects of the proposal, from the literature review through analysis. The reviewer never lost sight of the problem, specifically how QUAL and QUAN data sources would be used to examine the problem, why we needed the two very different data types, or what we would do with each of the data types. An additional value of using an explicit theoretical framework consistently was the ability to demonstrate the relationship between QUAL and QUAN aspects of the research project with each other. Within a given theoretical construct, we could show the various places in the broader field that were more or less developed, and how QUAL and QUAN data types might be appropriately employed.

Innovative Research Design

The reviewers were clearly convinced that both of the methods could and should be used, and that they could be combined in comprehensible ways. This is partly related to the clear use of the theoretical framework, as well as the review of both QUAL and QUAN studies throughout the background section. However, references to the qualifications of the team to carry out this study also refer to the numerous QUAL and QUAN studies we had conducted in the past, demonstrating our ability to handle both types of data sets. We believe that a mixed method study requires this kind of background work to be able to truly carry out a mixed method study. For example, the PI had carried out two large ethnographic studies, as well as a series of social psychological studies that resulted in QUAN publications. This gave the reviewers detailed information to evaluate not only the composition of the overall team, but also gave qualifications for the PI to lead the diverse training and skill sets needed to carry out the overall study.

Finally, the proposal provided balanced attention to each of the data sources, as well as their analysis. It was helpful to use both the theoretical framework and the clear specific aims to do this. For example, since we had a clear QUAN aim, we could organize our QUAN data sources and analysis separately. Then, for our QUAL aim, we were careful to provide a parallel level of detail for instruments and analysis. Subsequently, in our combined aim, we had the detail we needed to write a compelling mixed method section. Undoubtedly, not all mixed methods studies necessarily can be organized in this way, however, and mixed methodologists must continue to develop ways to write specific aims and design statements that can contain methodological diversity.

We believe that the large sample and the random sampling design were an important feature in a study of this type because this method addresses the transferability of the research to the overall population of Japanese women (and possibly women from other Asian cultures as well). While we, and the reviewers, noted the need for a more heterogeneous sample in order to be representative of the population of Asian immigrants, we acknowledged that a focus on a sub-population was appropriate given feasibility issues. In addition, while it was not stated specifically, we believe that the careful plans for, and a track record of, translation of materials, and the fact that this study was carried out exclusively in the Japanese language, contributed to the overall enthusiasm. The evidence that the team could do this again was demonstrated by the team's preliminary work.

Discussion

This research included here is an example of an integrated mixed method design with parallel data collection and iterative interpretation, with terminal mixing of data. While it was possible to have designed a project that would have used this interpretation to revise measures during the study process, we elected to keep the data collection methods informed by each other, but not revised. This conservative approach was selected by the PI because of the newness of this design at the NIH and the relative unfamiliarity of mixed methods at the NIH at the time of submission. Contemporary mixed methods investigators might propose alternative designs in the current NIH climate that seems ready to welcome innovations. However, clarity about these approaches is critical.

The NIH is interested in exploration of new territories, are seeking work that is innovative, and therefore supports qualitative research. However, as stated above, the urgency of the complex health problems has created a need for research designs that examine multiple levels of phenomenon in the same study. The NIH usually requires that research is driven by theory that can explain phenomenon, allow comparisons, and enhance our ability to intervention testing in large samples. At the risk of stating the obvious, mixed methods proposals should incorporate theoretical frameworks that can leverage the strengths of each method, and assist in casting the approach as one that moves the science toward elimination of health problems. We understand that QUAL analysis can be used to enhance and explicate existing theory, or can even help show how existing hypotheses are incomplete or inaccurate. In addition, QUAL research must retain the flexibility to explore phenomenon outside of the a priori constraints imposed by theory. In order to address the relationship of our QUAL instruments in a theory driven proposal, we used the argument that published and convergent QUAL research about cultural themes or trends could be used to create appropriate theoretical frameworks and QUAN measures, but that there were still critical aspects of the phenomena that were as yet unclear or unknown.. Therefore, while our theory helped us select QUAL questions and initial analytic categories, we used an analysis approach that retained our ability to find new themes and processes.

Most of the 27 institutes and centers at the NIH have their own strategic plans and initiatives. At the time of this writing, most of the institutes are investing in innovations in research methods that will illuminate how health problems are multi-determined. Regardless of the illness or disease state, the NIH is searching for new and better ways to carry out research that will lead to faster development of clinical practices that can improvement health (Dankwa-Mullin, 2010; Ruffin, 2010). However, despite this stated importance of these new research initiatives, there is still no clear understanding about how mixed methods projects should be written and evaluated. According to Plano Clark, the prevalence of funded proposals is still low, representing only a small fraction of the number of proposals funded. National Institute of Nursing Research had the highest number (2.3%) and it was the only agency that had a percentage greater than 1% (Plano Clark, 2010). Readers are encouraged to explicitly link their arguments to NIH strategic plans and specific funding announcements in order to demonstrate how their mixed methods application can move the science forward. Of course, mixing methods is not necessarily sufficient to make a meritorious application. It is the justification that using a variety of data sources, research designs, or data integration will illuminate processes that have been obscured by single methods alone. Palinkas et al reviewed 60 mental health mixed methods funded projects between 2005–2009, and found that most did not provide an explicit rationale for using mixed methods, and most primarily emphasized quantitative methods, rather than truly mixing multiple methoids (Palinkas, Horwitz, Chamberlain, Hurlburt, & Landsverk, 2011).

The second consideration for the use of mixed methods in an NIH application is to understand the grant review study section composition and procedures. A study section is the group or people assembled by NIH to evaluate groups of applications aimed at addressing certain populations or problems. Borken (2004) notes that the mixed method research paradigm is emerging and as yet unknown to most scholars. This makes it difficult for the researcher to cite and justify their design, making it more difficult to get funding (Borkan, 2004). The study section participants are a mixture of seasoned research funding recipients and relatively new recipients with specific expertise. These people vary widely in terms of experience and knowledge about research methods. Some may embrace the dominant methods of their respective fields, while others are more easily persuaded about innovations. In general, however, it is the clarity of the argument that is the important criterion. One of the best ways to determine whether a given application is clear is to have people both inside and outside of the mixed method research community examine applications. Discovering the gaps in one's methodological plan can strengthen an application before it is reviewed.

It is possible to use grounded theory, phenomenology, traditional ethnographic field methods, or any other appropriate QUAL methods in an NIH study. However, estimations of sample size are more or less challenging depending on the QUAL method. No matter what sampling method is chosen, it its essential to address how research with any given sample can adequately answer the research questions, and how that answer can eventually enhance the health of the public. Our research used Analytic Ethnography because it allowed us to use the rigor of coding for emerging themes, as well as to allow us to determine some of our analytic domains a priori, yet retaining flexibility to learn new information. In addition, AE allowed us to predetermine a sample size for budgeting purposes. This was necessary because all of our QUAL data needed to be translated, which made the cost of the QUAL arm of the study especially high. Cultural research has used primarily qualitative methods with small groups. However, cumulative findings across similar groups can lead to theorizing about how cultural dynamics relate to health related outcomes. Mixed methods research can extrapolate from qualitative findings, test these theoretical concepts in large groups, and keep qualitative methods to discover places where new concepts might be needed. In addition, qualitative methods can also fill in the gaps of how known concepts fit together.

We searched for our theoretical constructs from literatures across a wide variety of disciplines, making the task more challenging, but much more fruitful. This approach meets the interdisciplinary goals of the NIH. Using this approach requires researchers to use a wide-angle lens; a method with which the mixed method research must have facility. Once these concepts and relationships are hypothesized, the research can use the appropriate methods to answer the specific research questions. For example, some concepts were less developed and a robust understanding of their full nature has not yet been fully elucidated. In some cases, the concept contained cultural biases, so that clarification of culture nature was required. We made the case that this underdevelopment provided justification for exploration with QUAL methods. Other concepts are more completely understood, with culturally appropriate operationalized and tested measures. These warranted large scale samples to discover how they interact with each other. In our study, many of the concepts had been studies, but rarely together, and the specific mechanisms of interactions were much less clear, warranting measurement with both QUAL and QUAN methods, but for different purposes.

The review sheets implied that the application was clear, comprehensive and understandable. As an NIH reviewer, the clarity of a proposal is critical. Good science is often buried in jargon and needless complexity. Mixed method research projects almost invariably involve complex study designs. As implied above, a theoretical model helps the reader track concepts, and to organize large amounts of complex information efficiently. In our study, we framed the concepts in the background, and then we laid out how each method would be used to examine each concept. This allowed the reader to have a clear picture of what we were studying, why each method was necessary for each concept, specifically how we would deal with the material from each method, and how each method contributed to the overall model. This strategy required creative uses of charts and linking tables, as well as the careful use of the same concepts presented from different angles, always returning to the specific aims. A good example of this strategy was the table presented for the reviewers that linked the theoretical domain, the QUAL analysis categories, the data sources and the sample coding (Table 1 above). This table spelled out what the concept was, where the questions were in the interview, and how we would recognize the concept in the data. This level of detail was helpful for reviewers who were less familiar with QUAL methods or mixing methods.

Of course, as with any NIH application, this application received negative critiques. Concerns about acculturation of the sample, concerns about whether the Beliefs about Mental Illness Scale was a stigma measure, and concerns about the transferability of the findings to other populations dampened enthusiasm. Other concerns were related to the feasibility of our recruitment plan. In addition, we received critique that our sample size estimations were insufficient. One reviewer was especially negative, stating that the use of this mixed method design was not innovative, and that the data did not appear to be “mixed.” These critiques point out the need to clarify the links between assumptions, concepts and measures. It is also critical to clarify how all of the data will be handled, so that no aspect can be left unaddressed. For example, our proposal included almost a page of analysis for out mixed method Aim 3, but this reviewer was not persuaded that the importation of QUAL data into a QUAN dataset for analysis of trends constituted “mixing”. We were also not clear enough that parallel data collection and analysis was itself mixed methods. While we had a detailed sample size calculation, one reviewer challenged our estimates, pointing out that careful selection of team members and exquisite attention to detail are critical. Currently, large grant, RO1 applications are limited to 12 pages. This makes it imperative that material is concise, crisp, clear and compelling, while at the same time training the reviewers about the complexities and nuances of mixed method research.

The challenge that remains for us is to merge the complex data in order to find meaningful patterns across datasets. We have 26 interviews at the time of this writing that contain the qualitative codes, the CESD-K-R and CSC symptoms, the demographics and cultural variables and the psychiatric evaluation data combined into a single dataset. We now are using creative methods to discern patterns from these data, such as QUAL analysis, factor analysis and multidimensional scaling, and allowing the unique types of findings to inform one another. Because we used QUAL analysis alongside the QUAN data gathering, and we analyzed the same concepts from both angles, and we have data from the same women, gathered at the same time. This multiple methods dataset will allow us to use an iterative process of hypothesis generation and testing between datasets that can be done with the full, mixed methods dataset. One hypothesis is that social patterns of help seeking for women might vary by their symptoms patterns, beliefs about mental illness, and beliefs about causation. In our study, we can also look at symptom patterns in the CSC in relation to the psychiatric diagnosis of the same woman. For example, we found higher than expected endorsement of most of the symptoms of “psychosomatic disorder” in both the psychiatric evaluation and in the CSC. We can now look at how these relate to other psychiatric and psychological evaluations of “depression”. We can also examine whether women seek help for these symptoms differently than those who experience more “emotional” forms of distress. Lastly, we can examine all of these symptoms in relationship to causal models of illness both on the Symptom Interpretation Questionnaire, as well as the causal model codes in their interviews.

We believe that the state of the science for culture and health disparities is in mixed method research (Dankwa-Mullan, et al., 2010; Plano Clark, 2010). This study provides examples of how we can construct complex and theory driven NIH applications. Furthermore, we have shown how mixed method research emerges from truly interdisciplinary models of understanding health and illness, and how this responds to NIH priorities. Crafting a proposal like this allows us a vehicle to mobilize all of our diverse methods toward a single question, and interact from diverse perspectives about the answers. We hope this offering will fuel creativity, clarity and precision in such applications. These kinds of studies are expensive but have extremely high yield. Mixed methods research can move us much closer to understanding the relationship between culture and health outcomes.