Mixed methods and survey research in family medicine and community health

Abstract

Many family medicine and community health researchers use surveys as an original research methodology. Our purpose is to illustrate how survey research provides an important form of quantitative research that can be effectively combined with qualitative data to form a mixed methods study. We first provide an overview of the key principles in survey research and in mixed methods research. We review the various ways that survey can be used in mixed methods studies, citing options such as beginning a study with a survey, using a survey as the second form of data collection, or combining a survey and a form of qualitative data in a single data collection procedure. Finally, we illustrate in a specific example six steps in conducting a mixed methods study using survey research. In a mixed methods study using a survey, primary care researchers should consider six steps. Step 1. Articulate the rationale for mixed methods study. Step 2. Detail quantitative and qualitative databases. Step 3. Identify a mixed methods design. Step 4. Analyse and report the results of the quantitative and qualitative databases. Step 5. Present and show integration. Step 6. Explicate the value of using mixed methods. The ability to combine and integrate survey research into a mixed methods study provides a more rigorous approach to research than conducting only a survey or conducting just a qualitative interview. While requiring skills beyond traditional survey approaches, surveys in primary care offers an opportunity for a high level of sophistication in research methodology.

Introduction

Many primary care researchers consider the implementation of a survey as means for original data collection. While there are many resources guiding the conduct of a survey, these resources typically have not been written with primary care researcher in mind. Due to many of the complex questions that arise, primary care researchers are often interested in assessing a phenomenon both quantitatively and qualitatively.

Both survey methods and mixed methods research are distinct methodology approaches in the health and social sciences. However, they can be combined in a single mixed methods study with appropriate planning and thought about their combined use. The purpose of this article is to present an applied discussion about how to develop surveys (or questionnaires) and use them in a mixed methods study. To this end, we will first discuss the basic principles involved in mixed methods research, highlight the essential characteristics of survey research and end with a discussion about the steps for using surveys in a mixed methods study. We will further explain the steps using a published mixed methods study in the health sciences.

Basic principles of mixed methods research

Founding

Mixed methods research as we know it today began between 1985 and 1990. At that time, several individuals founded this new methodology. They came from various fields and countries, such as evaluation, management, sociology, medicine and education.¹ They were writing from Great Britain and the USA, and thus mixed methods can be seen as primarily an Anglo-American invention. They were not in touch with each other, and perhaps the most well-known writers were Greene from the USA and Bryman from England. Their impetus for developing mixed methods was a perceived need that both quantitative and qualitative research had value, and there was no need to keep the two forms of social and health research separate. This is despite differences in philosophical stances between the quantitative and qualitative researchers within social science sociology, anthropology, education, and evaluation perspectives and those with epidemiological orientations.

Designs and philosophies

Over time, probably during the mid-1990s, the idea began to form that mixed methods was collecting and analysing both quantitative and qualitative data, and also that additional insight might be gained from combining or integrating the two databases and linking them in a creative way. Thus, qualitative research explores phenomenon while quantitative explains the results of tests of hypotheses or research questions. Combined, mixed methods provides the insight of both exploration and explanation. By 2003, in the Handbook of Mixed Methods in Social and Behavioral Research,² authors advanced specific designs or procedures for conducting mixed methods research. This was not an unusual development in research methodology in that specific designs were well known in types of quantitative research, such as experimental research,³ and in forms of qualitative research such as grounded theory.⁴ At the same time, individuals were developing an increased understanding of the philosophical underpinnings of mixed methods research that have unfolded in approaches such as pragmatism, critical realism, dialectic pluralism and yin/yang Eastern philosophy.⁵ These philosophical assumptions advanced the key idea that researchers bring to their mixed methods study core assumptions or beliefs that shape the types of procedures used in their studies. These twin developments—the designs or procedures and the philosophies—have led us to today a complete methodology called ‘mixed methods research’. Thus, to characterise mixed methods is to consider how it spans the process of research that includes a philosophy, an orientation to research problems and questions, specific approaches to collecting and analysing quantitative and qualitative data, and implications from the research that provides additional insight beyond what might be gained from simply collecting and analysing quantitative data or qualitative data.

Integration and basic characteristics

Further, mixing or ‘integration’ of the two forms of data has become a central feature of mixed methods research. Integration has developed as the buzzword for the innovative feature of mixed methods research, and it provides insight beyond what is learnt from the quantitative and qualitative databases separately. For example, the ability to compare the results of both databases provides a more complete understanding than either database alone. This procedure is called a ‘convergent mixed methods design’. It provides the opportunity to check one database against the other (how do people respond when they rate questions on a questionnaire vs when they are asked in person in an interview?).¹ Insight can emerge from collecting survey data initially and then following up with interviews to help explain the survey results in more detail. This type of design is call an ‘explanatory sequential design’. Insight can also emerge from exploring first with interviews to understand the culture and specific perspectives of individuals, and then designing questionnaires or experimental interventions that respond to these cultural features (called an ‘exploratory sequential design’). A new feature to emerge in mixed methods is to think beyond these insights that come through integrating these ‘core designs’ into more complex procedures, such as experimental trials, social network analyses, evaluation procedures or into community-based health practices. These designs are called ‘complex mixed methods designs’.¹ In short, what has evolved in recent years is an understanding of the key characteristics of a well-designed mixed methods study as shown in box 1.

Box 1. Key characteristics of a well-designed mixed methods study.

• Researcher collects and analyses both quantitative and qualitative data.

• Researcher engages in rigorous procedures with both databases, such as systematic sampling, adequate sample size, use of good instruments and protocols, multistep analysis and standards of qualitative such as validity, replicability, generalisability and accuracy of the findings.

• Researcher combines (or mixes) the two databases in a systematic procedure called ‘integration’, where the overall intent is to bring the databases together, and to conduct analysis that accomplishes this aim and provides new insights.

• Researcher conducts the integration within a type of mixed methods design, such as converging or comparing the responses, explaining the findings of one phase, or exploring first before measuring or assessing perspectives to build in a contextual, cultural feature of the study.

• Researcher frames the study within larger philosophical assumptions, beliefs or orientations that the researcher brings to a study and incorporates into the study design a theory or conceptual model that informs what the researcher hopes to learn from the study.

Understanding the etiology of mixed methods research, how integration operates and the key characteristics of a good mixed methods project are essential understandings that lead to a contribution of mixed methods in family medicine. For researchers in the health sciences, the ‘Best Practices in Mixed Methods in the Health Sciences’ from the National Institute of Health⁶ is a useful guideline for understanding an applying mixed methods research.

Basic principles of survey research

History

Survey research is a quantitative approach to social and health science research. Survey research designs are a set of research procedures in which investigators administer a survey to a sample or to the entire population of people to describe the attitudes, opinions, beliefs, perceptions, behaviours or characteristics of the population. In this procedure, survey researchers collect quantitative, numbered data using questionnaires (eg, mailed questionnaires) or interviews (eg, one-on-one interviews) and statistically analyse the data to describe trends about responses to questions and to test research questions or hypotheses. Researchers also interpret the meaning of the data by relating results of the statistical test back to past research studies. Surveys typically involve quantitative items, but researchers might include qualitative open-ended questions as well.

Survey research evolved as a research methodology during the 20th century. During World War II, for example, US surveys examined issues central to the war effort, such as the morale of soldiers, production capacity for weapons and the effectiveness of strategies. Through these studies, survey researchers refined and developed their techniques of large-scale assessments, enabling the emergence of large social research organisations in American universities after the war. For example, investigators established social research centres at the University of California, Berkeley (Survey Research Center), at the University of Chicago (National Opinion Research Center) and at the University of Michigan (Institute for Social Research). In addition, opinion polling organisations, such as Gallup, Roper and the RAND Corporation, furthered the understanding of large-scale data collection. The founding of polling and survey organisations, combined with the use of computers, the availability of data archives and storage, and funding from the federal government, helped to establish the popularity of surveys in research by the mid-20th century. In recent years, both federal and state governments have funded national and state surveys. Recently, individuals have increasingly used the Internet to collect survey data. Researchers can now generate an online survey to easily administer to anyone with Internet access.⁷

Types of surveys

Surveys can be conducted over time, longitudinal surveys, or administered at one point in time, cross-sectional surveys. In mixed methods research, cross-sectional surveys are frequently used. The key characteristics of good survey work include several features. The participants who fill out a survey are individuals in a specific population (eg, individuals with chronic heart failure). Survey researchers then select a sample from this population to identify individuals to complete the survey. The most rigorous form of survey sampling is probability sampling, where each individual in the population has an equal chance of being selected. The idea, then, is that the researcher draws conclusions from this sample to make inferences about the entire population. Thus, it is not necessary to sample the entire population. However, the size of the sample receiving the survey is important, and it is useful to select as large a sample as possible. Sample size tables in survey texts help identify the appropriate number.⁸

Different forms of surveys exist, and they can be broadly grouped into questionnaires and interviews. A questionnaire is a form used in a survey design in which participants in a study complete a mailed instrument and return it to the researcher. The participant chooses answers to questions and supplies basic personal or demographic information. An interview survey, however, is a form on which the researcher records answers supplied by the participant in the study.

A survey researcher does have the choice of different types of questionnaires and interviews: a questionnaire mailed to the participant to be filled out, a questionnaire administered online through the Internet or through email, an interview used in a one-on-one interview or with a small focus group, or a telephone interview.⁵ Of these types, mailed questionnaires are the most popular form used in mixed methods research. Also, increasingly popular is the use an Internet survey with a software program such as Qualtrics⁹ or Survey Monkey.¹⁰ In the health sciences, focus groups are frequently used, and in these the researcher identifies an instrument to record data, convenes a small group of individuals (ie, typically six) and holds a discussion about questions on the instrument.¹¹

Use of a survey and common problems

In terms of a survey instrument to use, researchers can develop their own survey instrument (requiring skills in scale development and design), use an existing instrument or modify (with the authors’ permission) an existing instrument. If the researcher chooses to design their own instrument, the instrument will involve different types of attitudinal or behavioural questions, the instrument needs to have psychometrically rigorous question construction and the instrument should to be pilot-tested with a few individuals before its general administration. In a pilot test, the researcher administers the instrument to a small group of participants, asks them to comment on any weaknesses in the instrument, and then modifies or changes the instrument. Further, conducting cognitive interviews of the items to ascertain that the survey participant interprets the meaning of questions as intended is important. The types of questions the instrument will include closed-ended questions with items that require a check for the most appropriate answer, such as strongly agree, agree, undecided, disagree and strongly disagree. Also, the instrument may include open-ended questions which will result in collecting qualitative data from questions that permit the participant to give short answers. Common problems in constructing these questions include using vague words, asking multiple questions in a single question, writing wordy or lengthy questions with many parts, using negative or jargon language, having response categories overlap or be unbalanced, and having a mismatch between the question and the responses.

Analysing a survey

It is useful to consider the major components of a good mailed survey instrument and how it is analysed.⁵ It should include a cover letter to the participants asking for their input, have an introduction to the survey stating the reasons for the survey, include questions of a length that could be reasonably answered by the participants and have closing instructions thanking the participant for their help in providing data. The questions or items on a survey instrument need considerable thought. Basically, four to five items can be grouped into a variable that measure attitudes and behaviours. Then these variables can further be grouped into scales. An instrument thus would contain several scales. Knowing this, the data analysis steps on data collected on an instrument can now be identified as illustrated in table 1.⁵

Table 1.

Steps during the analysis of survey data

Steps	Explanation
1. After receiving the instruments back, the researcher needs to be concerned about the size of the response rate.	This means that the participants need to be notified several times to complete the instrument, including often a second mailing of the instrument to gather data. Most importantly is the concept of response bias—whether the responses received are biased in a certain way based on when the response are returned. Several ways to check to see if responses are biased include monitoring the responses as they are returned to see if the viewpoints differ depending on the early versus late responses. Also, follow-up phone calls can be made to those who do not respond to determine if their responses were significantly different than those who did respond.
2. When all responses are in, the researcher enters the data into a computer program such as SPSS and needs to review the data and ‘clean’ it for responses that are obvious errors. It also involves a preliminary analysis of the data.	When these are corrected, the researcher then conducts a descriptive analysis of all of the answers to note the means, SD and ranges of the scores to each item.17
3. This step is followed by grouping the items into scales that had been determined prior to the administration of the instrument.	Further checks then can be made to examine the reliability of the scales to see if the items determined to group into a scale provide a meaningful scale.
4. The researcher conducts inferential analysis to look at the relationship among the variables/scales and to compare different personal variable characteristics and the variables/scales.	In addition, the researcher may want to compare groups in terms of variables/scales. These analyses help to answer the research questions posed at the beginning of the study.

Various ways surveys can be used in mixed methods research

Up to this point, we have discussed the basic ideas involved in both mixed methods research and survey research. Both of these can be viewed as distinct methodological approaches for a study in the health sciences and they need to be rigorously conducted. The options for researchers to bring surveys into mixed methods research can be seen in table 2.

Table 2.

Options for using survey in a mixed methods research investigation

Approach for using a survey in a mixed methods study	Explanation
As the initial data collection in a mixed methods study	Surveys, as a quantitative approach to research, can begin a project and then be followed up by open-ended data collection such as focus groups. In this way, the researcher can explore further the results of the survey to drill deeper into the data.
As a follow-up data collection in a mixed methods study	Surveys can be developed in a mixed methods study where the researcher first collects qualitative data through forms such as focus groups, then develops a survey or modifies an existing instrument from what is learnt in the focus groups, and finally administers the survey instrument. In this way, the survey is suitable for the participant and is context specific.
As confirming evidence in a mixed methods study	Surveys can be collected at the same time as the qualitative data, such as focus group, and then the responses can be compared. In this way, the researcher-directed quantitative survey and be compared with the participant-directed qualitative data so that a more complete understanding results.
As an adjunct to larger processes in research	Surveys can form the quantitative data collection in a larger process that also involves collecting qualitative data. For instance, in evaluating a programme or an experimental intervention, a survey can be used to measure outcome variables. When combined with qualitative data, to assess the process individuals’ experiences, the study becomes mixed methods. As another example, in an experimental intervention trial, survey data can be collected during the trial to assess pre-test and post-test results. After the trial, qualitative data can be gathered to understand the trial results in more detail. This configuration becomes a mixed methods study that combines or integrates a quantitative trial with a qualitative follow-up. Given these variations, what is the process of adding survey research into mixed methods?

Example of using survey research in a mixed methods study

We will use an explanatory sequential design in mixed methods because this is a design that honours a strong quantitative initial data collection, is easy to administer and is one often used by beginning mixed methods researchers. Our specific example would be the study by Sonnenberg et al ¹² on an assessment of resident physicians’ communication and collaboration competencies by an interprofessional (IP) team (clinicians, pharmacists, laboratory technicians, therapist and others).¹² This is a study in competency-based medical education where there is increasing emphasis on assessing resident performance in abilities to communicate and collaborative effectively in the workplace. This assessment can take place by non-physician members on IP teams, and such teams who have valuable expertise have not been adequately engaged in resident skill assessment.

Thus, the authors in the Sonnenberg et al ¹² study used an explanatory sequential mixed methods design to examine the ability of the IP clinicians to provide feedback to pediatric residents during their rotation.¹² Using survey research in the first phase, the researchers compared IP supervisors and physician supervisors in terms of communication and if collaborative training objectives were met. These objectives were based on the widely used CanMEDS Roles Framework developed by the Royal College of Physicians and Surgeons of Canada. In this Framework, seven essential skills for medical practice were identified (eg, communicator, collaborator, leader, health advocate, scholar, professional and expert). Then, in the second phase, the researchers conducted follow-up qualitative focus groups to probe a contextual understanding of the factors that influenced the process of assessment. Thus, the purpose of this mixed methods study was to examine IP clinicians’ perceptions of their ability to provide formative feedback compared with physician supervisors on the rotation and to qualitatively explore potential barriers to the feedback process.

What were the steps they engaged in to conduct this study? These steps could be applied to many mixed methods projects using survey research. See table 3 for information from the Sonnenberg et al example.¹²

Table 3.

Steps in a mixed methods survey investigation as illustrated by the Sonnenberg study

Steps in the investigation	Illustration from Sonnenberg et al 12
Step 1. Articulate the rationale for mixed methods study.	In the resident physician competencies mixed methods study,12 the authors state that it was important to understand the context. Because they were studying the behaviours of residents, they state that subjective experience and perceptions could not be fully captured in the quantitative survey. This rationale for the use of mixed methods was located in the section of the methods where the authors first introduce the qualitative methods.
Step 2. Create the quantitative and qualitative databases.	In the resident physician competencies mixed methods study, rigorous procedures were used.12 The quantitative data consisted of an on-line Adobe Survey sent to MDs and to interprofessional (IP) clinicians (N=45). The characteristics of the sample was reported in a table. The online survey was carefully described in terms of the learning objectives, the scales used and the coding procedure. Frequencies, means and SD were calculated for each item, and the items were clustered into two variables, one on observable skills and one on assessable skills. T-tests were used to compare the MDs with the IP team responses. The qualitative data consisted of an interview guide administered to focus groups. The interview guide was based on the survey results. Qualitative data analysis consisted of theme analysis proceeding through several steps to generate the themes.
Step 3. Identify a mixed methods design.	In the resident physician competencies mixed methods study, the authors announced early in the study abstract that the explanatory sequential design was used in the study. A diagram was not presented that would be helpful in understand the study. The authors do say that the qualitative interviews built directly from the quantitative data collection and analysis. Further, the explanatory sequential design is not defined for the reader, a definition that would have been helpful for beginning researchers.
Step 4. Analyse and report the results of the quantitative and qualitative databases.	In the resident physician competencies mixed methods study,12 the report of the results does mirror the design with the quantitative results first followed by the qualitative findings. We find that there was no significant difference between the MDs and the IP clinicians in terms of observable and assessable skills. For the qualitative results, a table shows an example of one theme with codes and quotes. Four themes became headings in the qualitative results discussion.
Step 5. Present and show Integration.	In the resident physician competencies mixed methods study,12 the authors only mention the value of collecting qualitative data to develop a contextual understanding of the survey results. There was no joint display that would have shown how the authors used the survey results to develop the qualitative interview guide. Thus, integration lies largely hidden in this project although we know that the quantitative survey built into the qualitative data collection.
Step 6. Explicate the value of using mixed methods	In the resident physician competencies mixed methods study,12 it is mentioned that the qualitative data added insight into the differences between the MDs and the IP clinical teams as they viewed the skills of residents. The culture did shape the expectations placed on the residents.

Step 1. Rationale for mixed methods. Determine if mixed methods research is the best methodology to use to answer your research questions. Provide a reason for the use of mixed methods in the project.

Step 2. Quantitative and qualitative databases. Identify the types of quantitative and qualitative data to be collected and analysed. Consider these two types of data as distinct databases. In a mixed method research study, it is important to collect and analyse both quantitative and qualitative data. Use rigorous survey procedures and rigorous qualitative procedures.

Step 3. Mixed methods design. Determine what mixed methods design you will use. Draw a diagram of the design. Mixed methods researchers like to have a visual picture of their procedures. The diagram needs to be simple and straightforward without complicated components.

Step 4. Analyse and report the results of the quantitative and qualitative databases. Present the results of the study showing the quantitative statistical results and the qualitative thematic results. Keep these discussions of analysis separate under distinct headings. They should also mirror the steps in the design, and show a clear linkage between the quantitative and qualitative components in the study.

Step 5. Present and show integration. Pay attention to the point of integration of the two databases and make it specific in the diagram of the design. Use an arrow in the diagram to show the point of integration of the two databases. Discuss the ways integration appears in the study. There are two major ways of representing integration. One is to simply discuss the quantitative and qualitative results side by side in a discussion. The other way is becoming popular in mixed methods research: a joint display. A joint display is typically a table in which the quantitative and qualitative results appear side by side. In this way, comparisons can be made between the quantitative and qualitative results. In an explanatory sequential design, the quantitative results are first reported in a first column, and the qualitative results that help explain the quantitative results appear in a second column. Further, a third separate column shows the impact of the integration in the study. A template is useful for considering the type of joint display that can emerge using an explanatory sequential design as an example.

Step 6. Explicate the value of using mixed methods. In a discussion section at the end of the study, include comments about the value of using mixed methods as a methodology. In this way, readers of the study will see the utility of mixed methods in research. This means drawing the implications of using both quantitative and qualitative data in a mixed methods study.

Additional resources to consider

In addition to Creswell and Creswell, when conducting mixed methods survey studies, it is important to study survey research and consult books on it by leading writers such as Babbie, Fowler and Dillman.5 8 13 14 In mixed methods, consider introductory mixed methods texts by Plano Clark and Ivankova, Creswell and Creswell, Plano Clark, and Creswell, and Guetterman as important resources.^{1 15–17} Look for practical guides to both survey and mixed methods research.

Discussion

The illustration of using a survey in mixed methods based on an explanatory sequential design is simply one possibility of using surveys. However, it provides an opportunity to see how a survey fits into mixed methods. In order to conduct mixed methods research, investigators need to know the basics of both mixed methods research and survey research, and recognise the various ways the two methodologies can be combined. Further, this combination is a rigorous design that takes time for both the quantitative and qualitative components. Some researchers may find it more economical and time-saving, to employ only quantitative or qualitative research. Knowing how to effectively combine the two methods requires knowledge across multiple research methods. Still, surveys combined with mixed methods research leverage two useful approaches.