Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2014 Feb 24.
Published in final edited form as: Stud Health Technol Inform. 2013;192:107–111.

Developing a Survey to Assess Factors that Contribute to Physician Involvement in Clinical Research

Vanina Taliercio a, Judith R Logan b, Jayashree Kalpathy-Cramer c, Paula Otero a
PMCID: PMC3932813  NIHMSID: NIHMS553676  PMID: 23920525

Abstract

Background

Investment in research, including clinical research, has positive effects both on health of a population and economic growth of a country. Several factors have been suggested as being related to the performance of clinical research. The goal of this work was to develop and perform initial validation of a survey that measures both current research involvement of physicians, as well as previously noted factors and additional informatics factors affecting this involvement. The survey was developed in both English and Spanish with the goal of its use in Latin America.

Methods

The initial survey was developed primarily from experience with other validated surveys developed for similar purposes. It was validated in three stages with modification or elimination of questions as indicated by this testing.

Results

The final survey contains 33 questions in the categories of research experience, education in research, environmental factors, computer experience, and collaboration.

Keywords: Biomedical Research, Capacity Building, Medical Informatics

Introduction

Making an investment in clinical research is important because research can lead to benefits in the current and future health of a population, and may also have a positive impact on the economic growth of a country [1]. Despite these well-described positive effects of clinical research, there is limited investment in research in Latin America. Along with this low level of investment, the number of researchers and the number of research projects and publications have traditionally been low [2,3]. The lack of adequate government funding is an obstacle to carrying out research, particularly in terms of retaining qualified researchers in the country [2]. As a result of the low level of government investment in research, a low number of projects are financed and published [4].

Previously published articles have shown a variety of factors to be related to the performance of clinical research, and have found the problem to be complex and multidimensional [5–9]. The purpose of this work was to create and validate a survey instrument that can be used to appraise the current involvement in research of physicians in a geographical region, initially in Latin America. In addition, the instrument should quantitate factors which are believed to be associated with research productivity, including environmental and local factors. The availability of health information systems such as Electronic Health Records (EHRs) and computer literacy should also be included as possible factors associated with involvement in clinical research.

The Institutional Review Board of Oregon Health & Science University gave oversight to and approved this project.

Materials and Methods

In reviewing the literature looking for factors previously described as predictors of physician involvement in clinical research, we discovered that these factors belong to the following themes: physicians’ attitudes toward the use of scientific information, physicians’ skills for doing research, and workplace readiness for research (including institutional support, financial reimbursement, etc.). We based our survey especially on the works from Kagan and associates [10] and Sarre and Cooke [11], and also included questions about physicians’ levels of computer experience [12] and workplace informatics resources. An initial survey containing 74 questions was developed in both English and Spanish; the intent was to initially use this survey in Latin America.

Sarre and Cook in 2009 [11] met a sample of experts in the field in order to develop indicators to measure the readiness for research in primary care organizations. They recognized indicators that measure infrastructure (e.g., posts with research responsibilities, protected time to do research), linkages and partnerships (e.g., mentorships, joint posts with universities), skill development, dissemination, research activity (e.g., number of principal investigators), proximity to practice, continuity and sustainability, leadership, and research culture. After this project they developed “The Organizational Support Tool for Research Capacity Development,” a tool designed to measure the readiness for research in an organization. Because this tool was aimed towards management and administration, we reworded some questions so that it could be applied and understood by physicians.

Kagan and associates in 2009 [10] engaged a broad range of stakeholders to develop a framework for the evaluation of the international HIV/AIDS Clinical Trials Networks. This framework evaluates a number of factors, including setting of biomedical objectives, collaborative communication within and between networks, operations and management, development of policies and procedures, resource utilization, community involvement, and relevance to participants. This framework was specific for this clinical trial network, but because many elements or ideas listed in the framework were shared with the aims of our project, we borrowed many of the concepts.

In addition, questions were added in order to understand the role of informatics resources in readiness for clinical research. Questions were added both on computer experience, for which a number of prior surveys have been developed [12] and also on availability of, and experience with, typical informatics tools.

The compiled initial survey had five themes.

  • The research experience theme intends to measure the current research productivity of the respondent.

  • The education in research theme seeks to quantitate prior education in research, plus the availability of educational opportunities.

  • The environmental factors theme seeks to measure those factors in the respondent’s work environment, however defined by the respondent, that contribute to research involvement. Factors include policies and procedures, the availability of research resources such as funding and mentorship, a scientific agenda and an legacy of collaboration.

  • Computer experience is the fourth theme, including using a computer for various tasks required in research activities.

  • The final theme also involves technology, i.e., the availability and use of collaborative technologies that may contribute to research productivity.

A clinical research informatics expert performed the first review of the survey, evaluating whether the instrument appeared to assess the desired qualities, and also whether the questions covered a representative sample of the domain to be measured.

We next performed iterative cycles of pre-testing the survey, in which we conducted several interviews with Latin American physicians using the Spanish version of the survey. The aim of the pre-testing was to assess:

  1. Meaning (i.e., whether or not each term has the same meaning for the respondent and the interviewer)

  2. Ambiguity (i.e., whether or not the respondents felt that a term might have different meanings)

  3. Comprehensibility of texts and of each survey item (i.e., ease of understanding)

  4. Enhanced language options and appropriate synonyms

  5. Restricted response range (i.e., determining whether a subject’s lack of response to a specific item was because they did not understand the question, or because the subject felt that the instrument did not include an appropriate response)

  6. Average time required to complete the survey

Based on this pre-testing, several questions were reworded for clarity.

Measurement of survey properties

In the second stage of testing, the feasibility of administering the survey and the properties of the measurement itself – endorsement and intra-observer reliability – were assessed. To accomplish this, we conducted a limited test of the survey instrument using a convenience sample of 30 physicians in Argentina. These physicians are similar to the target population.

Feasibility Analysis

As part of this validation study, we examined the question-by-question response rates as a way of assessing the feasibility of the survey as a whole. We investigated the proportion of surveys that were returned, as well as the proportion of questions that were answered completely. The purpose of this was to identify potentially problematic questions; our evaluation outcomes for this phase were the proportion of non-responses (missing or otherwise) for each question, as well as the overall proportion of surveys that were filled out completely (as opposed to being left partially completed).

Intra-observer reliability (test-retest reliability)

We also measured the intra-observer reliability by administering the survey on two occasions separated by an interval of 15 days. Intra-class correlation coefficients (ICC) were calculated.

Endorsement or ceiling effect

We analyzed the frequency of responses for each answer option for each of the questions. We ruled out those questions whose responses where very similar across all of our pilot subjects, as these questions will not help us to discriminate between groups.

Eliminating redundancy

Finally, at the request of the web portal through which we intend to initially administer this survey, we eliminated all questions that were felt by the authors to be similar enough that they could be considered redundant.

Results

From June to August 2011, we performed iterative pre-testing cycles of the survey by conducting several interviews with physicians. at Hospital Italiano, Ciudad de Buenos Aires, Argentina. The pilot study was initiated in September 2011. Figure 1 shows the number of responses to the pilot study.

Figure 1.

Figure 1

Open in a new tab

Flowchart of pilot testing cycle

Feasibility Analysis

A total of 30 surveys, 80 % (n = 24) were returned, 95% (n = 23) of those were fully answered. The only survey that was not fully answered had only one missing answer.

Intra-observer reliability (test-retest reliability)

Intra-observer reliability refers to the ability of an instrument to measure attributes consistently. Using an ICC value of 0.65 or above as a test for reliability, 14 items were felt to be unreliable and were eliminated from the final survey. These low values may indicate either that the scale is unreliable; that the test is reliable but the phenomenon changed over time, for example, with relatively quick changes; or that taking the test previously influenced physician’s responses on the second survey administration.

Endorsement

No questions were eliminated as a result of frequency of responses on answer options. Some trend of answers was seen (e.g., more “strongly disagree” or “disagree” responses than “agree” or “strongly agree”) but the highest rate of response for any question option was only 60%.

The final survey contained 33 questions. Beginning with 74 questions, 14 were eliminated because of unreliability, leaving 60 questions felt to be valid and reliable. Because of the constraints placed on the planned initial widespread survey administration through a Latin American medical web portal, 27 additional questions were eliminated, all of which were felt to have a high degree of redundancy with remaining questions.

Note that no demographic information, although useful for analysis, were included in this survey. This data is available from the web portal through which potential subjects were will be recruited.

The final survey in English is included here as Table 1. The formatted English and Spanish versions are available at http://www.cori.org/clinical.research.survey/.

Table 1.

Survey for Assessing Factors that Contribute to Physician Involvement in Clinical Research

  1. Research Experienceia

    • 1.1

      Since graduation from medical school, how many research presentations have you made at national or international conferences?

      (Options: 0, 1–5, 6–10, 10–15, > 15)

    • 1.2

      How many research presentations have you made at national or international conferences in the last 3 years?

      (Options: 0, 1–2, 3–5, >5)

    • 1.3

      Since graduation from medical school, how many peer-reviewed publications have you had?

      (Options: 0, 1–5, 6–10, 10–15, > 15)

    • 1.4

      How many peer-reviewed publications have you had in the last 3 years?

      (Options: 0, 1–2, 3–5, >5)

    • 1.5

      Including all clinical research activities (e.g. protocol development, research conduct, data analysis, and presentation of findings), how many hours a month do you dedicate to clinical research?

      (Enter number of hours per month)

  2. Education in research. In this section we are asking about the development of appropriate skills for research through training and opportunities to apply those skills.

    • 2.1

      Select the highest level of training in statistics, epidemiology or clinical research that you have received.

      (Options: No courses since medical school in statistics, epidemiology or clinical research, At least one course covering basic concepts in statistics, epidemiology or clinical research, At least one course covering advanced concepts in statistics, epidemiology or clinical research, Masters degree in statistics, epidemiology or clinical research, PhD or other doctoral degree in statistics, epidemiology or clinical research)

    • 2.2

      Please select your level of agreement for each statement about the availability of educational resources in your environment.

      (Options for each: strongly agree, agree, neutral/no opinion, disagree, strongly disagree)

      • 2.2.1

        My work environment facilitates and enables research supervision and mentorship.

      • 2.2.2

        My work environment facilitates or offers clinical research training through scholarships, local graduate programs, or courses in statistics, epidemiology, or clinical research.

  3. Environmental factors. Environmental factors include the structures and processes that are set up to enable the smooth and effective running of research projects and research-related activity

    Note:Your “environment” may be your practice, the practices in your geographical area, a research or practice network, a hospital, or a research institution, i.e. wherever you do or might do research.

    • 3.1

      Policies and procedures

      • 3.1.1

        In your environment, is there a central Institutional Review Board? (Options: Yes, No, Do not know)

    • 3.2

      Resource Utilization. Please select your level of agreement regarding these statements about the resource utilization in your environment.

      (Options for each: strongly agree, agree, neutral/no opinion, disagree, strongly disagree)

      • 3.2.1

        Investigators in my environment have adequate qualifications and experience to perform research.

      • 3.2.2

        Adequate protected time to do clinical research is provided in my environment.

      • 3.2.3

        Local scientific results are published and disseminated widely.

      • 3.2.4

        Research projects in my environment are funded by national institutions.

      • 3.2.5

        Research projects in my environment are funded by international institutions.

    • 3.3

      Scientific Agenda. Please select your level of agreement regarding these statements about the scientific agendas in your environment.

      (Options for each: strongly agree, agree, neutral/no opinion, disagree, strongly disagree)

      • 3.3.1

        The selection of a research topic follows national health priorities.

    • 3.4

      Collaboration and communication. Please select your level of agreement regarding these statements about collaboration and

    • communication for research in your environment.

      (Options for each: strongly agree, agree, neutral/no opinion, disagree, strongly disagree)

      • 3.4.1

        There is good communication and collaboration in research between investigators in my environment.

      • 3.4.2

        There is good communication and collaboration in research with other research centers.

      • 3.4.3

        There are good support services in my environment to help with clinical research, for example, statistics services or clinical research services.

  4. Computer experience

    • 4.1

      Please select the informatics tools available at your place of work:

      (Options: Electronic Health Records (EHRs), Admission and or scheduling system, Patient or Disease Registries, Clinical decision support system, Clinical Trials Management System, Statistics software like SPSS, Stata or Epi Info, Other) with text requested for “Other”

    • 4.2

      In a typical week, how many hours do you personally use a computer?

      (Enter number of hours per week)

    • 4.3

      How would you rate your computer skills?

      (Options: Unable to use computer, Unskilled at using a computer, Moderately skilled in using a computer, Very skilled at using a computer, Expert in using a computer)

    • 4.4

      To what extent do you use a computer for each of the following professional tasks?

      (Options for each: Always use a computer for this task, Often use a computer for this task, Sometimes use a computer for this task, Perform this task but never use a computer, Never perform this task)

      • 4.4.1

        Obtaining advice on a specific patient's diagnosis or therapy.

      • 4.4.2

        Writing (grants, research papers, teaching materials).

      • 4.4.3

        Preparing presentations.

      • 4.4.4

        Performing statistical analysis on clinical research data.

      • 4.4.5

        Searching medical literature.

      • 4.4.6

        Teaching.

  5. Collaboration

    • 5.1

      To what extent do you use the following technologies for professional communication and collaboration?

      (Options for each: I use this tool daily or almost daily, I frequently use this tool, I use this tool sometimes, I rarely use this tool, I never use this tool)

      • 5.1.1

        Group calendaring (for example, Google calendar, Outlook group calendar).

      • 5.1.2

        Tools for coordinating meetings (for example, Doodle).

      • 5.1.3

        Digital video for communication (for example, Skype).

      • 5.1.4

        Audio conferences.

      • 5.1.5

        Discussion boards.

      • 5.1.6

        Wikis or similar tools for collaborative online work (for example, Google Docs).

Open in a new tab

Discussion

For this survey, we sought out the factors associated with clinical research with the goal of survey administration in Latin America. The survey is intended to be broad and drew on prior surveys intended for other audiences. In addition, questions specific to informatics competencies and support were added as an important component of the research process. The survey instrument was tested in several phases and questions that did not have high test-retest reliability were eliminated.

The resulting survey is novel in that it has been validated in Spanish and because of the broad range of questions, including educational experience, community/government support, institutional factors, informatics factors and computer experience. Validation was performed with a population representative of the intended population.

While testing in this population may limit the generalizability, the main limitation of this survey is that 40% of the initial questions had to be eliminated in order to administer the survey broadly through the chosen method. Retesting for reliability and endorsement was not performed again after this reduction in survey length, but there is little reason to believe that these factors would be any different in the smaller survey.

Future work will include administration of this survey in Latin America with further validation in this specific population. Aims will be to characterize the population of Latin American physicians based on their involvement in clinical research, and to apply principle component analysis to determine components that correlate with these populations. We will in particular be interested in the relationship of informatics skills and resources to involvement in clinical research. The long-term hope is to suggest and support resources that can improve the clinical research environment in this region.

Acknowledgments

This work was funded in part by a grant D43TW008439 from the Fogarty International Center, National Institutes of Health, William R. Hersh, MD, PI (Drs. Taliercio, Logan and Otero).

References

  • 1.Frank C, Nason E. Health research: measuring the social, health and economic benefits. CMAJ. 2009 Mar 3;180(5):528–34. doi: 10.1503/cmaj.090016. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.Thorn K. Science, Technology and innovation in Argentina [Internet] WB Working Paper. 2005 Available from: http://siteresources.worldbank.org/INTARGENTINA/Resources/ScienceTechnologyandInnovationinArgentina.pdf.
  • 3.Karlberg J. Biomedical Publication Trends by Geographic Area. Clinical Trial Magnifier. 2010 [Google Scholar]
  • 4.Man JP, Weinkauf JG, Tsang M, Sin DD. Why do some countries publish more than others? An international comparison of research funding, English proficiency and publication output in highly ranked general medical journals. Eur J Epidemiol. 2004;19(8):811–7. doi: 10.1023/b:ejep.0000036571.00320.b8. [DOI] [PubMed] [Google Scholar]
  • 5.Askew D, Schluter PJ, Claravino AM. Changing GPs’ attitudes to research - do N of 1 trials hold the key? Aust Fam Physician. 2008 Jul;37(7):578–82. [PubMed] [Google Scholar]
  • 6.Jowett SM, Macleod J, Wilson S, Hobbs FD. Research in primary care: extent of involvement and perceived determinants among practitioners from one English region. Br J Gen Pract. 2000 May;50(454):387–389. [PMC free article] [PubMed] [Google Scholar]
  • 7.Askew DA, Clavarino AM, Glasziou PP, Del Mar CB. General practice research: attitudes and involvement of Queensland general practitioners. Med J Aust. 2002 Jul 15;177(2):74–77. doi: 10.5694/j.1326-5377.2002.tb04670.x. [DOI] [PubMed] [Google Scholar]
  • 8.Lester HE, Carter YH, Dassu D, Hobbs FD. Survey of research activity, training needs, departmental support, and career intentions of junior academic general practitioners. Br J Gen Pract. 1998 Jun;48(431):1322–6. [PMC free article] [PubMed] [Google Scholar]
  • 9.Greenwood J, Gray G. Developing a nursing research culture in the university and health sectors in Western Sydney, Australia. Nurse Educ Today. 1998 Nov;18(8):642–8. doi: 10.1016/s0260-6917(98)80062-2. [DOI] [PubMed] [Google Scholar]
  • 10.Kagan JM, Kane M, Quinlan KM, Rosas S, Trochim WMK. Developing a conceptual framework for an evaluation system for the NIAID HIV/AIDS clinical trials networks. Health Res Policy Syst. 2009;7:12. doi: 10.1186/1478-4505-7-12. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Sarre G, Cooke J. Developing indicators for measuring Research Capacity Development in primary care organizations: a consensus approach using a nominal group technique. Health Soc Care Community. 2009 May;17(3):244–53. doi: 10.1111/j.1365-2524.2008.00821.x. [DOI] [PubMed] [Google Scholar]
  • 12.Cork RD, Detmer WM, Friedman CP. Development and initial validation of an instrument to measure physicians’ use of, knowledge about, and attitudes toward computers. J Am Med Inform Assoc. 1998 Apr;5(2):164–76. doi: 10.1136/jamia.1998.0050164. [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES