Abstract
A team approach in health care involves an interprofessional approach to patient care. We wanted to develop and validate a tool that would evaluate the interprofessional team approach to patients of a family medicine team. We performed a descriptive study in three consecutive phases: a literature review, consensus development panels, and a cross-sectional validation study. Three rounds of consensus development panels were carried out in order to evaluate and adapt the initial scale. The cross-sectional study was carried out in all Slovenian family medicine practices, each invited 10 consecutive patients. In the quantitative study, 3,292 patients participated (a 50.7% response rate), of which 1,810 (55.0%) were women. The mean age of the sample was 53.1 ± 1.2 years. The final Cronbach’s alpha was 0.901. A factor analysis of the 9-item scale put forward two factors (Team Approach and Person-Centred approach) which explained 68.6% of the variance. This study provided a new scale for the evaluation of patient satisfaction with the interprofessional family medicine team from the patients’ point of view. It opened the question of family medicine team competencies and pointed towards the need to develop a family medicine interprofessional team competency framework and a comprehensive tool for its assessment.
Introduction
Person-centred care is one of the main competencies of family medicine [1] and encompasses the ability to provide longitudinal continuity of care as determined by the needs of the patient, referring to continuing and coordinated care management [2]. Person-centred care requires interdisciplinary and interprofessional teamwork in order to enable the management of the patient as a whole person [3]. Teamwork is defined as the provision of health services to individuals, families, and/or their communities by at least two health providers who work collaboratively with patients and their caregivers—to the extent preferred by each patient—to accomplish shared goals within and across settings to achieve coordinated, high-quality care [4]. Interdisciplinary team work is a complex process in which different types of staff work together to share expertise, knowledge, and skills to impact on patient care [5].
A team approach involves an interprofessional and coordinated approach to patient care, and requires clear definitions of the individual team members’ roles. A team approach to patients in family medicine has been shown to achieve better outcomes, a higher quality of care, worker satisfaction, and cost containment [6].
The theoretical framework that describes an interprofessional team-based primary care includes four domains: cognitive domain, affective/relational domain, behavioural domain, and leadership. There are almost 50 instruments that measure interprofessional team-based primary care according to the mentioned framework. Mostly, these are surveys, focusing on team member, and include at least three of the framework’s domains [7].
In 2011, an ongoing project at the primary care level in Slovenia was launched, with the support of the Ministry of Health. It introduced a new model of family medicine practice, where a family physicians’ working team, which at that time consisted of a family physician and a practice nurse, was extended by a nurse practitioner, with very well-defined competencies, working four hours per day or 0.5 full-time equivalents [8]. This approach implemented an interprofessional team practising person-centred care in Slovenian family medicine [8–12]. As a new team member was added to the family medicine practice team within the project, we developed a tool to measure patient satisfaction with the nurse practitioners, which evaluates their clinical approach, comprehensive approach, and patient-centred approach [9].
The limitation of the tools used to evaluate patient satisfaction in the Slovenian family medicine practices involved in the project was that they evaluated the experience of patients with one member of the team, not with the team as a whole. This was not in line with the interprofessional team-based nature of this project.
We therefore decided to develop a tool which would assess the interprofessional family medicine team approach from the patients’ viewpoint. This approach to patients is based on a theoretical framework of the patient management in Slovenian family medicine practices described elsewhere [13].
Materials and methods
Research design
We carried out a descriptive study in three consecutive phases: a literature review, consensus development panels, and a cross-sectional validation study (Fig 1). Each subsequent phase was informed by and built on the preceding phase.
Fig 1. Flow-chart of the scale development.
This flow-chart describes a step-wise development of the new interprofessional team approach scale.
Data collection
We carried out a narrative literature review in order to identify existing tools for the evaluation of the team approach to patients in primary health care. We used the following key words: Teamwork; Team Work; Interprofessional Team-based Approach; Family Practice; Family Medicine; Primary Health Care; Primary Healthcare; Patient Satisfaction. We searched in the following databases: MEDLINE (1966–2016), ISI Web of Science (1970–2016), ProQuest (1980–2016), EMBASE (1980–2016), SCOPUS (1960–2016), Springer Link (1980–2016), Sage Journals (1980–2016) and CINAHL (1982–2016) (Box 1).
Box 1. Full electronic search strategy used in the narrative review
#1. Teamwork/Team Work
#2. Interprofessional Team-Based Approach
#3. Family Practice/Family Medicine
#4. Primary Health Care/Primary Healthcare
#5. Patient Satisfaction
#6. (#1 OR #2 OR #5) IN #3 OR #4
Two researchers independently reviewed the scales identified through the process described above and prepared an initial scale, with items based on the identified scales. The items were translated to Slovene language by a standard forward-backwards translation procedure [14].
Two rounds of consensus development panels were carried out in order to evaluate and adapt the initial scale. Experts from family medicine (N = 4) and nursing (N = 4) participated in these panels. These experts were very familiar with the theoretical framework of the patient management in Slovenian family medicine practices [13]. Both panels took place in face-to-face meetings in 2016. During the first meeting, the initial scale was reviewed, and the experts gave their opinions and suggestions. During the second meeting, the final content of the scale in Slovene language was approved.
A cross-sectional study was carried out in January 2017 in Slovenian family medicine practices. All the family medicine practices in Slovenia that had implemented the new approach to patients [13] described in the introduction were invited to participate (N = 649). Each family medicine practice invited 10 consecutive patients who attended the practice on the same day to participate in the study. Patients that were unable to fill in the questionnaire (e.g. due to blindness, cognitive impairment, or terminal illness) or who did not want to participate or were younger than 18 years were excluded from the study. One member of the practice team explained to the patients the aim and method of the study and invited them to participate. They gave an oral informed consent. They filled-in the questionnaire in the waiting room and put it in the sealed box afterwards to secure the confidentiality of the data.
The questionnaire consisted of the newly developed 10-item Interprofessional Team Approach Evaluation Scale, and of questions on the demographic characteristics of the patients. The items could be answered on a 5-point Likert scale expressing the level of agreement with the items (1 –I don’t agree at all; 5 –I totally agree).
Data analysis
The data were analysed by SPSS 25.0 (SPSS Inc. Chicago, IL, USA). The face validity of the scale was confirmed by the consensus development panels described above. The content validity was determined through consensus development panels where the experts judged the appropriateness of the scale to measure all aspects if the underlying construct. The construct validity of the questionnaire was determined by factor analysis. Prior to that, we determined the multicollinearity and calculated the Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy and Bartlett’s test of sphericity. Exploratory factor analysis was performed, and a rotation method employed was Oblimin with Kaiser normalisation. Items with communalities less than 0.3 were excluded from the final scale [15,16]. The reliability of the scale was determined by Cronbach’s alpha.
Due to the high Cronbach’s alpha value, we calculated a composite score. We used a Baker & Hearnshaw equation [17] [(Σitems 1–i) × 100/(5 × i)] × 1.25–25) to range the scale’s score from 0 to 100. In this equation, Σitems 1–i represents the sum of the scores of i items; 5 represents the maximum score points of each item; and i represents the number of items. The other numbers are needed for mathematical purposes in order to range the scale’s score from 0 to 100.
For each factor, we also calculated the Cronbach’s alpha.
To test the normality of the dataset, we used the Shapiro-Wilk test.
Ethical consideration
The study received ethical approval from the National Ethics Committee of the Republic of Slovenia (No. 79/06/12).
Results
Literature review
Through the narrative literature review, we identified five tools that could be used to measure patient evaluation of the team approach in primary health care settings: the General Practice Assessment Questionnaire [18], the Primary Care Assessment Tool [19], the Nurse Practitioners Satisfaction Survey [20], the General Practice Nurse Satisfaction Scale [21], and the Consultation Satisfaction Questionnaire [22].
Consensus development panels
An initial 12-item scale was developed by the experts after the first consensus development panel. After the second consensus development panel, two items were removed and all the other ten items were reformulated. The 10-item scale was presented to the third consensus development panel and was accepted by the experts after minimal modifications in wording.
Cross-sectional validation study
A total of 3,292 patients participated in this study (a 50.7% response rate), of which 1,810 (55.7%) were women (Table 1). The mean age ± standard deviation of the sample was 53.1 ± 1.2 years.
Table 1. Demographic characteristics of the sample.
| Characteristic | Number (%) |
|---|---|
| Gender | |
| Male | 1439 (43.3) |
| Female | 1810 (55.7) |
| Registered by | |
| Public family physician | 2289 (69.6) |
| Family physician with concession | 1002 (30.4) |
| Region of Slovenia | |
| Celje | 415 (12.6) |
| Koper | 193 (5.9) |
| Krško | 141 (4.3) |
| Kranj | 191 (5.8) |
| Ljubljana | 939 (28.5) |
| Maribor | 422 (12.8) |
| Murska Sobota | 280 (8.5) |
| Nova Gorica | 267 (8.1) |
| Novo Mesto | 219 (6.7) |
| Ravne na Koroskem | 224 (6.8) |
| Education | |
| Unfinished primary school | 69 (2.1) |
| Primary school | 512 (15.8) |
| Vocational school | 804 (24.8) |
| Secondary school | 1001 (30.9) |
| University, PhD | 855 (26.4) |
| Employment status | |
| Employed | 1804 (56.1) |
| Unemployed | 350 (10.9) |
| In school | 47 (1.5) |
| Retired | 1013 (31.5) |
| Chronic disease | |
| Present | 1512 (48.8) |
| Not present | 1586 (51.2) |
The Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy was 0.918 and its significance (Bartlett’s test of sphericity) was < 0.001. The communalities of items were in general higher than 0.3, except for item 1 (Table 2).
Table 2. Items’ communalities (extraction method: principal axis factoring).
| Item No. | Item | Initial | Extraction |
|---|---|---|---|
| 1 | When making an appointment in this practice, I had to give the reason for an appointment. | 0.033 | 0.030 |
| 2 | The appointment system in this practice is flexible. | 0.339 | 0.362 |
| 3 | During the consultation, team members gave me enough information on self-care. | 0.562 | 0.684 |
| 4 | During the consultation, team members gave me understandable information about my health and planned treatment. | 0.592 | 0.679 |
| 5 | During the consultation, I was able to express my expectations regarding my treatment plan to the team. | 0.504 | 0.567 |
| 6 | In this practice, all patients are treated equally by the whole team. | 0.517 | 0.553 |
| 7 | Each team member in this practice knows their role. | 0.574 | 0.633 |
| 8 | Team members in this practice respect each other. | 0.592 | 0.676 |
| 9 | In this practice, the team handles my data confidentially. | 0.550 | 0.616 |
| 10 | In this practice, all patients are treated with respect by the whole team. | 0.581 | 0.638 |
The rotated solution extracted two factors. Item 1 was removed as its value was < 0.3 (Table 3). So, the scale was left with 9 items. Two factors had initial eigenvalues more than one. This factor analysis explained 54.4% of variance (Factor 1 explained 48.1% and Factor 2 6.3% of the variance).
Table 3. Factor analysis of the 9-item interprofessional team approach evaluation scale.
| Item No. | Item | Factor 1 –Team Approach | Factor 2 –Person-Centeredness |
|---|---|---|---|
| 1 | When making an appointment in this practice, I had to give the reason for an appointment. | Removed | removed |
| 2 | The appointment system in this practice is flexible. | 0.523 | |
| 3 | During the consultation, team members gave me enough information on self-care. | 0.872 | |
| 4 | During the consultation, team members gave me understandable information about my health and planned treatment. | 0.809 | |
| 5 | During the consultation, I was able to express my expectations regarding my treatment plan to the team. | 0.662 | |
| 6 | In this practice, all patients are treated equally by the whole team. | 0.627 | |
| 7 | Each team member in this practice knows their role. | 0.761 | |
| 8 | Team members in this practice respect each other. | 0.879 | |
| 9 | In this practice, the team handles my data confidentially. | 0.802 | |
| 10 | In this practice, all patients are treated with respect by the whole team. | 0.770 |
The scree plot showed that more variance could be explained by two factors than only one (Fig 2).
Fig 2. The scree plot of a 10-item interprofessional team approach evaluation scale.
This scree plot shows the variance explained by each factor in a factor analysis.
The reliability analysis of the 10-item scale showed that the Cronbach’s alpha was 0.774. When item 1 was removed, the value of the Cronbach’s alpha increased to 0.901 (Table 4). The Cronbach’s alpha of factor 1 was 0.827 and of factor 2 was 0.886.
Table 4. Interprofessional team approach evaluation scale: 10- and 9-item analysis.
| Item No. | Item | Corrected item—total correlation | Cronbach’s alpha if item deleted | Corrected item—total correlation | Cronbach’s alpha if item deleted |
|---|---|---|---|---|---|
| 1 | When making an appointment in this practice, I had to give the reason for an appointment. | 0.153 | 0.901 | Removed | removed |
| 2 | The appointment system in this practice is flexible. | 0.525 | 0.745 | 0.549 | 0.903 |
| 3 | During the consultation, team members gave me enough information on self-care. | 0.634 | 0.737 | 0.682 | 0.889 |
| 4 | During the consultation, team members gave me understandable information about my health and planned treatment. | 0.642 | 0.738 | 0.706 | 0.888 |
| 5 | During the consultation, I was able to express my expectations regarding my treatment plan to the team. | 0.624 | 0.738 | 0.680 | 0.890 |
| 6 | In this practice, all patients are treated equally by the whole team. | 0.628 | 0.737 | 0.692 | 0.889 |
| 7 | Each team member in this practice knows their role. | 0.650 | 0.739 | 0.716 | 0.887 |
| 8 | Team members in this practice respect each other. | 0.629 | 0.742 | 0.698 | 0.889 |
| 9 | In this practice, the team handles my data confidentially. | 0.615 | 0.747 | 0.684 | 0.891 |
| 10 | In this practice, all patients are treated with respect by the whole team. | 0.628 | 0.745 | 0.714 | 0.888 |
Therefore, the final version of our Interprofessional Team Approach Evaluation Scale consisted of 9 items.
The mean value of the summary score of the 9-item Interprofessional Team Approach Evaluation Scale was 95.1 ± 9.0 points (on a scale from 0 to 100); the minimum was 33.3 and the maximum 100.0, with a median of 100.0 and a variance of 80.7. The dataset was not normally distributed (Shapiro-Wilk test of normality was 0.62 with p < 0.001). The mean value of the factor Team Approach was 4.8 ± 0.4 (the minimum was 2.6 and maximum 5, on a scale from 1 to 5) and for the factor Person-Centeredness 4.7 ± 0.4 (the minimum was 1.3 and maximum 5.0, on a scale from 1 to 5).
The patients evaluated their team as very high in confidentiality, equity, and respect, and gave the lowest evaluation to the flexibility of the appointment system (Table 5).
Table 5. Respondents’ evaluation of the team approach based on the interprofessional team approach evaluation scale.
| Item | Mean (standard deviation) | Median (minimum, maximum) |
|---|---|---|
| The appointment system in this practice is flexible. | 4.7 (0.6) | 5.0 (1, 5) |
| During the consultation, team members gave me enough information on self-care. | 4.8 (0.5) | 5.0 (1, 5) |
| During the consultation, team members gave me understandable information about my health and planned treatment. | 4.8 (0.5) | 5.0 (1, 5) |
| During the consultation, I was able to express my expectations regarding my treatment plan to the team. | 4.8 (0.5) | 5.0 (1, 5) |
| In this practice, all patients are treated equally by the whole team. | 4.9 (0.5) | 5.0 (1, 5) |
| Each team member in this practice knows their role. | 4.8 (0.5) | 5.0 (1, 5) |
| Team members in this practice respect each other. | 4.8 (0.4) | 5.0 (1, 5) |
| In this practice, the team handles my data confidentially. | 4.9 (0.4) | 5.0 (1, 5) |
| In this practice, all patients are treated with respect by the whole team. | 4.9 (0.4) | 5.0 (1, 5) |
Discussion
Summary of main findings
The Interprofessional Team Approach Evaluation Scale assessing interprofessional team-based care in primary healthcare from the patients’ point of view, which adds a new approach to teamwork measurements and to measurements of patient satisfaction. The results of our study suggest that this scale could be a valid and reliable tool for assessing patient evaluation of family medicine practice teams in Slovenian settings. Team Approach and Person-Centeredness emerged as the key factors of the scale. These factors were also shown to be reliable. The patients evaluated their team as very high in confidentiality, equity, and respect, and gave the lowest evaluation to the flexibility of the appointment system.
Comparison to other tools
The Nurse Practitioner Evaluation Scale [9] and the European Patients Evaluation of General Practice Care (EUROPEP) [23] were used in the same settings for patient evaluation of family physicians and nurse practitioners. The EUROPEP scale includes five aspects of care: relation and communication; medical care; information and support; continuity and cooperation; and facilities, availability and accessibility [23]. The Nurse Practitioner Evaluation Scale involves three factors: the Clinical Approach, the Comprehensive Approach, and the Person-centred Approach) [9]. The latter factor was also found in our Interprofessional Team Approach Evaluation Scale.
The General Practice Assessment Questionnaire [18] was developed from the General Practice Assessment Survey [24], and has been found to be a valid and reliable tool for use in general practice [18]. However, it evaluates each member of the family practice team individually, and therefore does not evaluate a team approach to patients. The Primary Care Assessment Tool [19] contains seven factors: first contact accessibility, first contact utilisation, longitudinal interpersonal relationships, comprehensiveness of services available, comprehensiveness of services received, coordination, and community orientation. It also evaluates the patient’s experience mainly with physicians, not with the team as a whole. The Consultation Satisfaction Questionnaire [22] looks at four different areas of satisfaction: general satisfaction, professional care, depth of relationship, and perceived time. Again, it evaluates the physician, not the team. The Nurse Practitioners Satisfaction Survey [20] and the General Practice Nurse Satisfaction Scale [21] evaluate the nurse practitioners working in a family practice team [25–29].
Factor 1
The Team Approach factor was not recognised in our previously developed tool, which measured patient satisfaction with nurse practitioners within a family medicine team [9].
There are many instruments which measure different aspects of teamwork [6, 7]. Some of them focus in patient safety, such a SafeQuest [25] and a Safety Attitudes Questionnaire [26], some measure team relationships, such as a Survey of Organisational Attributes for Primary Care [27], and some measure working environment such as Clinical Staff Questionnaire [28]. However, only one scale measures interprofessional team-based approach from the viewpoint of the patients but it was developed for the emergency care settings [29]. Our scale, on the other side, offers the measure of the interprofessional teamwork from the aspect of patients [25, 26, 30].
Factor 2
A Person-Centeredness factor was already recognised in our previously developed tool, which measured patient satisfaction with nurse practitioners within a family medicine team as a patient-centred approach [9]. Both concepts differ in terms of the object of care; a patient is the one who is sick or being treated, and a person is a human being [3], but both concepts share the same approach. We decided to use a broader concept—Person-Centeredness—as within family medicine not all people that come for a consultation correspond to the term patient. Person-Centeredness is an important part of the family medicine approach and also has an important role in quality assurance [31]. It has also been recognised in other instruments that measure teamwork or patient satisfaction [19, 23]. Person-Centred care cannot be practiced by just one member of a team but only by the whole team, so this factor is important for teamwork, and is obviously also recognised by the patients.
Descriptive results of the study
The patients assessed the team approach in family medicine practices very highly. This is also in line with the previous studies on patient satisfaction in Slovenian family medicine practices [9, 32]. The item that received least points was the flexibility of the appointment system. This problem seems to characterise Slovenian family medicine practices already for some time as other studies showed [9, 32, 33].
Implications of the study
The results of our study showed two factors that might be important in the evaluation by patients of the teamwork approach in family medicine. Still, other factors could also be important within this context, as shown by other studies focusing on teamwork [6]. Our study pointed out the need to develop a comprehensive theoretical framework of family medicine team competencies, which could then be used to develop a comprehensive tool for evaluating the team approach to patients. Several comprehensive frameworks for family medicine already exist, but they all focus only on one member of the team [34–37]. A competency framework for the whole team is needed, as some studies have shown that the competencies of individual members of the team are not fully understood and could be a source of less effective teamwork [21, 38, 39].
Future studies should focus on developing the aforementioned framework and consequently a comprehensive tool for its measurement.
Strengths and limitations of the study
This was the first study that dealt with patient evaluations of family medicine teams in Slovenia and probably also internationally. We developed a new tool for assessing patient evaluation of the team approach in family medicine practices through a multistep process ensuring its validity and reliability. We used consensus development panels which are a qualitative method for obtaining agreement in areas of uncertainty or where there is a lack of definitive information. Furthermore, consensus development panels help bring professionals together to directly comment and develop tools and techniques [34]. The development process could have been more rigorous, involving focus groups with experts including representatives of patients which would result in a more comprehensive framework. This could also be a reason why some factors recognised in other studies were not recognised in our study. Also, the content validity could be confirmed by the content validity index which we did not calculate but used qualitative information on items’ content validity as judged by the experts. We also did not perform a collection of a response process and we did not split the respondents to two groups to determine the coherency of the concepts with exploratory and factor analyses. These, too, could be considered as limitations of our study. The strength of this study is the large sample of patients included in the quantitative part of the study, which means the results can be generalised.
Conclusion
This study provided a new scale for the evaluation of patient satisfaction with the family medicine team as a whole. It opened the question of family medicine team competencies, and pointed towards the need to develop a family medicine team competency framework and a comprehensive tool for its measurement. This would strengthen the team approach and possibly enhance the safety culture in family medicine.
Supporting information
This supporting information contains the anonymised dataset used in the study.
(XLSX)
Acknowledgments
We would like to thank all the experts and patients who participated in the study.
Data Availability
Patients that were unable to fill in the questionnaire (e.g. due to blindness, cognitive impairment, or terminal illness) or who did not want to participate or were younger than 18 years were excluded from the study. One member of the practice team explained to the patients the aim and method of the study and invited them to participate. They gave an oral informed consent. They filled-in the questionnaire in the waiting room and put it in the sealed box afterwards to secure the confidentiality of the data.
Funding Statement
ZKK’s work on this project was partly supported by the Slovenian Research Agency [Research core funding research in The Field of Public Health No. P3-0339]. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. All authors were internally supported by their employer Ljubljana Community Health Centre. There was no additional external funding received for this study.
References
- 1.Allen J, Gay B, Crebolder H, Heyrman J, Svab I, Ram P. The European definition of general practice/family medicine. Evans P, editor. Ljubljana: Wonca Europe; 2011. [Google Scholar]
- 2.Heyrman J. EURACT Educational Agenda. Leuven: European Academy of Teachers in General Practice; 2005. [Google Scholar]
- 3.Zhao J, Gao S, Wang J, Liu X, Hao Y, "Differentiation between two healthcare concepts: person-centered and patient-centered care", International Journal of Nursing Sciences, 2016;3:398–402. [Google Scholar]
- 4.Mitchell P, Wynia R, Golden B, McNellis B, Okun S, Webb, et al. Core principles and values of effective team-based health care. Discussion Paper. Washington, CD: Institute of Medicine; 2012.
- 5.Nancarrow SA, Booth A, Ariss S, Smith T, Enderby P, Roots A, “Ten principles of good interdisciplinary team work”, Hum Resour Health, 2013;11:19 10.1186/1478-4491-11-19 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Valentine MA, Nembhard IM, Edmondson AC, "Measuring teamwork in health care settings: a review of survey instruments", Medical Care, 2015;53(4):e16–30. Epub 2013/11/06. [DOI] [PubMed] [Google Scholar]
- 7.Shoemaker SJ, Parchman ML, Fuda KK, Schaefer J, Levin J, Hunt M, et al. , "A review of instruments to measure interprofessional team-based primary care", J Interprof Care, 2016;30(4):423–32. Epub 2016/05/24. 10.3109/13561820.2016.1154023 . [DOI] [PubMed] [Google Scholar]
- 8.Klemenc-Ketis Z, Poplas-Susic A, "Are characteristics of team members important for quality management of chronic patients at primary care level?", J Clin Nurs, 2017;26(23–24):5025–32. Epub 2017/08/10. 10.1111/jocn.14002 . [DOI] [PubMed] [Google Scholar]
- 9.Klemenc-Ketis Z, Kravos A, Poplas-Susič T, Svab I, Kersnik J, "New tool for patient evaluation of nurse practitioner in primary care settings", J Clin Nurs, 2014;23(9–10):1323–31. 10.1111/jocn.12377 [DOI] [PubMed] [Google Scholar]
- 10.Petek D, Mlakar M, "Quality of care for patients with diabetes mellitus type 2 in ‘model practices’ in slovenia—first results", Zdr Varst, 2016;55(3):179–84. 10.1515/sjph-2016-0023 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Poplas-Susic T, Svab I, Klančar D, Petek D, Vodopivec-Jamsek V, Bulc M, et al. , "Screening and Registering Patients with Asthma and Copd in Slovenian Primary Care: First Results", Zdr Varst, 2015;54(3):161–7. 10.1515/sjph-2015-0023 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Klemenc-Ketis Z, Svab I, Poplas Susic A, "Implementing Quality Indicators for Diabetes and Hypertension in Family Medicine in Slovenia", Zdr Varst, 2017;56(4):211–9. Epub 2017/10/25. 10.1515/sjph-2017-0029 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Poplas-Susic T, Svab I, Kersnik J, "The project of model practices in family medicine in Slovenia", Zdrav Vestn, 2013;82(10):635–47. [Google Scholar]
- 14.Beaton DE, Bombardier C, Guillemin F, Ferraz MB, "Guidelines for the Process of Cross-Cultural Adaptation of Self-Report Measures", Spine, 2000;25(24):3186–91. [DOI] [PubMed] [Google Scholar]
- 15.Rattray J, Jones MC, "Essential elements of questionnaire design and development", J Clin Nurs, 2007;16(2):234–43. 10.1111/j.1365-2702.2006.01573.x [DOI] [PubMed] [Google Scholar]
- 16.Ferketich S, "Focus on psychometrics: aspects of item analysis. ", Res Nurs Health, 1991;14:165–8. [DOI] [PubMed] [Google Scholar]
- 17.Baker R, Hearnshaw H. A method for surveying patient satisfaction audit protocol PS1. Leicester: Eli Lilly National Cinical Audit Centre; 1996. [Google Scholar]
- 18.Mead N, Bower P, Roland M, "The General Practice Assessment Questionnaire (GPAQ)—development and psychometric characteristics", BMC Fam Pract, 2008;9:13 Epub 2008/02/22. 10.1186/1471-2296-9-13 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Shi L, Starfield B, "Validating the Adult Primary Care Assessment Tool", J Fam Pract, 2001;50(2):161w–75w. [Google Scholar]
- 20.Agosta LJ, "Psychometric evaluation of the Nurse Practitioner Satisfaction Survey (NPSS).", J Nurs Meas, 2009;17(2):114–33. [DOI] [PubMed] [Google Scholar]
- 21.Halcomb EJ, Caldwell B, Samlamonson Y, Davidson PM, "Development and Psychometric Validation of the General Practice Nurse Satisfaction Scale", J Nurs Scholars, 2011;43(3):318–27. [DOI] [PubMed] [Google Scholar]
- 22.Poulton BC, "Use of the consultation satisfaction questionnaire to examine patients’ satisfaction with general practitioners and community nurses: Reliability, replicability and discriminant validity", Br J Gen Pract, 1996;46(402):26–31. [PMC free article] [PubMed] [Google Scholar]
- 23.Grol R, Wensing M, Mainz J, Ferreira P, Hearnshaw H, Hjortdahl P, et al. , "Patients in Europe evaluate general practice: an international comparison", Br J Gen Pract, 2000;50(460):882–7. [PMC free article] [PubMed] [Google Scholar]
- 24.Ramsay J, Campbell JL, Schroter S, Green J, Roland M, "The General Practice Assessment Survey (GPAS): tests of data quality and measurement properties", Fam Pract, 2000;17(5):372–9. Epub 2000/10/06. . [DOI] [PubMed] [Google Scholar]
- 25.de Wet C, Spence W, Mash R, Johnson P, Bowie P, "The development and psychometric evaluation of a safety climate measure for primary care", Qual Saf Health Care, 2010;19(6):578–84. Epub 2010/04/22. 10.1136/qshc.2008.031062 . [DOI] [PubMed] [Google Scholar]
- 26.Sexton JB, Helmreich RL, Neilands TB, Rowan K, Vella K, Boyden J, et al. , "The safety attitudes questionnaire: Psychometric properties, benchmarking data, and emerging research ", BMC Health Serv Res, 2006;3:44. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Ohman-Strickland PA, John Orzano A, Nutting PA, Perry Dickinson W, Scott-Cawiezell J, Hahn K, et al. , "Measuring organizational attributes of primary care practices: development of a new instrument", Health Serv Res, 2007;42(3 Pt 1):1257–73. Epub 2007/05/11. 10.1111/j.1475-6773.2006.00644.x . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Jaen CR, Crabtree BF, Palmer RF, Ferrer RL, Nutting PA, Miller WL, et al. , "Methods for evaluating practice change toward a patient-centered medical home", Ann Fam Med, 2010;8 Suppl 1:S9–20; s92 Epub 2010/06/16. 10.1370/afm.1108 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Henry BW, McCarthy DM, Nannicelli AP, Seivert NP, Vozenilek JA, "Patients’ views of teamwork in the emergency department offer insights about team performance", Health Expect, 2016;19(3):702–15. Epub 2013/10/15. 10.1111/hex.12148 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Deilkas ET. Patient safety culture—opportunities for healthcare management. Oslo: University of Oslo, Norway; 2010. [Google Scholar]
- 31.Makivić I, Kersnik J, Klemenc-Ketis Z, "The Role of the Psychosocial Dimension in the Improvement of Quality of Care: A Systematic Review", Zdr Varst, 2016;55(1):86–95. 10.1515/sjph-2016-0004 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Klemenc-Ketis Z, Petek D, Kersnik J, "Associations between family doctors’ practices characteristics and patient evaluation of care", Health Policy, 2012;106(3):269–75. 10.1016/j.healthpol.2012.04.009 [DOI] [PubMed] [Google Scholar]
- 33.Kersnik J, "An evaluation of patient satisfaction with family practice care in Slovenia. ", Int J Qual Health Care, 2000;12(2):143–7. [DOI] [PubMed] [Google Scholar]
- 34.Czabanowska K, Klemenc-Ketis Z, Potter A, Rochfort A, Tomasik T, Cziszar J, et al. , "Development of a Competency Framework for Quality Improvement in Family Medicine: A Qualitative Study", J Contin Educ Health Prof, 2012;32(3):174–80. [DOI] [PubMed] [Google Scholar]
- 35.The CanMEDS 2005 Physician Competency Framework. Frank JR, editor. Ottawa, Ontario: Royal College of Physicians and Surgeons of Canada; 2005. [Google Scholar]
- 36.Health Professions Education: A Bridge to Quality. Greiner AC, Knebel E, editors. Washington, DC: Institute of Medicine; 2003. [PubMed] [Google Scholar]
- 37.Institute of Medicine. Crossing the Quality Chasm: a New Health System for the 21st Century. Washington, DC: National Academic Press; 2001. [PubMed] [Google Scholar]
- 38.Klemenc-Ketis Z, Deilkas ET, Hofoss D, Bondevik GT, "Patient Safety Culture in Slovenian out-of-hours Primary Care Clinics", Zdr Varst, 2017;56(4):203–10. Epub 2017/10/25. 10.1515/sjph-2017-0028 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39.Klemenc-Ketis Z, Maletic M, Stropnik V, Deilkås ET, Hoffos D, Bondevik GT, "The safety attitudes questionnaire—ambulatory version: psychometric properties of the Slovenian version for the out-of-hours primary care setting", BMC Health Serv Res, 2017;17:36 10.1186/s12913-016-1972-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
This supporting information contains the anonymised dataset used in the study.
(XLSX)
Data Availability Statement
Patients that were unable to fill in the questionnaire (e.g. due to blindness, cognitive impairment, or terminal illness) or who did not want to participate or were younger than 18 years were excluded from the study. One member of the practice team explained to the patients the aim and method of the study and invited them to participate. They gave an oral informed consent. They filled-in the questionnaire in the waiting room and put it in the sealed box afterwards to secure the confidentiality of the data.