Development, validity and reliability of instrument to evaluate implementation fidelity of the Family Doctor Concept (FDC) programme in public primary care clinics in Malaysia

Abstract

Introduction:

The Family Doctor Concept (FDC) programme was introduced to public primary care clinics in late 2013 as part of the reform agenda in the primary healthcare delivery system. The study aimed to develop a validated and reliable instrument to evaluate the FDC implementation fidelity in primary care clinics.

Methods:

The instrument, which adapted the concept of patient-centred care (PCC), resulted from a series of expert discussions, a literature search, an FDC guideline, and a review of meeting minutes. A 2-step process was conducted with experts to achieve content and face validity. Consequently, the instrument was piloted in 5 public primary care clinics in Selangor involving 8 trained raters. Inter-rater reliability was estimated using intraclass correlation (ICC), while internal consistency was measured using Kuder—Richardson Formula 20 (KR-20).

Results:

The final instrument comprises 3 sections (instructions, clinic’s characteristics and assessment items), with Section 3 containing 15 items divided into four components — population registration, formation of a primary care team, integrated treatment, and monitoring and evaluation. The ICC for total score was excellent, 0.981, while the ICCs of the individual component scores were good to excellent (population registration: 0.937, formation of primary care team: 0.742, integrated treatment: 0.996, and monitoring and evaluation: 0.996). The value of KR-20 was 0.615, which was considered adequate.

Conclusion:

The instrument developed was found to be valid in terms of face and content validity and reliable in measuring the fidelity of FDC implementation with excellent inter-rater reliability.

Introduction

An estimated 4.1 million people aged 18 years and above in Malaysia currently live with 2 or 3 major cardiovascular diseases risk factors, and the number is increasing.¹ Adding to the multimorbidity is an ageing population who typically need long-term care from a team of professionals from multiple disciplines. However, overall, the healthcare delivery system in Malaysia is currently disease-oriented, and decisions concerning treatment options are aimed solely at improving clinical outcomes. This approach to care is less responsive to the comprehensive needs and preferences of people with multimorbidity and may even be harmful to them.² For example, people with multimorbidity are typically seen by healthcare providers from multiple fields who may prescribe various medications without realising the potential danger of drug interaction. Thus, reform towards more responsive health systems is urgently needed, especially in the primary healthcare delivery system, to cater to the comprehensive needs and preferences of people with multimorbidity.³

The definition of primary health care (PHC) stems from the declaration of Alma-Ata and encompasses the broader spectrum of services beyond the traditional healthcare delivery system. Accordingly, this concept is defined as “a whole society approach to health that aims at ensuring the highest possible level of health and well-being and their equitable distribution”.⁴ It incorporates 3 inter-related and synergistic components — primary care and essential public health functions, empowered people and communities, and multisectoral policy and actions. Thus, primary care is a subset of PHC that is more focused on healthcare delivery. It is defined as the provision of integrated healthcare services by clinicians who are accountable for addressing a vast majority of personal healthcare needs, developing a sustained partnership with patients, and practising in the context of family and community.⁵ Since the 1970s, the delivery of public primary care services in Malaysia has comprised a 2-tier system consisting of networks of health centres and community clinics.⁶

As part of the reform of primary care service delivery, the Ministry of Health (MOH) Malaysia embarked on the Family Doctor Concept (FDC) programme in late 2013.⁷ It began with a pilot project involving 14 public primary care clinics across the country in 2015 and was expanded each year to another 48 clinics. Currently, 322 out of 1091 public primary care clinics have implemented the programme. It ensures that the population, whether as individuals or as family units, will be able to enjoy patient-centred care that is continuous, holistic and comprehensive. This objective can be achieved by assigning one primary care physician or a team of healthcare workers to a family.⁸ By following this approach, individuals in a family will be treated by the same team of providers, resulting in more patient-centred or personalised care. Subsequently, the patient—doctor relationship will improve, enhancing the quality of care and increasing patient compliance towards the treatment given.^9,10

In general, patient-centred care is made up of 7 dimensions, including respect for patients; coordination of care; information, communication and education; physical comfort; emotional comfort/alleviation of fear and anxiety; involvement of family and friends; and transition and continuity.¹¹ Recently, the integrative model of patient-centredness was introduced to include more elements, such as the essential characteristics of a clinician, clinician—patient relationship, clinician—patient communication, the patient as a unique person, biopsychosocial perspective, patient information, patient involvement in care, involvement of family and friends, patient empowerment, physical support, emotional support, integration of medical and nonmedical care, teamwork and team building, access to care, coordination and continuity of care.¹² However, in developing an evaluation instrument in this study, we chose to adopt Greene et al.’s 3 components of patient-centred care: structural, clinical and interpersonal, as shown in Table 1.¹³ This framework is thought to be actionable and relatable to FDC.

Table 1. Dimension and attributes of patient-centred primary care clinics, as described by Greene et al.

Structural dimension (System features)	Clinical dimension (Provision of care)	Interpersonal dimension (Relationship)
1. Built environment Calm and welcoming space Cleanliness Signage, easy to find way Electrical appliances and furniture	1. Clinical decision support Ensuring shared decision Best-available evidence Supporting self-management Health education and promotion	1. Communication Training on communication skills, retraining Sharing of information with patients Patients’ feedback on communication Communication with family & friends
2. Access to care Population registry Easy appointment system Minimised waiting time	2. Coordination and continuity Care coordination between providers Seamless flow of information Coordinates with community and hospital	2. Knowing the patient Training on patient-centredness for complex, chronic illness Surveying patients about the care experience Multidisciplinary meetings
3. Zoning of clinic Formation of team Family database Tracking patients’ preferences, values and needs	3. Types of encounters Home visits and domiciliary care Accommodating virtual visits (Phone and email) Palliative care services	3. Importance of teams Ensuring responsiveness of care team to patient’s and family needs Engagement of employees for improvements Involvement of community/panel of advisers

Due to the heterogeneity of the public primary care clinics in Malaysia¹⁴ (for example, some clinics having more resources than others or differences between urban and rural locations), the implementation of the FDC programme may differ across settings. Thus, assessing implementation in comparison with the standard set by the programme developers is crucial to facilitate comparison between clinics. Subsequently, policymakers can use the findings to implement appropriate improvements to the programme. Implementation fidelity, also called “integrity”, is defined as the degree to which a programme is implemented as intended.¹⁵ It provides a picture of the degree to which a programme is successfully implemented into an organisation and mediates the relationships between the programme and the intended outcomes. The probability of having a more successful programme is often associated with higher implementation fidelity.¹⁶

The purpose of this study was to develop a validated and reliable instrument to measure the fidelity of FDC implementation in primary care clinics. The practice instrument developed can be used for future monitoring and evaluation activities.

Methods

The study was conducted from August to November 2019. Figure 1 illustrates the steps followed in developing and validating the instrument. In addition to measuring implementation fidelity by creating a scale, the instrument also collected general data on the characteristics of the clinics, including the size of population covered, location, facility type, availability of a Family Medicine Specialist (FMS), and the use of an electronic medical record (EMR) system called TelePrimary Care (TPC).

Figure 1. Flow diagram of checklist development, validation and reliability.

Step 1

Content and face validity of the checklist

The establishment of content validity is a fundamental step in the development of a new measuring instrument because it represents a precursor for linking abstract concepts with observable and measurable indicators. According to the guideline, the evaluation of content validity consists of a 2-step process, starting with the development stage, which is followed by the judgement stage.¹⁷

Development stage

At this stage, the content for the instrument constituting several domains was obtained primarily from an extensive review of the literature on FDC, person-centred care (PCC), personalised care and other quality improvements interventions, such as the lean healthcare model. In this study, 26 articles were identified as relevant to these topics. In addition, we went through the FDC guidelines provided by the MOH,⁷ reviewed meeting minutes, went through PowerPoint presentations and held discussions with a panel of experts. All possible components of FDC were extracted to develop the first version of the 4-page instrument, which consisted of three sections, nine components and subcomponents, and 37 items.

A group of experts with experience in FDC implementation provided input and feedback in 2 rounds of consultation and one approval round. The expert group consisted of public health physicians (PHP, n=10), family medicine specialists (FMS, n=6), medical officers (MO, n=4), nurses (n=5), and assistant medical officers (AMO, n=5) from all over the country to achieve a balanced group in terms of multidisciplinary representation and state representation. The first version of the instrument was presented to the experts in a workshop for the development of the instrument, and they provided either written or verbal feedback.

Many of the experts argued that measuring fidelity with a 1 to 5 or 1 to 3 Likert scale would be tedious and might not be practical in the field. Hence, the panel of experts was in agreement that a “Yes” or “No” response was adequate to capture the implementation of components of FDC. The authors discussed the feedback from every consultation round and agreed on the revisions that were made.

Judgement stage

At this stage, a smaller group of experts, consisting of 12 individuals, 2 from each professional role (PHI, FMS, MO, nurses, AMO and support staff), was required to work independently to evaluate the final version of the instrument. These individuals were selected based on their expertise in healthcare programme implementation, academic credentials and seniority in the MOH. They were also asked to judge the relevance of each component and item in the instrument. At this stage, the instrument contained 3 sections with 22 items.

Face validity

The instruments face validity was obtained from a non-expert panel involving 10 individuals, 2 from each professional role, including FMS, MO, nurses, AMO, and support staff. They were selected to resemble the target respondents as closely as possible. All of them worked in a primary care clinic located in Selangor that had implemented the FDC programme since 2017. The assessment of comprehensiveness, accuracy, completeness and meaning of all items was conducted in cognitive interview sessions.¹⁸ In these sessions, the participants were asked about their thinking process as they went through the instrument, such as how they interpreted the meaning of questions and possible responses, what they thought about when considering their answers, how they decided on their answers and what each answer meant. Simultaneously, an assessment of the grammar, syntax, organisation, and appropriateness and logical sequence of the items was obtained from the participants. At the end of this stage, the expert panel decided that the 4 components and 15 items with binary “yes” or “no” responses were adequate to assess the implementation fidelity.

Step 2

Reliability analysis

A pilot study was conducted in 5 primary care clinics in Selangor in November 2019 to assess the feasibility and reliability of the instrument. Reliability was measured using internal consistency and inter-rater reliability. Internal consistency measures the degree of interrelationship or homogeneity among the items, while inter-rater reliability refers to a measurement of the consistency of the absolute value of evaluators’ ratings.¹⁹ Additionally, this smaller-scale study was supposed to guide in planning and allow modification of the protocol before researchers would proceed with the real study.²⁰ Thus, this pilot study was also conducted to assess the procedures for approaching the study site and the data collection process.

Eight medical officers with research backgrounds were invited as raters to pilot the instrument by rating 5 purposively selected primary care clinics. They were chosen based on their competency in programme evaluation and vast experience in primary care. The clinics were selected based on their readiness to participate in the project with at least 2 years of experience in the implementation of FDC. The sample size was determined based on a minimum sample of 5 clinics to be assessed by raters that would be required to achieve the statistical significance for an alpha-value set at 0.05 and with the minimum power of at least 80.0%.²¹ Eight raters were required to produce 40 observations, representing the minimum number needed to measure inter-rater reliability.²² The raters were given one hour of training on how to use the instrument and the source of where they could retrieve the information. A glossary containing a description of each item was also given as a guide. The instruments were recorded independently, and raters were not allowed to discuss the findings among themselves. This strategy was employed since the reliability determination depends on the raters, their training and competency.²³

Statistical analysis

The characteristics of the participants and primary care clinics were calculated through descriptive analysis. The internal consistency was measured using Kuder—Richardson Formula 20 (KR-20). The intraclass correlation coefficients (ICC) with a 2-way random effects model and absolute agreement definition and their 95% confidence intervals were used to assess the inter-rater reliability of the items in the checklist. The following classification was used to interpret the ICC values: weak reliability, ICC<0.5; moderate reliability, ICC 0.5-0.75; good reliability, ICC 0.75-0.90; and excellent reliability, ICC≥0.90.²⁴ The level of significance was set at 5% (P<0.05), with a 2-tailed distribution. The data were analysed using IBM SPSS 22.0 for Windows.

Ethical considerations

The study was previously approved by the National University of Malaysia ethical review board (Code: UKM/PPI/111/8/J EP-2019-584) and the Medical Research and Ethics Committee, Ministry of Health Malaysia (Code: NMRR-18-3871-44034).

Results

The final instrument was in the form of a checklist used to measure the implementation fidelity of the clinics based on a scoring system. This 2-page list was easy to complete and took raters an average time of approximately 30 to 40 minutes. The checklist consisted of 22 items, divided into the following 3 sections:

Section 1: The instructions for raters to complete the instrument.

Section 2: Seven items on the characteristics of primary care clinics evaluated, including the size of the population covered, location, facility type, availability of Family Medicine Specialist (FMS), and the use of the TelePrimary Care (TPC) electronic medical record (EMR) system.

Section 3: Fifteen items divided into 4 components (population registration, formation of primary care team, integrated treatment, and monitoring and evaluation)

All items in Section 3 required a binary response of either “yes” or “no”. One point was granted if the answer was “yes” and no points if the answer was “no”. The researchers counted and totalled up the score.

Step 1

Content and face validity of the checklist

At the content-validity developmental stage, several changes were madein terms of the structure of the checklist. The initial draft was longer, consisting of 4 pages divided into 3 sections. Section 3 contained 30 items divided into the three components of patient-centred care proposed by Greene et al.¹³ However, out of 30 items, only 15 were retained; these were rearranged into 4 components. For example, from the structural component, only population registry and formation of teams were considered relevant and retained in the instrument. The majority of items in component 2 involving the clinical component were dropped because they were already measured in the clinical audit; these were replaced with client treatment in an integrated manner component. For component 3, involving interpersonal dimensions, 3 items were considered important: 1. client satisfaction survey, 2. staff personnel survey, and 3. regular meetings. Thus, these items were grouped together and renamed as the monitoring and evaluation component. The measurement of items was also converted from a 3-point Likert scale to a nominal yes or no scale to make the tool more user-friendly for the raters. The changes made were confirmed in the content-validity judgement stage.

The face-validity assessment consisted of the face-to-face interview sessions and professional appraisal of the checklist. The instrument was assessed by physicians, paramedics and support staff, who made 4 changes to the wordings of the items. For example, “integrated care” in component 3 was changed to “treatment in an integrated manner”. The face-validity assessment also resulted in 2 changes to linguistic and interpretative terms in the questionnaire. For instance, “health staff” was changed to “health personnel”.

The final version of Section 3 of the checklist translated from the Malay language is shown in Table 2.

Table 2. Final version of Section 3 of the checklist, containing 15 items in 4 components.

Items	Yes (1)	No (0)	Notes
Component 1: Population registration
1. Has a system for registering population. 1. Mempunyai sistem untuk mendaftarpopulasi.
2. Clinic updates client information in the population register from time to time. 2. Mengemaskini maklumat klien dalam daftarpopulasi dari semasa ke semasa.
3. The population register has complete personal information details. 3. Daftar populasi mempunyai butiran peribadi yang lengkap.
4. The population register has complete health information details. 4. Daftar populasi mempunyai butiran kesihatan yang lengkap.
5. Population register is used in planning health programmes at clinic level. 5. Daftar populasi digunakan dalam merancang program — program kesihatan.
Component 2: Formation of primary care team
6. Division of health personnel from various disciplines into several groups. 6. Pembahagian anggota kesihatan pelbagai bidang kepada beberapa kumpulan.
7. Formal discussions among team members are held from time to time. 7. Perbincangan dalam satu PHT diadakan dari semasa ke semasa.
8. Notification to clients in the form of signboards or guides on team division. 8. Pemakluman PHT kepada klien dalam bentukpapan tanda atau pemandu arah.
9. Clinic achievements are distributed according to their respective teams. 9. Pencapaian klinik diagihkan mengikut PHT masing - masing.
Component 3: Client treatment is provided in an integrated manner
10. Clients from one team are given treatment in the same place. (No OPD and MCH division). 10. Klien dari satu pasukan diberikan rawatan ditempat yang sama. (Tiada pembahagian OPD dan MCH)
11. Medical officers treat patients based on team division. (No OPD and MCH medical officers). 11. Pegawai Perubatan merawat pesakit berdasarkan kepada PHT (Tiada pegawai perubatan OPD dan MCH)
12. Training to treat patients in an integrated and holistic manner is given to health personnel. 12. Latihan merawat pesakit secara integrasi dan holistic diberikan kepada anggota kesihatan.
Component 4: Monitoring and evaluation
13. Client survey on FDC. 13. Kaji selidik klien tentang FDC.
14. Health personnel survey on FDC. 14. Kaji selidik anggota kesihatan tentang FDC.
15. Regular clinic-level FDC meetings. 15. Mesyuarat FDC peringkat klinik secara berkala.
Total Score

Step 2

Reliability test

Table 3 shows the characteristics of the 5 primary care clinics chosen. In the pilot study, a list of liaison officers (LO) for each primary care clinic was obtained from the person in charge of the primary care clinic (either FMS or MO). The job of the LO was to assist the raters’ evaluation process. There were no obvious signs of hesitancy or changing opinions based on the markings made by the raters on the instrument.

Table 3. Characteristics of primary care clinics for the pilot study.

Characteristics	Clinic A	Clinic B	Clinic C	Clinic D	Clinic E
1. Number of individuals covered	7,595	308,906	No data	182,526	79,906
2. Clinic type	4	3	3	3	3
3. FMS or No FMS	-	FMS	FMS	FMS	FMS
4. TPC or No TPC	-	-	-	-	TPC
5. Location	RURAL	URBAN	URBAN	URBAN	URBAN

Table 4 displays the characteristics of the 8 raters.

Table 4. Characteristics of the 8 raters.

Characteristics	Age	Gender	Profession	Level of education	Years of service
Rater 1	38	Male	MO	Master	10
Rater 2	34	Male	MO	Master	8
Rater 3	35	Male	MO	Master	9
Rater 4	35	Male	MO	Master	9
Rater 5	42	Female	MO	Master	15
Rater 6	33	Female	MO	Degree	8
Rater 7	31	Female	MO	Degree	7
Rater 8	32	Male	MO	Degree	8

The minimum total score that raters gave was 5, while the maximum total score was 11. The ICC for total score was excellent, 0.981, while the ICCs of the individual component scores were good to excellent (population registration: 0.937, formation of primary care team: 0.742, integrated treatment: 0.996, and monitoring and evaluation: 0.996), as shown in Table 5. The KR-20 for total items was 0.615, which was considered adequate.

Table 5. ICC and KR-20 values.

Component score	ICC	CI (95%)	P value	KR-20
Component 1: Population registration	0.937	0.802-0.993	P<0.001
Component 2: Formation of primary care team	0.742	0.076-0.971	P<0.001
Component 3: Client treatment provided in an integrated manner	0.996	0.988-1.000	P<0.001	0.615
Component 4: Monitoring and evaluation	0.996	0.988-1.000	P<0.001
Total score	0.981	0.939-0.998	P<0.001

Discussion

In this study, we were able to develop a checklist based on the components of FDC that all stakeholders found satisfactory. The scaling up of the FDC programme in public primary care clinics over the years has posed challenges to project managers in terms of evaluating its components. FDC is a relatively new concept, introduced in the rather complex environment in healthcare settings; hence, some clinics may find implementing all of its components understandably difficult. Moreover, the concept of patient-centred care is multi-faceted, and the published literature offers well-established instruments to measure each component.^25,26

Although a patient-centred care experience may seem aspirational and ideal, Greene et al.’s framework highlights its importance. In the context of FDC, adjustments were made because public primary care clinics had the capacity to implement only a portion of patient-centred care elements, mainly due to limited resources,²⁷ limited consultation time, patient overload and increased clerical and administrative tasks.^28,29 For example, the interpersonal dimension, such as effective communication that involves active listening to both the patients’ medical and non-medical needs (e.g. their values, expectations and life experiences), was not included in the checklist even though it was considered key to delivering patient-centred care.

The findings showed that the ICC values for the individual component scores can be considered excellent, except for component 2, which was good, while the ICC for the total score was excellent (ICC 0.981, P<0.001). The results also revealed that the confidence interval (CI) for ICC for component 2 scores was exceptionally wide, 0.076–0.971. This broad CI could be attributed to the limited number of levels of the component (4 items); moreover, the data were markedly non-normal.³⁰ Additionally, only 7 (17.5%) out of 40 evaluations (made at 5 clinics by 8 raters) gave scores of 3 for domain 2, while the rest yielded scores of 2. None of the raters indicated scores of 1 or 4, resulting in the lower ICC with wide CI. Thus, although this outcome displayed a lack of variability in component 2 scores, it was retained to ensure the content validity of the checklist. Nonetheless, the ICC values for individual score and total score were still in the range of good to excellent.

The study has several limitations. For example, we were not able to select more public primary care clinics in Selangor because not many such clinics fulfilled the criteria for implementing FDC for at least 2 years, which was the MOH’s established timeframe for the clinics to adapt to the new approach. The checklist allows the identification of components that require improvement and the identification of potential barriers to the successful delivery of FDC but, it can be further improved by testing the construct validity for each component using a larger sample size. Other limitations involved the lack of variation in the raters. All 8 raters who participated in the study were medical officers with some experience in research. Moreover, all of them had at least a medical degree, along with more than 5 years of service in the MOH.

Conclusion

This study successfully developed an instrument for evaluating FDC implementation fidelity in public primary care clinics. The resulting instrument is valid in terms of content and face validity and has a good to excellent inter-rater reliability with adequate internal consistency.

The inter-rater reliability of individual domain scores was excellent, except for component 2, while the inter-rater reliability of the total score was excellent.

Conflicts of Interest

All authors declare that they have no conflict of interest.

How does this paper make a difference to general practice?

• This paper presents an instrument to evaluate implementation fidelity of the Family Doctor Concept (FDC) program established in public primary care clinics.

• The standardised instrument to evaluate implementation fidelity developed can be used in all public primary care clinics so that comparison can be made between them.

• This paper highlights the importance of fidelity to ensure effective implementation of various programs in public primary care clinics