EVALUATION OF DEVELOPMENTAL SURVEILLANCE BY PHYSICIANS AT THE TWO-MONTH PREVENTIVE CARE VISIT

Abstract

We evaluated the effects of feedback and instruction on resident physician performance during developmental surveillance of infants at 2-month preventive care visits. Baseline data were obtained by videotaping 3 residents while they performed the physical and developmental exam components. Training consisted of individualized feedback and a brief instructional module, after which the residents were again videotaped while they performed preventive care visits. All 3 residents showed improved performance following training.

Approximately 17% of all children in the United States have a developmental disability (Boyle, Decoufle, & Yeargin-Allsoup, 1994). To ensure appropriate medical care of these children, the American Academy of Pediatrics established recommendations for developmental surveillance (informal developmental assessment and monitoring) during all preventive care visits (Council on Children with Disabilities et al., 2006). The policy outlined five components of developmental surveillance, one of which is observation of the child through physical and developmental examinations. In addition, the Accreditation Council for Graduate Medical Education and the American Board of Medical Specialties have mandated the adoption of competency-based assessment by training programs.

Leake, Barnard, and Christophersen (1978) evaluated the effects of training and feedback on performance during preventive care visits. However, no experimental studies have been published on training pediatric residents to conduct developmental surveillance. This study provides a systematic replication of the procedures described by Leake et al. during the physical and developmental examination component of developmental surveillance at the 2-month preventive care visit.

METHOD

Participants and Setting

The study was conducted at an academic children's medical center between August, 2007, and July, 2008. Participants were pediatric resident physicians in their 2nd year of training. Recruitment was initiated by providing information to the residency class regarding the general topic (child development) and methodology of the study without disclosing the exact behaviors being observed. Residents participated voluntarily and signed consent forms prior to enrollment. Seven resident physicians, all in their mid-twenties and including two men, volunteered to participate; however, only three completed the study. (One man and three women did not complete the study due to personal issues.) Each participant was observed and videotaped while performing preventive care visits for 2-month-old children scheduled through the residents' general pediatric clinic. Only visits for children with parents fluent in English and children without previously identified genetic syndromes, medical conditions, or developmental delay were eligible for observation. Parents provided permission for videotaping the visit for research purposes prior to the visit. Neither the resident nor the family received compensation for participation.

Dependent Variables

The videotaped preventive care visits were reviewed for the presence or absence of resident assessment, either by history or physical examination, of five specific developmental components: (a) the infant's ability to follow an object past midline (the resident was expected to attempt to gain the infant's attention using an object or light and attempt to entice the infant to follow the object or light past midline); (b) the infant's ability to hold an object placed in his or her hand (the resident was expected to attempt to place an object such as a tongue depressor in the infant's hand to assess whether the infant was able to hold it briefly); (c) the infant's ability to demonstrate a social smile (the resident was expected to either attempt to engage the infant to elicit a smile or ask the parent if the baby smiles at him or her socially); (d) the infant's ability to coo (the resident was expected to either attempt to have a “cooing conversation” with the infant or ask the parent if the infant makes cooing or vowel sounds); and (e) the infant's ability to lift his or her head and shoulders off the table while in the prone position (the resident was expected to place the infant in the prone position and look at the infant to assess whether the infant was able to complete this skill). The resident's score was not affected by the infant's ability to complete any of these skills, but it did require that the resident's physical examination techniques were used appropriately in an attempt to evoke the behaviors. For example, the resident was not given credit for assessment of the infant's ability to follow an object past midline if the object was moved too quickly or held too closely (less than 8 to 10 inches) to the infant's face. Components of assessment could occur anytime during the visit, which was approximately 20 min in duration.

Design and Procedure

A nonconcurrent multiple baseline design across participants was used to demonstrate experimental control. To establish baseline, participants were observed and videotaped while they performed a minimum of three 2-month-old preventive care visits until a stabilization of skills was reached. Stabilization of skills was defined as minimal variability in skills or absence of a trend toward improvement. Once baseline was established, the intervention was administered individually to each participant within 24 hr of baseline. During the intervention, an experimenter gave the resident individual feedback on his or her performance and administered the educational module. Individual feedback consisted of outlining the purpose of the study and the five targeted assessment components. The resident was then provided with his or her scores for completed visits (e.g., “You completed three of the five expected components of assessment. The components you missed were the infant's ability to hold an object placed in hand and the infant's ability to coo.”). The educational module was lecture based, with video components demonstrating assessment of milestones expected of 2-month-old infants as a model of behaviors. The educational module was standardized by using PowerPoint slides that contained the videos described above and a listing of the developmental milestones so that each resident received the same intervention. Both components of the intervention were administered during the same 20- to 25-min period. After exposure to the intervention, the resident was observed and videotaped while he or she performed the next three scheduled 2-month preventive care visits. The amount of time that separated each visit varied due to scheduling technicalities.

Data Collection and Reliability

The videotaped examinations were viewed and scored independently by the primary investigator and by a subinvestigator who was blind to the experimental condition (i.e., baseline or postintervention) of the preventive care visit. The reviewers assessed the presence or absence of resident surveillance of developmental status based on the five components outlined previously. To obtain a measure of interobserver agreement, a second observer reviewed a minimum of 25% of the tapes in each condition (baseline, postintervention) for each participant. Agreement was calculated by summing the agreements on occurrence and then dividing by the total number of occurrences measured. The two observers disagreed on just two occurrences across all participants, both of which were the assessment of the infant's ability to follow an object past midline. This resulted in a mean interobserver agreement score of 94% (range, 80% to 100%).

RESULTS AND DISCUSSION

Figure 1 shows the number of assessment components completed by each resident during the 2-month preventive care visits. None of the residents completed all of the components during baseline; however, all three residents improved their assessment following receipt of individual feedback and the educational module. The first resident was observed three times during baseline and assessed the infant's ability to demonstrate a social smile during all of the visits and the ability to coo during two of the visits. After the intervention, this resident assessed all of the components except the infant's ability to hold an object placed in his or her hand during each visit. The second resident was observed four times during baseline and assessed the infant's ability to follow an object past midline, demonstrate a social smile, and coo during three of the visits and the infant's ability to lift his or her head and shoulders off the table during two visits. The resident did not assess the infant's ability to hold an object placed in his or her hand during any of the visits. After the intervention, this resident neglected to assess the infant's ability to hold an object placed in his or her hand during the first two postintervention visits and did not assess the infant's ability to coo during the third visit. The third resident was observed five times during baseline and assessed the infant's ability to lift his or her head and shoulders off the table during each of the visits. The resident also assessed the infant's ability to demonstrate a social smile during four of the visits, coo during three of the visits, and follow an object past midline during one visit. After the intervention, this resident assessed all five criteria during each visit.

Figure 1.

Number of the five components of developmental surveillance of infants conducted by resident physicians at the 2-month preventive care visit.

The results of this study indicate that individual performance-related feedback coupled with a standardized educational module that contained video components demonstrating assessment of milestones is an effective method to increase resident physician demonstration of competency in developmental surveillance at the 2-month preventive care visit. This study is a systematic replication of Leake et al. (1978) and successfully extends their work to include developmental assessment. A strength of this study is that the behavior of the residents during the examination was directly measured; we did not rely on questionnaires or surveys.

Residents enrolled in the study on a volunteer basis. Therefore, certain characteristics of the participants, such as a particular interest in child development, future career goals, or a heightened motivation to pursue individualized learning, may have influenced the results. Similar results might not have been found if the study had included the entire resident physician class. Also, absent follow-up data on resident performance, long-term effects of the intervention remain unknown. An additional limitation of this study was the brief time period over which it was conducted; as a result, the baseline lengths were staggered by only one visit.

It should be noted that the residents enrolled in this study were not novice pediatric residents. All had followed patients and performed preventive care visits as part of their weekly general pediatric clinic for at least 1 year prior to enrollment, yet skills displayed at baseline were marginal with regard to developmental assessment. The education the residents had received was in a textbook or other written format. No prior formal demonstrations of skills or instruction had been provided on this topic. Interestingly, the baseline observations were quite similar to those reported by Leake et al. (1978) 30 years ago.

Future research should separate the components of this intervention to determine if the results were due to the performance-related feedback, the educational module, individualized attention in the area of education, or a combination of the three. Future research should also evaluate the extent to which maintenance of treatment effects is observed by conducting follow-up assessments several months after the intervention.