Assessing the impact of a web-based comprehensive somatic and mental health screening tool in pediatric primary care

Abstract

Objective

To evaluate how parents and physicians perceive the utility of a comprehensive, electronic pre-visit screener, and its impact on the visit.

Methods

A mixed methods design was used. English speaking parents were recruited from three primary care systems (urban MD and rural NY and VT) when they presented for a well child visit with a child 4 to 10 years of age. Parents completed an electronic pre-visit screen which included somatic concerns, health risks, and four mental health tools (SCARED5, PHQ-2, SDQ Impact, and PSC-17). Parents completed an exit survey, and a subset were interviewed. All PCPs were interviewed.

Results

120 parents and 16 PCPs participated. The exit surveys showed that nearly 90% of parents agreed or strongly agreed that the screener was easy to use and maintained confidentiality. During interviews, parents noted that the screener helped with recall, validated concerns, reframed issues they thought might not be appropriate for primary care, and raised new questions. PCPs felt that the screener enabled them to normalize sensitive issues, and simultaneously focus and be comprehensive during the visit. Parents and PCPs agreed that the screener helped guide discussion, promoted in-depth exchange, and increased efficiency. Findings were consistent across quantitative and qualitative methods and between parents and PCPs.

Conclusion

A comprehensive electronic pre-visit screening tool is an acceptable and practical strategy to facilitate well child visits. It may help not only with problem identification, but also with agenda setting, family engagement, and balancing of attention between somatic and psychosocial concerns.

INTRODUCTION

Screening for mental health problems with self-report questionnaires in well child visits is one way to enhance the detection of child mental health problems in primary care,^1–7 but many barriers that have limited its widespread use. Many popular instruments are condition-specific, and thus a comprehensive assessment of a child’s problems could require using a variety of tools and be difficult to administer, score, and interpret.⁸ With less than 20 minutes typically allotted to a pediatric well-child visit, screening during the visit cuts the time for discussion considerably.^9–12 Finally, use of a mental health screener may not be viewed as a primary care priority when it is competing with screening for developmental issues, safety, and nutrition, which are central to pediatric practice.¹³

Pre-visit screening has been proposed as a way to increase efficiency of by reducing in-visit time spent on assessment and allowing more time to discuss concerns.^14–18 Web-based pre-visit screeners have the potential to further improve efficiency by eliminating some administrative tasks, reducing response burden with the use of skip patterns, and automating scoring.⁸ Studies in which adolescents have used computerized or web-based screens in a variety of medical settings have found increased disclosure and improved agenda setting, ^3,7,19–20 and teens have provided favorable feedback about the impact on communication and quality of care.^20–22

The impact of computerized or web-based screening on disclosure or engagement in visits with younger children has not been examined, nor has the impact of embedding mental health screening within a set of questions about other concerns usually targeted for surveillance or anticipatory guidance during primary care visits. Only three studies have examined computerized or web-based mental health screening for school-aged and younger children,^23–25 but they did not assess the impact of their interventions on visit processes.

The goal of the present study was to determine parent and provider impressions of the impact of a using web-based, pre-visit screener for school-aged children. The screener, which was completed by parents, embeds mental health items within a larger set of environmental, somatic, and behavioral concerns rather than focusing solely on mental health. As with adolescent screening, there is a suggestion in the literature that screening can promote parents’ willingness to discuss child mental health problems, independent of the screening results themselves.²⁶ We hypothesized that use of a comprehensive screener would be seen by both parents and primary care providers as increasing visit efficiency and helping to orient the visit to those issues – psychosocial or somatic -- of greatest importance to the parent.

METHODS

Instrument Development

The instrument used in this study was adapted from the Healthy Families Screener developed by a team at Dartmouth Medical School based on national recommendations.²⁷ The screener assesses common concerns, nutrition, physical activity, safety, and oral health issues recommended for well visits of children ages 4–10. It comprises 25 root questions that can branch to as many as 57 based on parent answers. The two item version of the Patient Health Questionnaire (PHQ-2)²⁸ was included to screen for parental depression as per national recommendations.²⁹ Three widely used and validated mental health assessment tools were included: 1) the PSC-17, adapted from the original Pediatric Symptom Checklist,³⁰ validated against structured psychiatric interviews, and with recommended cut-points indicating possible clinical diagnoses; 2) the 8-item “Impact Supplement” from the Strengths and Difficulties Questionnaire (SDQ),³¹ which also has recommended cuts-points for possible clinician diagnoses and asks questions about mental health-related functioning that are not included in the PSC-17; and 3) the SCARED-5, adapted from the Screen for Child Anxiety & Related Disorders³² and used to screen for childhood anxiety disorders. The screener is available in English; the reading level is grade 5.4.

The screener is designed so that all respondents are asked about somatic and mental health concerns, general health risk, anxiety (SCARED-5), parental depression (PHQ-2), and whether or not the child is having difficulties due to emotions, attention, behavior, or getting along with others. Those who report emotional, attention, or behavioral difficulties are then given the PSC and SDQ Impact questions. The screener automatically calculates scores for the validated scales it contains. A summary screen tallies positive responses within broad categories and highlights scores for the validated assessments above the standard cut offs. In addition, all individual responses are listed and can be quickly reviewed by scrolling through the summary report.

Sample

The study was conducted in four primary care pediatric practices across three sites. Pediatricians were recruited from South Royalton and Bradford, VT, in two community-based rural clinics serving a predominantly lower income mostly white rural population, and from a hospital-based clinic serving a similar population in Cooperstown, NY. Pediatric residents were recruited from an urban, hospital-based primary care practice in Baltimore, MD, serving a predominantly low-income African American population. Here we refer to both the pediatricians and the pediatric residents as PCPs. PCPs at each site were trained briefly on how to interpret the questionnaire’s raw and summary results and how to respond to concerns raised by the questionnaire. Each site had a standing protocol on how to respond to the identification of a mental health problem, and at the Maryland site, co-located mental health clinicians and attending pediatricians were available for consultation. The New York site had previously used paper pre-visit questionnaires, but the other sites had not previously used systematic pre-visit screening.

We recruited English-speaking parents of children 4–10 years of age without major developmental or medical problems who were being seen by a PCP for health maintenance visits. Upon arrival, parents were invited to participate. Those who agreed completed the screener before their child was seen by the PCP. A description of the enrolled children is presented in the Results section below.

Study Design & Procedures

The study used a mixed-methods design with both qualitative and quantitative methodologies. The three research sites obtained approval from their respective Institutional Review Boards. As part of the protocol, parents were told that the purpose of the research was to determine if electronic screening is acceptable and helpful to parents and doctors.

At the Maryland site, parents completed the screener on a computer in the exam room prior to the PCP’s arrival, and the time taken for completion was recorded. The research assistant offered help with completion but did not actively participate or observe the parents’ responses. A few parents asked for clarification of some questions, but none required having the instrument read to them. Paper copies of the results screen were given to the parents and PCPs just before the visit. At the Vermont sites, parents completed the screener on tablets in the waiting room, and staff provided the parent with information about the project and oriented the parent to the electronic entry process. At the New York site, the PCP introduced the screener to parents, and parents completed the screeners on a tablet in the waiting or exam room. The parent was able to view the summary report once they completed the questions, and the tablet was given to the PCP to review the summary report prior to meeting with the parent.

At the end of the visit, the parent completed a 3-minute paper exit survey focusing on the screener content and format. At the Maryland and New York sites, parents were asked if they were willing to provide contact information for a subsequent telephone interview. From among these parents, blinded to exit survey results, we selected a subset (at least 1–2 per participating PCP) for a call within two weeks of their visit. We attempted calls to 15 parents and successfully interviewed 12 at each site. Across the three sites, all participating PCPs were interviewed by phone for 15–20 minutes.

Measures

The Parent Exit Survey included 11 questions asking about comfort with the screener format and content. Parents responded using a scaled format capturing degree of agreement with statements about the format and content, with areas for free text (See Appendix A).

The Parent Interview Guide included open-ended questions about four general domains (See Table 1). It first asked for general feedback on the visit and included probes about how the parent felt about the discussion and how it may have differed from prior visits. Then it asked about the questionnaire content and format, with various probes such as how the screen questions compared to questions usually asked by the doctor. The third area of inquiry focused on the questionnaire results, and probes included questions on how the doctor talked about the results and how the parent felt about them. The final section asked about the impact of the questionnaire with probes touching on how the use of the questionnaire affected recall and disclosure, the discussion with the doctor, and the doctor’s understanding of the parent’s concerns.

Table 1.

Primary Questions and Select Probes from the Parent and Provider Interview Guides

Domain	Parent Interview Guide	Physician Interview Guide
Opener	As best as you can remember, please describe what happened at the visit from the time you completed the questionnaire until the time you left. How did you feel about the visit/discussion? How did the visit differ from what was expected, if at all? from other visits in the past?	How do you usually figure out what emotional and behavioral problems a child might have? What are all the sources of information you use to identify mental health problems? What process do you use to determine the nature and severity of a MH problem?
Questionnaire Content	If you can remember, please describe what you thought about the questionnaire content. The questionnaire included questions on several different topics. Are there any questions you remember that you think should not be asked in a questionnaire? Were there any questions that you would prefer the doctor ask you in person?	What suggestions do you have for improving the content, if any? Do you have comments on the scope of the topics covered? the order of questions? the length of the overall questionnaire? other?
Questionnaire Format	Could you also describe your thoughts about using a computer to fill in the questionnaire? What do you think about doing this type of questionnaire on a computer? How was the length of the questionnaire?	How was it to obtain and read the results on the computer or from the computer printout? What did you think of the format of the questionnaire results? What do you think are the advantages and disadvantages of using a web-based screener?
Results Delivery/Impact on Efficiency	If you were given the results at the visit, could you describe how the results were given to you? Could you describe how the results were presented on the computer? What did you think about this type of summary of your results? During the visit with the doctor how did s/he bring up the results? How did s/he talk about them?	How did use of this questionnaire impact efficiency, including time, effort, energy, stress, or hassle (if at all)? How did it affect the time/effort spent on determining how to manage any problems identified? Would the screener act as a time-saver or a potential additional burden to regular practice procedures in terms of time and effort?
Impact on Visit	How do you think filling out the computerized questionnaire affected your visit, if at all? What kinds of things, if any, did the questionnaire make you think of to bring up with your doctor that you had not thought of before? How did having the questionnaire results affect your conversation with your doctor (if at all)?	In general, how do you think use of the questionnaire affects visits with patients and their families? How did have the screener affect your exchange with the patient and parent? How did the use of the questionnaire affect your ability to understand and address parents’ concerns?

In the Physician Interview Guide the first question asked about current practice for mental health assessment, and the probes included questions on sources of information and barriers to assessment (See Table 1). The second domain of interest was the questionnaire content with probes on scope, order, length. A third area in the guide asked about efficiency, and probes covered how the use of the screen may have affected time managing concerns. The final section focused on the impact of the questionnaire with probes how it affected assessment, treatment decisions, the exchange with parents, and family willingness to seek treatment.

Data Analysis

Web-based Screener

The screening data were analyzed with SPSS v.19. We conducted descriptive analyses, including frequencies for each item as well as subscales. For example, for the PSC, we examined symptom frequencies, scores for the three subscales (internalizing, externalizing, attention), and total score.

Exit Survey

Using SPSS, we calculated frequencies for each item and tested the differences between rural and urban sites using chi-square tests of association, with significance at p<.05.

Parent and Physician Interviews

Each interview was audio-recorded and transcribed, and inductive thematic analysis was used to identify themes and patterns in the data.³³ For the parent and provider interviews, researchers read through the transcriptions and identified passages that related to the acceptability (e.g., content, format) and impact of the screener (e.g., on disclosure, engagement). The transcripts were coded by hand (by three readers for the parent interviews and two readers for the provider interviews). For both sets of interviews, the readers met to review the coded data and develop consensus among the emerging themes. Then for each theme, sub-dimensions were identified, and any discrepancies were resolved through consensus, which resulted in refining, eliminating, or expanding codes.

RESULTS

Characteristics of Sample

Total sample

Table 2 summarizes the total sample. In total, 172 parents were eligible for the study, and 120 (70%) were recruited and completed the screener. Of the 52 eligible who were not recruited, 12 (23%) declined; the other 40 were not recruited for various reasons: clinic concerns about time and patient flow (14); technical difficulties (8); PCPs thought the family would be too difficult (6); the RA was late or with another patient (6), the patient was recruited for another study (2), unknown reasons (4). Of participating parents, 68 (57%) were from rural sites, 104 (87%) were female, and the average age was 33. Just over 51% of the children were male, and the average child age was 7.

Table 2.

Frequency of Select Characteristics of Participants in Web-based Screening, Exit Surveys, and Interviews

Site	Screeners	Exit Surveys	Parent Interviews	PCP Interviews
MD (urban)	52	49	12	9
VT (rural)	45	20	0	4
NY (rural)	23	21	12	3
TOTAL	120	90	24	16
Average age	6.78 (child)	33 (parent)	NA	NA
% rural	56.7	46	50	44
% female	47.7	87	83	81
% residents	NA	NA	NA	56

Samples by site

At the urban site in Maryland, nine categorical pediatric residents participated (one PL-2 and eight PL-3). Fifty-two parents completed the screener, of which 49 (94%) completed an exit survey. Qualitative interviews were conducted with 12 parents.

At the two practices in Vermont, 4 PCPs participated, and 45 parents completed the screener, of which 20 (44%) completed exit surveys. At the clinic in New York, 3 PCPs participated, and 23 parents completed the pre-visit questionnaire, of whom 21 (91%) completed the exit survey and 12 were interviewed. The 7 PCPs at the rural sites had been in practice from 3–30 years.

Screening Results

As shown in Figure 1, 34% of parents had some type of somatic concern, and 25% had concerns about their child’s behavior, mood, fears, or worries. Almost 13% of children had been previously diagnosed with a mental health problem, and nearly 13% scored positive (>=3) on the SCARED-5. Just over 33% reported that their child had minor, definite, or severe difficulties with emotions, paying attention, behavior, or getting along with others. Those who reported any difficulties (n=39) were then asked the PSC and SDQ Impact questions. Of this subgroup, 39% had an abnormal score (>=2) on the SDQ impact measure, 29% had a positive score (>=15) on the PSC, and 53% wanted help with their child’s behavioral and emotional problems. When considering these as rates among the full study population (N=120), 13% had abnormal SDQ Impact scores, 9% had abnormal PSC scores, and 17% wanted help with their child’s behavioral and emotional problems.

Figure 1.

Screener Skip Pattern and Responses for Select items

Of all children in the study, urban children were significantly more likely to have an abnormal score on the PSC attention subscale than the rural children (40% vs. 6%, p=.013) and more likely to have an abnormal score on the SCARED-5 test for anxiety (21% vs. 6%, p=.012) (data not shown). In addition, parents from the urban site were significantly more likely to have abnormal scores on the PHQ -2 measure of depression (23% vs. 9%, p=.039). Finally, parents at the urban site were significantly more likely to want help with their child’s school problems than parents at the rural sites (45% vs. 6%, p=.013). At the one site where screener completion was timed, the screener took on average 6.27 (range 3.55–13.03) minutes to complete.

Exit Surveys

Overall, the exit surveys indicated high acceptability of the screener among parents. (Table 3). Nearly 90% trusted the security of the screener, and 87% thought their answers would be confidential. Ninety-two percent thought the screener was a good way to ask routine questions, 95% were comfortable with the mental health questions, and 89% thought the screener helped with sharing of concerns. Only 7% found it difficult to use the electronic device (computer or tablet). There were no significant differences in acceptability of the screener between the urban and rural sites.

Table 3.

Select Exit Survey Results (N=90)

Survey Item	% Agree/ Strongly Agree
Answers were confidential	87
Trust the security in place	89
Good way to ask routine questions	92
Helped me share my concerns	89
Difficult to use the handheld/desktop computer.	7
It took too long to complete the computerized screener	4
Had trouble entering responses, for example, touching the right spot or using the mouse	7
	% Comfortable/Very Comfortable
Questions about the concerns I wished to discuss	91
Questions about child’s mental health	95
Questions about child’s difficulties caused by emotional, behavior, social or learning problems	87
Questions about family issues (stress and other problems)	90
Questions on health issues such as BMI, nutrition, exercise	87
Questions on other health issues, such as safety.	89

Parent Interviews

The interviews provided a greater depth of understanding about the acceptability and impact of the screener. Four themes emerged from the qualitative analyses: visit format and content; problem identification and disclosure; discussion with the doctor; and visit efficiency.

Format and Content

Parents described the screener as easy to use and fast. They liked the color-coding used on the summary screen, noting that this was an easy and quick way to see the issues of concern. Parents did not find the mental health questions intrusive.

One parent at a rural site noted, “I think the thing I liked the best about it was it generated a report at the end which highlighted the areas of concern that the doctor really needed to touch on.”

Problem Identification and Disclosure

Parents expressed several ways in which the screener helped with problem identification. For some it reminded them of concerns they wanted to discuss with the pediatrician. For others, the screener touched upon issues they had not thought of or had not considered discussing with the doctor. Parents also reported that identification of issues on the screener prior to the visit helped them feel comfortable bringing up their concerns with the doctor and gave them confidence that the doctor would be interested in these concerns.

A parent from an urban site expressed, “I think it made me see problems in a different light, things I wouldn’t have thought of, that I thought about it and I felt like well, maybe the doctor has other resources like behavior. Sometimes people feel like behavior isn’t something the doctor can talk about.”

Discussion with Doctor

Parents noted that completing the screener in advance and highlighting concerns on the summary screen helped to focus the conversation with the doctor around issues that were of most concern to them. In particular, some commented on how the focus on their concerns allowed for a more in-depth discussion that they might not have had otherwise.

One parent (urban site) noted, “Well, I know the first thing I noticed that was different [when using the screener][was]when the doctor came in is she already knew what points to hit on first, it’s all on the questionnaire, so it made the rest of the visit transition easier.”

Visit Efficiency

Of note was the fact that parents felt the screener saved time on the front end of the visit, and therefore allowed for more discussion of important concerns. They felt the PCPs could focus more clearly because they did not have to ask a battery of questions at the beginning of the visit.

Provider Interviews

Similar themes emerged from the interviews with the PCPs. These included: content and format; problem identification and disclosure, engagement, and efficiency.

Format and Content

The PCPs’ comments on format and content related to the summary report of the screening results. They noted that a major deterrent, calculating scores from the paper and pencil screeners, was eliminated with the computerized version. PCPs also noted how helpful it was to know that the parent had been asked about a full set of issues, eliminating the need for the PCP to take the time to assess these issues. They also noted that several of their patients’ parents had commented positively about the inclusion of mental health and other non-physical health issues on the screener.

PCP1 (rural): “Most of the parents seem to like it. They like going over the summary screen. They like seeing, you know, stopping at the elements of the summary screen that has yellow and red highlights. They seem to react favorably to the screening in that they feel it’s pretty comprehensive, meaning that it covers a lot of things that are important, that should be asked during the visit.”

Problem Identification and Disclosure

PCPs commented frequently about how they thought the use of the screener helped the parents identify the concerns they wished to discuss and feel comfortable bringing them up.

PCP7 (rural): “I think it gave the parents more of a forum to tell me what their concerns were, where sometimes they might have had trouble bringing it up. I guess it made it more accessible.”

PCPs also noted that some parents had not previously realized they could discuss certain issues (e.g., mental health, family stress) with their pediatrician, and by seeing these issues on the screener, the parent realized they could bring them up in the visit. A few PCPs commented that the screener helped them identify issues they had not previously discussed with patients whom they knew well. The use of the screener helped direct PCPs to probe these issues further during the visit.

Engagement

PCPs noted that the screener enhanced family engagement. Some noted that because the parents knew that all parents were being asked the same questions, they did not feel their child was singled out for having any particular problem. Since sensitive topics were introduced in the screener, it was easier for the PCP to raise these issues with the parent. The provider felt this made it easier for the parents to talk about these issues. A few PCPs found that the screener results provided “objective support” for the need for treatment, raised parental awareness of the seriousness of the problem, and allowed parents to more readily engage in care.

PCP9 (urban): “It made it a little easier because the information is already out there. The parents already know it’s going to come up and so they’re probably less worried or less anxious about talking about it.”

Efficiency

PCPs reported consistently that the use of the screener improved the efficiency of the visit. By having all the questions asked of parents prior to the visit, the PCP did not have to take time to go over everything on the usual list. Instead, s/he could focus the discussion and narrow the priorities of the visit. Some noted also that use of the screener helped decrease the number of “door knob” questions—those questions that parents bring up just as they are walking out the door.

DISCUSSION

This study found that both parents and PCPS thought a comprehensive web-based screener for parents of children aged 4–10 years was acceptable and positively affected the well-child visit. These positive reports were among families in both urban and rural settings and among both pediatric residents and practicing pediatricians. Exit survey responses were almost entirely favorable about the screener content and format. Parents said they understood the items, found the content appropriate, and were able to use the electronic platform, findings similar to studies of screening with adolescents.^8,20 These results indicate that a comprehensive instrument that includes mental health questions could be acceptable for use with parents of younger children.

Interviews with the parents and providers provided insight on the impact of using the screener on the visit. As previously found with adolescents, use of the screener did not appear to supplant discussion with the PCP but rather seemed to enhance it by providing a structure around which to address priority concerns.⁸ One important function of the screener seemed to be its ability to signal to parents the acceptability of discussing topics they had assumed to be out of scope for a pediatric visit. This might make a systematic, comprehensive approach to screening more desirable than that proposed in a recent study by Anand and colleagues²³ where screening items were selected by computer algorithms based on each child’s known history. In addition, parents noted that both the comprehensive nature of the screener and the fact that it was “routine” helped them feel comfortable bringing up sensitive issues without feeling targeted or stigmatized. In a study of behavioral health screening in an emergency department, Pailler et al, (2009) found that adolescents and their caregivers raised concerns about feeling singled out and preferred processes that avoided targeting.³⁴

The study has several limitations yet points the way for additional work. Though the technology required to use a web-based screener is available in most practices, not all may have designated pre-visit time or staffing necessary for screening. This study did not measure the additional staff time needed to present the screeners to families or assist them, when necessary, with completion. Also, not all practice sites may have the capacity to respond to positive screens. While the present study results establish the acceptability of using electronic comprehensive screening for children, the need for improving the capacity of PCPs to respond to mental health concerns, particularly in low-resource areas, should be prioritized. Expansion of mental health “access” programs such as those in Massachusetts and Washington may promote wider use of screening without concern for care of newly detected problems.^35–36 Further, although several electronic behavioral health screens are available for use in primary care,^{8, 20, 22} we know of only one electronic tool with broad somatic and behavioral coverage that is readily available for use with younger children, and it is significantly longer and proprietary, and its impact on visit process has not been reported.²⁴

We did not conduct a head-to-head comparison of comprehensive versus targeted mental health screening, and so can only provide preliminary data regarding the benefits of the comprehensive approach. We also did not record visits, so we can only use PCP and parents reports as a basis for the finding that attention to these concerns seemed balanced and satisfactory to both parties. These reports could be subject to some social desirability bias, but we sought to minimize this by having a member of the team who was not involved in recruiting conduct the interviews and by doing them by telephone after the study visits had been completed.

Because we included only four primary care practices in the northeast U.S., our findings have limited generalizability to other areas and language groups. Although we did not observe differences between residents and more experienced physicians, our sample size precludes firm conclusions about what differences there may be. Further study is needed to determine whether PCP and parent race/ethnicity, or parent socioeconomic status and health literacy may affect the feasibility and acceptability of electronic screening.

Conclusion

Parents and PCPs reported that use of a comprehensive, electronic, pre-visit screening tool for health maintenance visits among children 4–10 helped with parent engagement, parent-doctor communication, agenda setting, and visit efficiency. The impact on visit processes is a novel finding, though additional research is needed to learn more about its underlying mechanisms. The high level of acceptability and benefit found in this study, if replicated, could motivate efforts to overcome some of the logistical barriers that currently stand in the way of screening implementation.

What’s New.

Comprehensive screening in well-child visits has the potential to do much more than detect physical and mental health concerns. Rather than developing an unmanageable list of concerns, parents and physicians agree it can improve processes such as agenda setting, patient disclosure, doctor-patient communication, and visit efficiency.

Appendix A: Parent Exit Survey