Table 2.

Patient and clinician feedback about PatientViewpoint functionality

	Patient feedback	Action
Round 1
E-mail reminders	Unclear that e-mail reminders are from Johns Hopkins Subject line unclear	Add “Hopkins” to e-mail Subject and From lines
	Some reminders going to spam
	Reminder not generated if questionnaire assigned within 3 days of visit	Reminder generated immediately if within 3 days of visit
	Need more than 1 reminder	Reminder frequency increased to 3 times
Completing the survey	No warning that there is a character limit for entering free-text description of important issues	Warning added about limit on free-text characters
	Weak WiFi access in clinic	x
Viewing the results	Unclear that must hit Submit button to get graphical results	Submit button made clearer
	Meaning of numbers is unclear	x
	Unaware of option to get explanation of PRO domain	Added explanations on results page
	Unclear what yellow highlighting of scores means
	Not obvious that trend in score over time is being displayed
	Unclear if higher score indicates better or worse health	Added arrows to indicate direction better/worse
	Display on 0–100 scale difficult to interpret, especially if, e.g., a score SO is normal
	Description of score is too technical (e.g., standard deviation)	Study
	Want comparison to normal scores for “someone like me”	x
	Want tailored description of own score
	Scores should go in same direction for all scales	Study
Other comments	Want program to remember their password	x
	Want free-text option after each group of questions	x
	Clinician feedback
Comments	Scores don’t really tell them anything new; patients bring up same issues during visit	–
	Time required for viewing is a barrier to use	–
	Want more lead time to have the survey completed before the visit day	x
	Want direct link from website to patient record in EHR without multiple logons	x
	Want direct link from email reminder to patient results	x
	Prefer to be given hard copy of results	Results can be printed in clinic
	Want to be prompted to review results	x
	Links to explanations are not obvious	Reference to explanations added on results page
	Want simpler language to explain meaning of scores	Study
	Scores should go in same direction for all scales	Study
	Want free-text comments to be more obvious	Moved text comments to top of page
	Like graphs and yellow highlighting of problematic scores	–
	Want patients with highlighted results to be highlighted	x
	Want customized list of own patients	x
	Want program to remember their login/password	x
	Results in EHR are more convenient, but PatientViewpoint website provides better information	–
	Want easy, one-click printing	x
System Issues	Need alert when there is missing data	Added alert for missing data
	Need to be able to change frequency of completion for already assigned questionnaires	x
	Difficulty adjusting ordering frequency if previous survey in a series was already completed	x
Round 2
E-mail reminders	Emails work fine and liked emails	–
	Value link from email reminder to survey	–
	Adding “Hopkins” to From and Subject lines is helpful	–
	Some reminders still go to junk mail	–
Completing survey	Able to complete with no problem	–
	Prefer paper to computer completion	–
	Appreciate that scrolling not needed within surveys	–
	Annoyed with passwords but like ability to change to own choice	–
	Survey helps identify issues that might not come up otherwise, reminds of things to discuss with clinician	–
	Free-text comment boxes useful	–
	Difficult to print results from home	–
Viewing the results	Value ability to track scores over time	–
	Rely on the arrows to determine which direction is better/worse; variation in direction of scores confusing to some	Study
	Yellow highlighting helpful	–
	Cut off scores currently result in too many “yellow” highlighted scores	Study
	Use explanatory links infrequently	–
	Difficult knowing what the numbers mean	Study
Other comments	Several useful features of the system	–
	Value completing from home, because it addresses things that come up between visits which they may not necessarily bring up on their own during a visit	–
	If doctor does not look at it, feels like it is a waste of time	–
	Would like opportunity to comment at the end of each set of questions	x
	Would like descriptive comments right next to graphs	Reference to explanations added on results page
	Would like directional arrows to be more noticeable	Directional arrows made more prominent
	Defer to doctors for suggestions of what to do	–
	Want to see complete score history; not just last 5 scores	x
	Want more guidance to help navigate through completing the questionnaire, submitting comments, seeing scores	x
	Want to know when doctors see scores	x
	Clinician feedback
Comments	Improved ability to find patients	Improved searching function
	Would like the graphs to be displayed in EHR	Defer pending Epic migration
	Would like to be able to link scores with other clinical information
	Too much work to logon to PatientViewpoint website separately	x
	PRO results helpful as a reminder	–
	In many cases, scores confirm what they already know; provide verification about how patient is doing	–
	Leaves it up to the patient to bring up problems even if scores are low	–
	Only looks at the data if patient completed before coming to clinic	–
	Helps identify/track issues	–
	Like free-text comments	–
	Want description of meaning of scores put right next to graph	Study
	Like arrows but could be more prominent; should be included in description of what scores mean	Study
	Like graphic score report; would like “normal” range shaded	Study
	Would like more detailed referrals to specific resources for “What can I do”	x
	Would like smarrphone app	x
System issues	May forget password	x
	Learn from the links early on, then do not use as often later	–

– Not applicable, x not done

EHR electronic health record, PRO patient-reported outcome