Table 1.

Lessons Learned and Recommendations to Facilitate CDS Development and Interoperability

	LESSONS LEARNED	RECOMMENDATIONS
Early stakeholder engagement	Early engagement of health system IT leadership allowed introduction of project’s goals, networking with leaders locally, and access to EHR training coursework that will support integration of tool with EHR.	Engage key stakeholders early to facilitate open dialogue and support when implementation challenges arise.
	Prototype pilot testing addressed gaps in instructions provided to new users.	Pilot test prototype with new users prior to usability testing to identify gaps that developers may overlook due to familiarity with their tool.
	Patient input is challenging. A patient advisory committee was recruited from focus group participants for future input on tool development.	Involve patient end users at every stage of development. There is a growing community of patient representatives, advocates, and volunteers who could help provide this input.
EHR integration	Despite physician user frustration with EHR usability, vendors are developing patient-centered interfaces, and they know who else is doing similar work and how best to integrate new tools with their software.	Network with EHR vendors to find people whose goals are aligned with yours. Epic’s App Exchange and Developer Guide are a good starting point.
	Other researchers have worked on similar integration challenges. One group integrated a Web service into provider decision support workflow with our EHR vendor. As a result, the vendor has incorporated the ability to access a Web service from within the EHR as a standard for the latest version of their software.	Build off of experiences of others in the field. Programming a Web service that can be accessed from within the EHR can maintain EHR functionality and allow the possibility of being device- and platform agnostic.
	EHR vendors create barriers to research applied to their interface. There are many logistical challenges to completing vendor-specific EHR training to learn skills to do this type of research.	Begin EHR vendor training coursework early, and anticipate delays and challenges to coursework completion.
Feedback from prototype demonstrations to outside expert groups	Tool will achieve its goals only if it can promote conversation between patient and provider.	Include designers on development team so tool facilitates high quality conversation between patient and provider. Consider using psychometrics for shared decision-making—like the OPTION scale and decisional conflict scale.67,68
	It is more difficult to make changes to a computerized prototype if programming is started too early in the development process.	Complete wireframing exercises with end users as an inexpensive and effective method for the early phases of rapid prototyping. Once programming has commenced, programmers must be flexible with evolving specifications as input for the tool is given by key stakeholders and end users.
Challenge of having two users	Best practices for authentication of two users for a single tool on a single device are not established. Double user authentication is particularly difficult given HIPAA regulations.	Work within the institution’s EHR constraints to permit authentication of both users and maintain HIPAA compliance.
	Usability testing methods for two users on a single device is not established—limiting data collection options for both users using conventional usability evaluation methods and software.	Evaluate one user type at a time (e.g., provider) and simulate the other user with a standardized script.
Project management	A high volume of programming specifications can be difficult to track and prioritize.	Use cloud-based project management software to allow asynchronous communication and tracking of specification requests and progress.
Accessibility	Section 508 of the Rehabilitation Act of 1973 in the United States requires all federally funded information technology (IT) to be accessible to people with disabilities.	Use online resources to assess 508 compliance as well as accessibility for those with limited literacy and color blindness.