Table 3.

Themes collected from the “Think Aloud” session categorized into codes.

Code		Themes	Example
1. Usability	Positive	PGx-CDS developed were user-friendly even for users with less knowledge of PGx. CDS provides clear instruction, so a participant was able to formulate decisions without assistance.	“This one [PGx-CDS] is a lot more user friendly.”—MD03. “It seems pretty user-friendly even for someone who is not experienced in pharmacogenomics.”—MD03. “I think it’s really straight-forward and one of the more usable alerts we get.”—MD04.
		The consistency of CDS’s format across alerts allows participants to learn and adapt fast (learnability).	Participants improved time spent on BPA1 compared to BPA2.
	Negative	Acknowledge reasons mechanism was confusing. The system automatically switched to ‘Keep’ current order (from the default being ‘Remove’) if an acknowledged reason was selected. This issue is not PGx-CDS specific.	The participant was confused by the acknowledge reason section. They did not realize you only fill it out when you elect to keep the ordered medication. The participant filled out ‘Other’ for the acknowledge reason section and did not realize that it changed the order to ‘Keep.’ They were confused after selecting ‘Accept’ when the next page was the medication order screen.
		The “More information” button was not intuitive. Participants were looking for a summary of information from reference pages.	“Using my surgery brain, I’m just looking for the high yield colorful pictures.” Participant was confused on how to navigate within the more info link and eventually gave up, seeming not to find it very useful—MD02.
	Suggestions	Alternative medications recommended should be orderable and should link directly from the CDS to avoid the cumbersome work of creating a new order.	“It’s a little bit cumbersome to leave that [alert] screen… [it’s] easier if there were links to orders.”—MD01.
		There should be a link available to the lab results since they can be hard to find in the HER.	“I don’t necessarily know how to find them [PGx lab results]… can be hard to find labs in EPIC® and so link to them would be nice.” Participant would like a link to help explain the labs more—MD03.
2. Visibility	Positive	Many participants mentioned that the pronounced yellow section (“Problem” and “Recommendations”) help catch their attention.	“The yellow box up top-fantastic. Very to the point.” Referring to content and color of BPA one—MD01.
		Using the title ‘Pharmacogenomics Alert’ and the DNA symbol can help make participants aware and distinguish this PGx alert from other types of alerts.	“I like that it says the ‘Pharmacogenomics Alert’ and then the picture of the DNA.”—MD02.
		Participants like PGx-CDS contents to be formatted using bullet points, bolding, and underlining.	“I like the bullets, I like the bold, the underlining.”—MD05.
	Negative	One participant (MD03) thought the color was too harsh.	“I think the color is really harsh. It definitely caught my attention and I think it’s easy to read the text on it. I just wish it was a little bit brighter.”—MD03.
		Directions for asking questions were confusing. Participants did not know what to do with the messaging system.	“If I were reading both of those, I would just call the phone number.”—MD02.
		Reference pages had too much information to make a quick decision.	Participant was confused on how to navigate within the ‘More info’ link and eventually gave up, seeming not to find it very useful. “Using my surgery brain, I’m just looking for the high yield colorful pictures.”—MD02.
		Phenotype presentation was hard to recognize—poor formatting in the lab result session.	“I read things kind [of] quickly and the ‘Poor metabolizer’ doesn’t jump out at me.”—MD06.
		It was not distinctive to recognize whether one or more genomics results were presented and discussed within the PGx-CDS.	“I actually missed them in the first scenario.” Participant was clear that the CYP labs were labs but missed them on the first BPA because there was only one line.—MD05.
		Participants only focused on the information presented in the pronounced yellow section, everything else was skimmed through.	“I am always running late [busy]. I might even not read the non-highlighted area… when I see this, I would just switch the prescription.”—MD09.
	Suggestions	Lab results presented in CDS were hard to recognize. Participants suggest using icons/symbols such as a test tube/beaker to help with recognition.	“[Some color to that area… some kind of test tube… or picture, something that shows that it’s a lab.” Participant would like to see some color or picture (gave an example of a test tube) added to the CYP labs area—MD01.
		Phenotype information such as ‘poor metabolizer’ should be bolded (or red) to grab attention.	Participant felt that the ‘poor metabolizer’ should be bolded as well, “[like] the ‘increased risk of adverse events [is bolded]”—MD06.
		Participants preferred to see lab results sooner, at the top of CDS, or they should be presented in the yellow box.	“[A]s opposed to the other one [BPA1]… it was harder to see this… ultra-rapid… I would want that more up high [at the top, in the pronounced yellow section of the alert].” Participant preferred to see the metabolizer status at the top, in the pronounced yellow section of the alert—MD06.
3. Workflow	Positive	A hard stop in the workflow is necessary for prescribers to read the contents of the PGx-CDS. Prescribers emphasized that they wouldn’t be able to remember all patient data and keep track of all “flags” for each patient.	“I like the fact that it clearly grabs your attention as soon as you try to order the medication… and there is an actual stop in the workflow.”—MD03. “Honestly, we get so many of these flags that we don’t really keep track of what’s been flagged on each patient.”—MD04.
	Negative	N/A.	N/A.
	Suggestions	For PGx related orders, participants prefer to see the PGx alerts repeat, if relevant. However, one participant (MD06) would like the alert to fire once for each gene–drug pair. That participant is concerned for alert fatigue if the alert goes off every time for the same gene–drug pair.	“I think it would be helpful to see [the BPA] again… there’s a huge chance of having different people ordering medications.” The participant would like the BPA to appear every time medication connected with BPA is ordered because different prescribers may be ordering the medication.—MD01. “I’d be worried about getting fatigued… so I think anytime I add a new drug I think it is appropriate if it applies but dosing changes, I don’t think I would want them to pop up every time.”—MD06.
		Provide a link to the PGx results in the results review section in the EHR as some participants are not familiar with where they are located.	“I don’t necessarily know how to find [PGx results].” The participant would like a link to help explain the labs more.—MD03.
		One participant is interested in an option to document changes they made based on CDS in the EHR.	“I want [this] information to retain in EPIC® . It gives me an opportunity to document my change… I like to document for myself from PGx result.” The participant asked where the information goes after they put in the medication order.—MD10.
4. Content	Positive	Alternative recommendations were useful when participants were not familiar with the different medication options.	“I really liked [BPA2] because it gave me multiple options.” Participant is referring to the recommendations section on BPA2, providing examples of medications they may select for each medication class option.—MD04.
		CDS problem statement directly told participants the risk for the gene–drug pair to know what to monitor.	“It actually tells you what the risk is, so that way you would know what to monitor the patient for if you were going to proceed in prescribing the medication.”—MD03.
		Providing phone numbers was helpful to get more information if needed.	“I like the fact that there is also a phone number there that I can call if I need to ask questions or get more information.”—MD03.
		Acknowledge Reason section is helpful and alerts the user that a change is needed.	“I like the fact that the ‘Acknowledge Reason’ included that I ‘Will adjust the dose’ as just another reminder that if I’m going to keep it that I need to do something different with it.”—MD03.
	Negative	Participants (MD02 and MD03) did not like ‘Disagree with recommendation’ and thought it should not be an option.	“I’d be curious to know who would select ‘Disagree with recommendation’.”—MD02. Interpreted ‘Disagree with recommendation’ as something to use for when there are medication shortages of other drugs. Participant didn’t think it would be a common choice.—MD03.
		Genotype information might not be useful.	“I don’t think the genotype is useful for me whatsoever, I’m not gonna know what the *3 *4 genotype actually means”—MD07.
		Providing genotype information is not helpful to making clinical decisions, and a shorthand description of ‘CYP2D6PHENO’ is confusing.	“I think it is confusing, the ‘CYP2D6PHENO’, I would change it to ‘PGx testing date’ and the result = ultra-metabolizer I think that would be more clear”—MD09.
		Information provided in the link is not useful to help describe the gene–drug pair.	“SSRIs metabolized by CYP enzymes, more information on physiology, possible interpretation of phenotypes to genotypes.” The participant expected this information in the ‘more information’ link—MD07.
		Adverse events listed due to the patient’s genotype are not specific enough, and giving examples would help explain the gene–drug pair.	When the participant was asked if listing the adverse events would have helped in his decision-making: ”I would want to know what to look out for as a reminder from medical school because that’s the last time I would have gone through the side effects.” Adverse effects were not listed in the second alert, unlike in the first alert.—MD06.
		Adverse events due to the patient’s genotype are too specific.	“Adverse reaction is really enough for me to change my prescription, whether it’s death or something else is not relevant.” The participant prefers a broader ‘adverse events may occur’ comment instead of a specific list of adverse events.—MD09.
	Suggestions	Keep genotype and phenotype lab results in CDS.	“I want to know the actual genetic phenotype and genotype.”—MD10.
		Acknowledge reasons should be more specific (for example, include an option to state ‘dose change’ if CDS recommends adjusting dose).	“If a lower dose would be considered safe, I would have that option [will adjust dose]… depending on the situation I guess, sometimes you absolutely cannot prescribe it.” The participant seems to like the ‘I will adjust dose’ option—MD09.
		Participant suggested adding a button to request a consult from the PGx team from CDS interface.	“One thing that would be great here, essentially this PMP monitoring, I know you can request a consult but this doesn’t tell you can request a consult if you’re not sure what the best option is.”—MD10.
5. Understandability	Positive	‘Problem statement’ and ‘Recommendations’ language in the alert is clear and easy to understand.	“What I get from that is… she does not metabolize it as fast or basically different from other people, so she [is] basically more sensitive.” The participant is referring to BPA wording.—MD02. “The [pronounced] yellow box [‘Problem statement’ and ‘Recommendations’ included here] gave me enough information for me to know what I should act upon, so I didn’t think I needed to click on that [more information] link.”—MD07. The participant states that information regarding the flag is clear and easy to understand.—MD04.
		Acknowledgment language was easy to understand.	“‘Will adjust dose’ makes sense because you can decrease it by 50%.” Participant thinks that options for ‘Acknowledge Reason’ makes sense, particularly with ‘Will adjust dose’ for BPA2 due to option to adjust dose in recommendation.—MD05.
	Negative	Difficulty interpreting and understanding the PGx labs (mainly by providers who have not seen PGx before).	“I didn’t realize necessarily that they were [CYP] labs. I’m not familiar with these phenotypes and genotypes.”—MD01. “She [the fictitious patient] is a *3/*4. I’m not totally sure what that means.”—MD02. “It’s very subtle… ultra-rapid metabolizer… I don’t know if that’s a severe or significant diagnosis vs. positive or negative”—MD02.
		‘P RX UF PMP MONITORING’ message is unclear. Participants do not understand who receives these messages and what the next steps are in the process. Participants are confused with the messaging system in EPIC® (i.e., haiku, canto).	“If I were reading both of those [messages], I would just call the phone number.” The participant stated that the message “P RX UF PMP MONITORING” was confusing. The participant did not know what it meant or how to use it, and stated he would just call the phone number if he had a question.—MD02. “I have not used the EPIC’s® messaging… it is not clear how to use it.” The participant stated they would just call the number if they had questions—MD09.
		Trouble understanding ‘Advisory sent via in basket message’.	The participant did not know what ‘This advisory has been sent via in basket’ meant—MD02. The participant stated that it does not say who it gets to or what happens when you message them—MD10.
		Participant not sure what “pheno” means, referring to the phenotype under lab results in the alert.	“What is PHENO”—MD09.
		Gene–drug pairs that have more than one genotype/phenotype are difficult to understand.	“CYP2D6? I thought we were talking about CYP2C19… so, the patient has both problems?” The participant was confused seeing CYP2D6 information when the alert is about CYP2C19. The participant stated that the alert ‘Problem statement’ is not clear regarding information with 2D6: “I don’t know the different between genotype and phenotype… I mean I know what they mean… how is the genetic testing different? I thought you just get a result.”—MD09.
	Suggestions	Make the metabolizer status association with the gene–drug pair clearer.	“I would want both of those to be… would want to say this person is a CYP2C19 poor metabolizer and CYP2D6 intermediate metabolizer… poor metabolizer status is associated with x, intermediate metabolizer status, right now CPIC doesn’t say anything about it… more likely to have adverse side effects down the line so that would also impact these choices”—MD10.
		Recommend removing genotype information (due to it being confusing and not understanding the value of having it in the alert).	I honestly again skimmed past it… the problem part in the very top was what I focused on… I don’t think the genotype is useful for me whatsoever, I’m not going to know what the *3 *4 genotype actually means.”—MD07. Phenotype makes sense to the participant, but genotype not as much. The participant stated that it is helpful to have it there for people who know more about the topic, such as about PM/IM/UM etc. and it needs to be in the pronounced yellow section at the top.—MD10.
6. Medical usefulness	Positive	BPAs were helpful at reminding the prescribers about PGx considerations such as potential adverse events.	The participant stated the following regarding the overall BPA: “the information is fantastic… I think really has the potential to cut down on adverse effects.”—MD01.
		Recommendations provided in the alert helped to make a clinical decision.	“It [the BPA] definitely made me think. It made me go back and change my order.”—MD01 “This message has me thinking as to what I should do [entire BPA]”—MD02. “We talk about pharmacogenomics and things a lot. We don’t take it into account all the time when prescribing, so I think personally this was a very useful message for me to come across.”—MD07.
		Alerts should fire off every time to help improve prescribing of PGx medications.	When participant was asked if they would like to see the alert every time the BPA can be applied: “Every time, because people forget.”—MD08.
	Negative	Not listing the adverse effects makes it difficult to determine the clinical importance.	“It [the BPA] tells you there’s just adverse effects, but it doesn’t really say what [those effects are]… I think it would be better if they can add on the actual adverse effects.”—MD08. “I would want to know what to look out for as a reminder from medical school because that’s the last time I would have gone through the side effects.” The participant stated this referencing those potential adverse effects that were not listed in BPA2, unlike BPA1.—MD06.
	Suggestions	Recommending alternatives for opioids would be more useful for decision making.	The participant states that an order set link would be helpful—MD05.
		Adding brand and generic would be more helpful for recognizing medications.	The participant thinks MDs would be more likely to choose an alternative they recognize (referencing the need for brand and generic to be listed)—MD05. “Like Dilaudid instead of Hydromorphone… people might not be as familiar with the non-brand name.” The participant recommends to consider putting the more common name used in practice for the medications or brand plus generic.—MD05.
7. Navigation	Positive	BPAs specific to PGx were not different compared to other BPAs that providers usually see, making the BPAs easy to navigate.	The participant stated: “that’s standard for how I currently use EPIC® .” referring to not having issues with the remove and keep section—MD07.
	Negative	Order of how acknowledge reasons generate is confusing and hard to follow.	“I didn’t have too many thoughts on it because I decided to seek an alternative treatment… if I had decided to keep the medication, I probably would have looked more closely at it.” The participant ignored the ‘Acknowledge reason’ section because they chose to remove the order.—MD03.
		Leaving a comment after choosing your ‘Acknowledge’ reason is confusing and participants are unsure of what information is needed.	“So, I am not really sure what comment they want me to enter.”—MD03.
		Scrolling is required to see select alerts, which may lead to confusion on how to proceed.	“I just didn’t scroll down far enough to trigger the accept button to light up so that was the only thing I didn’t know why I would need to write something in addition to selecting the ‘Will adjust dose’.”—MD06.
	Suggestions	Acknowledge section needs to be more flexible and responses to the action taken by users.	“Maybe hide the ‘Acknowledge Reason’ unless you select ‘Keep’…because you don’t really need to acknowledge it if you’re removing and changing.”—MD04. “I intuitively felt like I need to acknowledge it with a button… The only button that I was looking for was like a ‘recommendation acknowledged- will remove order’.”—MD02.