Table 3.

Evaluation for quality and impact outcomes quotations.

Frames	Quotations
Alpha or usability testing	“So I think, yeah I would tell people about this game or I would play this with my friends or, I believe this game is as good as other games that are related to health or how does this game compare to the games you play? Is it better or worse? More fun less fun? These are questions that we typically ask in usability but you can ask those type of questions. One of the questions I really like is 'how could we make this game better?' And the reason I like it is that it is positively oriented, its um, it’s got an altruistic style to it and it really find out what sucks in the game without them having to say it, so it’s really acknowledging the short fall of the game. So, it’s like they can say I would really change up this and blah, instead of them having to endure the sort of social desirability problem of telling you this thing sucks”
	“Usability is giving it your user, a new player and asking the same questions. You can do usability by doing that. You could do it with a survey. You can also do talk aloud usability test. You get them to play it and tell you what is going through their mind so um well, I’m going to pick the pink button because they think that you know that’s going to lead to a better result, I’m going to choose the middle door on my quest you know, why, I’m going to choose this avatar, why? We did our Asthma simulation, we found that the young males all gravitated to this older Hispanic woman who had this lighter color hair because she was just perceived as more attractive. So, you learn a bit about how you are presenting everything with that approach. So, then you may tweak it or you may think it’s ready for a trial and you can apply it. You can get enough samples, that would give you the power to check the outcome and you can write a pre-post that randomize the laboratory conditions”
	Let’s say it’s a simple card game and so now we have all these cards and all these rules, we sort of think how are we going to play it and we play it amongst ourselves and we play it till we thin kits cool and we thin kits a good one, so let’s now take it to some of our target population and see it too and check their responses. We haven’t programmed something yet, we are still working with the concept and the rules versus a table game. So that is proof of the test of the concept and it’s an acceptability test and a usability test. But, what we’re asking, what we have the opportunity to look at is, questions that are looking for evidence like, ease of use, ease of play, um, understandability, acceptability, of this a s a game, um, credibility, I think the information here is trustworthy. The perceived value um so I think the information I am getting here will help me make better decisions in the future.”
Data collection measures	“Um, so I think that part of it is super helpful, I mean, I like love self-measurement projects where its like already built in um, I mean I’m so, but, here’s something, okay. It’s actually something we are doing with google right now. It’s really cool. Um, we have all these digital ads that are like measuring behavior physicians. So we know predominantly what websites these physicians frequent, you have our ads on there, they are exposed to various kind of ads, banner ads, PSAs, maybe a 6 second pre-roll that they can continue watching or mandatory 30 second yadda yadda. And we’re actually able to use IP address chasing to determine if they have taken an action after seeing this. And um, those who have watched our 30 second video are 300 times more likely to then search around HPV vaccines and vaccinations”
	“I don’t know how technology works but I think there are ways to do that relatively easily. Um you could say on some level just, you know one measure of success is that engaging and do they play it. And the other is does it actually have the desired effects. Maybe it has undesired effects like you wouldn’t want to measure but behavior changes are often what the ultimate end goal but sometimes were harder to evaluate and practice”
	“if there’s a point in the game on where to get the vaccine you can track to see how many people get to that point. If they get to the location maybe there’s something on that end that they could verify, sort of prize or coupon or something like that. Obviously if you build in social media then you can track those analytics as well. I think you’d have a lot of data now. You can quantify that to success”
	“It just depending on what you want your audience to do. Do you want people to remember messages? So maybe you could build in a measure for recall. Send them an email in a month or something and give them a short quiz and see if they remember that stuff.”
	“Pre and post the game playing. You could do behavioral stuff and then even long-term behavioral of your same cohort”
Gaming to vaccine correlation	“And I would think, that doing the same kind of measurement, that this would have to be a planned evaluation project from the outset where we are going to enroll 200 young men or whatever the number is and we are going to have them play this game, and we are going to naturally allow them to accept or not accept the HPV vaccine, we’re going to find out what their rates were before and after that. We are going to pick HPV vaccine naïve men, so that we know how many get vaccinated or there could easily be a survey component in it too, because I think it’s more than just one person being vaccinated”
	“Like...this is not the best evaluation but like pre-your game what percent of the students or whatever vaccinated and then after. Did you change it? You could even do like demand at the clinic”
	“So, if you’re going to do this on college campus, maybe look at the pre vaccinations levels the university implements freshman orientation than game and then you look at the end of the year you look at the vaccination levels. Or the game can be more specific, questions for user have you gotten this pre vaccination? Maybe you can throw away a pop up that they can click on to get the vaccine”