Prototyping for context: exploring stakeholder feedback based on prototype type, stakeholder group and question type

Abstract

Engineering designers frequently use prototypes to gather input from stakeholders. Design guidelines recommend the use of quick and simple prototypes early and often in a design process. However, the type and quality of a prototype can influence how stakeholders perceive a new design concept and can therefore impact their responses. Additionally, different levels of experience, expertise, and preparedness for providing input to designers may lead stakeholders from different geographical or cultural settings to provide different responses, making the format of a prototype even more influential. Although design practitioners are known to intentionally align their prototyping approach with the specific design question to be answered, it is unclear the extent to which prototyping approaches should vary based on the stakeholders, context, and setting of a design project. To investigate how the format and quality of prototypes influence stakeholders’ responses, we conducted a field study with various medical professionals in Ghana. We presented prototypes for a medical device in different formats to stakeholders and collected responses to the design through semi-structured interviews. We found that professional expertise, prototype format, and question type influenced the types of responses that stakeholders provided. These findings suggest that designers seeking input from stakeholders on new concepts should consider context-specific prototyping strategies, especially when designing at distance and across cultures.

1. Motivation

Various factors, including the visual appearance of a prototype, can influence how individuals perceive the objects or ideas to which they are introduced. Even though prototypes may not always reflect the actual quality or utility, they are critical factors that can impact the perception of a product and possibly motivate a decision to purchase or use a product (Desmet & Hekkert, 2007; Sauer & Sonderegger, 2009). In design, when new ideas are shared with stakeholders – the individuals, groups or organizations who have direct or indirect interest in the product – through prototypes, several factors contribute to how these new ideas are perceived. Prototypes serve as vehicles for designers to communicate their thoughts to others, but the nature and level of refinement of a prototype often depends on the stage of a project (Atman et al., 2007; Christie et al., 2012; Menold, Jablokow, & Simpson, 2017). Prototypes used in the early stages of a design process might include conceptual sketches and crude mockups, while prototypes used during later stages might consist of more refined models with virtually indistinguishable properties from the production part (Crismond & Adams, 2012; Hilton, Linsey, & Goodman, 2015).

Stakeholders are important contributors to a design process, and designers need to consider the context of their stakeholders as well as the product itself. Consideration of a stakeholder’s background in design extends to the elicitation of feedback from stakeholders on prototypes. Stakeholders can vary in their backgrounds and experiences, which can influence the type and depth of design input they provide on prototypes. For example, some stakeholders may be focused on appearance, while others may be more focused on the underlying idea. As a result, promising design concepts might be overlooked by some stakeholders because of a less favorable presentation, while other, less-promising concepts might elicit false-positive responses motivated by a more refined form of presentation (Hekkert, Snelders, & Wieringen, 2003; Leder & Carbon, 2005). The notion that stakeholders should therefore be presented only with highly-refined prototypes is challenged by findings that prototypes that can be perceived as finished products might convey the impression that input is no longer needed or even possible, since a great amount of time and energy has already been invested in the design (Viswanathan & Linsey, 2011). Furthermore, products for global markets are typically designed at distance, and geographic, time and cultural differences may pose additional challenges to designers (Scrivener, Harris, Clark, Rockoff, & Smyth, 1993).

While factors like project setting, stakeholder level of experience, motivation and investment in the project might be difficult for designers to influence, they can exercise control over the type and quality of the prototype they share, as well as the questions they ask of those from whom they seek input. It is therefore critical that designers identify the presentation format and question type most appropriate for stakeholder interaction. In this study we investigated if and how stakeholder input on a medical device concept was influenced by type of prototype, type of question, and group membership.

2. Background

Timing and Fidelity of Prototypes in a Design Process

Engineering designers often use prototypes as tools for testing and validation. However, multiple studies have shown that prototypes can be useful throughout an entire design process (De Beer, Campbell, Truscott, Barnard, & Booysen, 2009; Moe, Jensen, & Wood, 2004; Viswanathan & Linsey, 2009; Yang & Epstein, 2005). For example, while functional prototypes like 3D-printed models and CAD models are frequently used for functional testing later in the process (Baxter, 1995), they can also be useful during front end design, the phases of design most commonly associated with problem identification and definition and concept development (Camburn et al., 2013; Christie et al., 2012; Kelley, 2001). Even though designers might use simple prototypes such as sketches and mockups primarily early in the design process to quickly inspire, communicate, elicit input and select from new ideas (Brandt, 2007; Campbell et al., 2007; Gerber, 2009; Houde & Hill, 1997; Kelley, 2007), they too can be helpful later in the process.

Design experts frequently call for a minimalistic, or “quick and simple,” approach to prototyping, constructing the quickest and cheapest prototype that still satisfies a particular requirement, e.g., the communication of an idea (Kelley, 2001; Moogk, 2012). Low-fidelity prototypes like sketches and cardboard models are often intentionally simple, incomplete, and sometimes crude representations that convey some critical characteristics of the intended end product. They can be created quickly and inexpensively and allow designers to share and evaluate a large number of ideas. This quick and simple approach enables iteration and decision making early in the design process and the selection of the most promising ideas before much “sunk cost,” i.e., time and money, is invested (Arkes & Blumer, 1985). Higher fidelity prototypes like 3D-printed models and CAD models that require additional resources such as time, skills and money to create, are typically reserved for later stages in the design process, when functional and/or simulated testing is necessary (Dieter & Schmidt, 2012; Rudd, Stern, & Isensee, 1996).

While collecting useful input from stakeholders can be challenging for designers (Castillo, Diehl, & Brezet, 2012; Mohedas, Daly, & Sienko, 2015), prototypes can serve to facilitate these interactions. For example, prototypes are increasingly being used during the earliest stages of a design process to support the elicitation of product requirements from stakeholders (Kelley, 2007; Schrage, 1999; Yock et al., 2015). Here, representations of diverse preliminary concept solutions might be presented to stakeholders to facilitate and support requirements elicitation interviews. Demonstrating ideas to stakeholders through the use of prototypes is preferable to providing a verbal description alone and is especially critical early in a project when designers are developing an understanding of stakeholder needs and wants – often across professional and geographical cultures (Jensen, Elverum, & Steinert, 2017; Kelley & Littman, 2006; Scott, 2008). In these situations, prototypes can serve as shared objects that support communication, engage stakeholders in the design process, allow them to better express their opinions and define requirements that designers might not otherwise discover or that can be difficult to synthesize. The new insight into the problem as well as the solution space of a design project often introduces elements of surprise, and new and unexpected circumstances can lead to problem-solution co-evolution that often inspires creative solutions (Dorst & Cross, 2001).

In a study in sub-Saharan Africa, Sabet-Sarvestani and Sienko (2014) noted that the quantity and quality of responses to requirements elicitation interview questions dramatically increased when the team presented physical and functional prototypes to stakeholders compared to theoretically-grounded elicitation questions that minimized bias by prompting stakeholders to provide responses about a hypothetical concept solution. Earlier conversations with stakeholders had not provided critical insight into the cultural viewpoints and concerns about an adult male circumcision device, but when the research team introduced physical prototypes, participants started to interact with the models, compared concepts, discussed differences and provided input about both the concepts and culturally relevant information that would affect implementation if not fully captured in the product requirements. This degree of insight could not have been gathered through interviews alone; it only transpired through discussions and observations supported by stakeholders’ interactions with physical prototypes.

Prototypes can also be invaluable tools for exploring design details and identifying potential issues early in a design process (Jensen et al., 2017). Often, the level of refinement, detail, and functionality of a prototype increases as designers develop a deeper understanding about the solution space and build on what they learned from earlier iterations (Ulrich & Eppinger, 2015; Yang & Epstein, 2005). Consequently, early prototypes do not always represent the quality and functionality of the intended end product, and stakeholders’ perceptions of a new idea might potentially be negatively influenced by the nature and level of refinement of the prototype with which they are presented (Crilly, Moultrie, & Clarkson, 2004; Hare, Gill, Loudon, & Lewis, 2013; Lim, Youn-kyung, Pangam, Subashini Periyasami, & Shweta Aneja, 2006).

Simply increasing presentation quality and functionality of a prototype, however, does not automatically lead to better input from stakeholders. Recent studies in the field of human-computer interaction concluded that a balance between quality and functionality of prototypes might be most beneficial for the collection of input from stakeholders (Hare et al., 2013; Lim, Youn-kyung et al., 2006). Further, the authors emphasized that the context surrounding the prototype feedback session, such as task scenarios, social and physical circumstances, as well as the participants themselves, can influence the type and quality of stakeholder input. For example, in a study by Sauer & Sonderegger (2009) examining the influence of prototype fidelity on user behavior, participants were presented with low, medium, and high quality prototypes of cell phones and asked to perform tasks like sending a text message and suppressing a phone number. The researchers found that the more attractive prototypes positively affected user emotions and consequently their judgment of usability of a concept. In a human-computer interface (HCI) study with simulated automatic teller machines (ATMs), perceived usability was strongly related to the perceived beauty of a design -- the more beautiful participants rated a layout, the more usable they thought it was (Tractinsky, 2000). In another study, participants judged the creativity of ideas for new toaster concepts represented by sketches (Kudrowitz, 2012). The concepts represented by the highest quality sketches were most likely to be ranked as the most creative ideas.

Variation in Stakeholder Feedback in response to Prototypes

Scholars in fields that leverage representations, such as the sciences, describe a variety of roles that representations can have in supporting processes and outcomes within the discipline. For example, models in science have been used for a variety of purposes including visualizing, forming hypotheses, critiquing ideas, examining theories, and deriving relationships (Daly & Bryan, 2010; Giere, 2004; Morgan & Morrison, 1999; Seidewitz, 2003). Similarly, in the design domain, many design professionals use prototypes in a variety of ways, and follow common best practice recommendations for using them in ways supportive of design decision-making (Lauff, Kotys-Schwartz, & Rentschler, 2017). However, it is unclear the extent to which these commonly accepted best practices are directly transferable across contexts, cultures, stakeholder characteristics or environments of a design project.

Stakeholders’ cultural norms may direct their focus on particular aspects of a prototype. In a study evaluating cultural differences of consumer purchasing behavior, stakeholders from one cultural group (Singapore) focused more on the functionality of the product, while stakeholders from another cultural group (Philippines) valued aesthetics more when making purchasing decisions (Seva & Helander, 2009).

Variation in experiences can also play a role in a stakeholder’s ability to give feedback. In the art domain, naive reviewers exhibit a tendency to stereotype based on personal taste (Parsons, 1989), and while novices in any field tend to have more emotional reactions, experts tend toward cognitive responses that lead to a more analytical way of reviewing an unfamiliar object (Winston & Cupchik, 1992). In studies of novices and experts in physics, Chi, Feltovich, and Glaser (1981) found that the understanding of examples differed based on expertise. For example, novices grouped physics problems together because they included “ramps,” while experts defined a category as “work problems.” This finding illustrates that differing levels of experience and expertise in a domain results in differences in how new examples are perceived. Translated to the design domain, when a stakeholder does not have experience or competency in a specific domain, he/she may not know how and what to look for. For those with less experience, being asked for their feedback on design may feel overwhelming, which can lead to frustration and put a stakeholder in a negative affective state about how they feel toward the object in question (Frijda, 1989). This negative emotional response then influences the aesthetic experience of how a reviewer processes and evaluates new information (Scherer, 2003).

Stakeholders also have a variety of motivations and experiences not related to their expertise in a particular subject matter, which can impact how they respond to a prototype (Chamorro-Koc, Popovic, & Emmison, 2009). Different emphases in feedback given by stakeholders may be related to different interpretations of the affordances of a product as represented by the prototype. Affordances are properties of an object that suggest possible interactions of the user with the object. For example, the shape of a lever may imply that it should be pushed rather than pulled, or the shape of a knob might suggest turning rather than moving (Gibson, 1966). As affordances are perceivable actions, or actions that are considered possible by the user, affordances that users identify depend on prior experiences and knowledge (Norman, 1990). Thus, stakeholder feedback may vary based on the affordances they interpret from the presented prototype, and the form and functionality represented in that prototype.

Finally, the questions asked of stakeholders can prompt variation in the types of feedback they provide (Creswell, 2013; Patton, 2014; Weiss, 1995). Specifically, questions that are positioned outside of a participant’s expertise can negatively influence their response (Leder, Belke, Oeberst, & Augustin, 2004). Therefore, interview questions need to be carefully designed to extract unbiased information from stakeholders and should consider cultural context (Glesne & Peshkin, 1991). Questions should aim to understand the underlying theory of a response and allow an interviewer the flexibility to adjust and avoid negative experiences such as boredom, annoyance, or even physical discomfort by the participants (Silverman, 2010). Semi-structured interview questions can provide a framework that guides a conversation while allowing for flexibility on both the interviewer and interviewee to explore and expand on interesting information (Patton, 2014).

3. Methods

Our study focused on one product category (medical devices) and multiple stakeholder groups (nurses, medical students and medical doctors) in one cultural context (Ghana) to provide initial insights into how prototype type, group membership (stakeholder characteristics) and question type can influence stakeholders’ perceptions of a design concept and the resulting feedback they provide. The research questions that guided our work included:

• ○How does prototype type influence stakeholder feedback?
• ○How does group membership influence stakeholder feedback?
• ○How does question type influence stakeholder feedback?

The medical device product category was pursued for this study because gaps exist between designing products that address the need for global health technologies and how these technologies are implemented in a new setting (Free, 2004; Howitt et al., 2012; World Health Organization, 2010). The study was performed in Ghana because of existing partnerships with multiple tertiary healthcare facilities and the fact that English is one of Ghana’s official languages.

3.1. Participants

Forty-five healthcare professionals from a teaching hospital in Ghana were recruited for participation in this study. They included 18 nurses or midwives, 10 medical students training to become medical doctors in obstetrics and gynecology, and 17 medical doctors. These participants represented a cross-section of the target stakeholder groups, are likely the most easily accessible respondents to design teams working in similar settings, and would either be using, advising, or training others in the use of the proposed device. The participants were recruited by the family planning department of the hospital and received a small gift for their participation (pen, mini-flashlight or USB memory stick). All participants were aware of long-term contraceptive implants, but none were familiar with the assistive insertion device or had seen it before.

3.2. Data Collection

We introduced participants who were stakeholders in this unique cultural context, Ghana, to the design of a medical device concept that assists with the insertion of a long-term contraceptive implant. Long-term contraceptive implants are particularly appealing in resource-limited settings where patients have limited access to healthcare providers (Funk et al., 2005). A small polymer rod is implanted into the subcutaneous tissue on the inside of the upper arm of the patient. Properly inserted, the rod releases hormones into the woman’s blood stream and, in contrast to oral contraceptives, does not require regular visits and monitoring by an obstetrician-gynecologist. The implants provide contraception for extended time periods, between three to five years, depending on the manufacturer. However, if not inserted properly, the rod can become embedded in the muscle tissue, restricting efficiency and complicating removal, sometimes even requiring a surgical procedure. Proper insertion is therefore critical and is typically performed by trained healthcare professionals like doctors and nurses. The proposed concept represents a task-shifting device (McPake & Mensah, 2008) that acts as a needle guide (Mohedas, Sarvestani, Daly, & Sienko, 2015). It allows lesser-trained healthcare providers like community health workers (CHWs) to perform correct insertions in rural areas with limited access to healthcare. This simple, low-cost device was first conceived by mechanical engineering students during a capstone design course and is representative of projects in which designers might seek input from a variety of stakeholders, from government officials to rural healthcare workers.

The device concept was presented through various prototypes that are commonly used during design. The four representations consisted of prototypes that designers commonly create during a design project and included a sketch, a cardboard mockup, an animated (rotating) CAD model and a 3D-printed, production-like representation of the device. The sketch and the CAD model were virtual, i.e. non-physical, representations that were shown either in low-fidelity, paper form (sketch) or on a laptop screen (high-fidelity animated CAD model). The cardboard mockup (low-fidelity) and the 3D-printed model (high-fidelity) were physical or tangible objects that were given to the participants for examination. The prototypes are shown in Figure 2.

Fig. 2.

The types of prototypes used in this study: a) Paper sketch, b) Cardboard mockup, c) CAD model, d) 3D-printed model

Each participant was first shown one prototype – either a low-fidelity prototype (sketch or cardboard mockup) or a high-fidelity prototype (CAD or 3D-printed model), and then asked them a series of questions to elicit feedback. These questions were developed to be ones that designers would likely ask when gathering input from stakeholders (Kelley & Littman, 2006) The questions that prompted participants to comment on several aspects of the device design and afforded the interviewer the opportunity to ask follow-up questions when clarification was needed. Nine questions were asked, designed to elicit participants’ impressions of the device and to encourage them to critique or add to the proposed design they reviewed. A full list of questions can be found in Appendix A. Prior to collecting the data, pilot interviews were conducted at a large, Midwestern university in the United States to test and refine the questions and the prototypes shown.

After the first prototype was shown and questions asked, each participant was then shown a second prototype of the same device but from another fidelity group and asked the same nine questions again. Introducing both low- and high-fidelity prototypes to the participants helped to minimize answer biases caused by the nature of the prototypes. The order and type of prototype participants saw were randomly assigned.

All data were conducted during a one-week period and were carried out by the same researcher in English. All interviews except one were audio recorded and later transcribed for analysis. One participant did not agree to the use of an audio recorder and handwritten notes of the interview were taken instead.

3.4. Data Analysis

After the interview data were collected, the audio files were transcribed for analysis. Three analytical methods were used to determine the usefulness of the answers that participants provided. These included 1) a deductive coding scheme developed by the research team to categorize the type of input elicited, 2) a modified version of the consensual assessment technique (CAT) to capture the quality of the input provided by the individual (Baer & McKool, 2009; Kaufman, Baer, Cole, & Sexton, 2008), and 3) a count of the number of words in participants’ responses to each question.

3.4.1. Metrics

Deductive Coding of Response Type

The research team developed a coding rubric to categorize responses. This rubric consisted of four answer categories as shown in Table 1. Category A answers included design input as part of the response. Here participants provided design input by suggesting for instance that a change in color, size or material should be considered. Category B answers consisted of the participants’ opinion backed by justification as to why they felt a certain way. Category C answers comprised unjustified answers that represented the opinion of the respondent only. Category D answers were non-useful and included statements by participants referring back to the design team’s expertise rather than giving their own opinion.

Table 1.

Rating Rubric for deductive coding of response type

Category	Code	Definition	Example
A	Answer with design input	Provides input / suggestions beyond just answering the question	“So maybe it should be designed in [different] sizes”
B	Justified answer	Answers and explains why	“I like the ability of the device to isolate the skin and then the subcutaneous tissue from the muscle”
C	Unjustified answer	Answers affirmatively but provides no explanation	“Yes” or “No”
D	Non-useful answer	Provides no answer, is unsure or answer was contradicting / made no sense	“I can’t say” or “if you get it right then it will work” or “you’re the designer”

Two researchers completed multiple rounds of coding and, once the individual answers were assigned a category, the results were normalized by adjusting the counts to a common scale to account for the different numbers of entries in each group. In a few cases (seven), a single interview question was not asked, and the missing data were removed, resulting in 99% valid answers across all interview questions. The inter-rater reliability for this coding activity calculated with Cronbach’s alpha was 0.943 and is considered substantial agreement (Landis & Koch, 1977). The power of the analysis for detecting a medium effect of w = 0.3 as defined by Cohen’s effect size index (Cohen, 1977) was 80%, 53% and 78% for nurses, doctors and students, 59%, 58%, 58% and 60% for sketch, cardboard mockup, CAD model and 3D-printed model respectively.

Consensual Assessment Technique to Assess Usefulness of Response

Another perspective on evaluating stakeholder input is to engage subject matter experts in an assessment procedure called the Consensual Assessment Technique (CAT) (Amabile, 1983; Amabile, Conti, Coon, Lazenby, & Herron, 1996; Baer, Kaufman, & Gentile, 2004; Howard, Culley, & Dekoninck, 2008). This technique, commonly used to rate creative products like paintings and poems, draws on a large number of experts who are presented with multiple artifacts. The experts are asked to rate the artifacts relative to one another on a Likert scale (Kaufman, Baer, Cole, & Sexton, 2008) based on a common criterion like creativity, composition, or use of color. Contrary to the deductive coding approach, the raters are not provided with detailed instructions. Instead, each rater develops their own justification for why they think one artifact should be rated higher, lower, or the same as another. Research has shown that even with the lack of detailed instructions, or a request to justify their decisions, a large degree of agreement can commonly be found among subject matter experts (Amabile, 1983). According to Landis and Koch (1977), a reliability between 0.61 and 0.80 is substantial, while agreements above 0.80 are considered almost perfect. Ideally, a large group of experts (for example 30) would judge a small sample of work (maybe two items). However, it is often not practical, and as a result, a small number of experts are often recruited to evaluate a large body of work (Kaufman et al., 2008).

In order to see if this technique would provide consistent results, we recruited five subject matter experts (designers with several years of experience in product design and/or medical device development) for this activity and asked them to rate the usefulness of the input participants provided. A key aspect of the Consensual Assessment Technique is that the raters define their own criteria (Amabile, 1983) and therefore the term “usefulness” was not defined beyond asking reviewers to rate how useful they thought an answer was for the design of the device. The answers to the individual questions from the first round of the interviews were printed and given to the reviewers in four sets to allow them to physically order the responses.

The four sets of data given to the raters consisted of:

• -Individual responses to question 3
• -Individual responses to question 6
• -Individual responses to question 9
• -Individual responses to all questions, i.e., the entire transcript of the first round of interviews

Due to the significant amount of time required to complete this activity, the experts were only asked to rate answers for three individual questions (see 3.4.4 Questions) and all nine answers combined from the first round of interviews. The experts were instructed to rate all 45 answers for each of the four sets on a 1 – 5 Likert scale on how useful they thought the answers were for improving the design (with 1 being the least useful and 5 being the most useful). Participants were asked to utilize the full scale (1 – 5) and perform their rating for all four sets of data independently. The raters performed three rounds of ordering for each set to ensure that they were satisfied with their final selection. Raters reported needing between 8 and 10 hours to complete the rating activities. Once the experts rated the data, Cronbach’s Alpha was calculated to measure consistency among the five experts. In this study, the agreement was 0.914 across all interview questions and 0.957 for questions 3, 6 and 9, representing significant agreement for all four data sets.

Word Count

In addition to the previous techniques, a word count analysis was conducted to determine if the volume of words contained in an answer provided any indication of value, here defined as usefulness, of the responses. If so, this technique would require the least effort for analysis, and would be easiest to perform. Thus, we investigated the relationship of word count with the other evaluation techniques.

3.4.2. Treatment of Data

We analyzed metrics for all interview questions first, followed by three individual questions to investigate if the type of question affected stakeholder feedback. Three individual questions were identified during data analysis to explore the potential effects of question type on stakeholder responses. The chosen questions represented distinct areas of interest to inform design decisions: critique of the idea in general (question 3: Do you think this concept would work?), what the patient receiving the implant would think of the device (question 6: How do you think patients would feel about this device being used during the implant procedure?), and finally, if the participants had any device-related design input (question 9: What would you suggest changing about this device?)

Due to the relatively low number of participants and high number of variables in this study, the results from the response type analysis were combined into category A or B answers, and category C or D answers. Additionally, non-physical prototypes (sketch and CAD) were combined into “virtual” prototypes, and physical prototypes (mockup and 3D-printed) were combined into “tangible” prototypes. Collapsing answer and prototyping categories for the response type analysis amplified the statistical power for the subsequent analyses. Both CAT and word count produced numerical values and did not require any collapsing. We focused on response type as the primary analytical output and referenced both CAT as well as word count when findings from these methods were significant.

Except for the Consensual Assessment Technique, where due to the volume of data, only responses from the first prototype feedback session were included, answers from both prototype reviews were included in the statistical analyses.

For response type, the research team performed individual Chi-squared analyses to determine any statistical significance among stakeholder groups and prototype types across all interview questions. Bonferroni corrections were applied to the typical p-value of 0.05 for the individual tests when groups (stakeholders or prototypes) confounded the results. The detailed results of the Chi-squared analysis can be found in Appendix B.

For the results of the CAT, we performed ANOVAs to evaluate if significant differences existed among the categories (stakeholder groups and prototype types). The ANOVAs were followed by t-tests with the same Bonferroni corrections. We also performed ANOVAs and t-tests with the same Bonferroni corrections to determine if word count revealed any significant differences among stakeholder groups and prototype types.

4. Results

4.1. How does prototype format impact stakeholder input?

Considering response type based on prototype format for all interview questions, we observed significant differences among prototypes: Tangible prototypes (mockup and 3D-printed) provided more category A or B answers than virtual prototypes (sketch and CAD). Tangible prototypes resulted in 77% of category A or B answers, while virtual prototypes resulted in 65% of category A or B answers. Virtual prototypes provided more category C or D answers than tangible prototypes. We also observed that low-fidelity prototypes resulted in fewer category A or B answers than high-fidelity prototypes (sketch vs. CAD and mockup vs. 3D-printed), but without statistical significance. A visual depiction of the results can be found in Figure 3.

Fig. 3.

Results for response category by prototype type for a) Individual answer categories b) Combined answer categories

A Chi-squared analysis of the combined answer categories (A&B, C&D) revealed statistical significance of the findings among individual prototypes (p=0.0011) for all questions (Table 2).

Table 2.

Chi-squared analysis of prototypes, Observed values among participants, Expected values among participants

Prototypes	SK	MO	CAD	3D	Total
A&B C&D	130 73	151 47	129 68	161 44	571 232
Total	203	198	197	205	803

SK	MO	CAD	3D	Total
144 59	141 57	140 57	146 59	571 232
203	198	197	205	803

p= 0,0011

When combining lower and higher fidelity prototypes, tangible prototypes resulted in significantly more category A and B answers (p=0.0001) than virtual prototypes for all questions (Table 3).

Table 3.

Chi-squared analysis of virtual and tangible prototypes, Observed values among participants, Expected values among participants

Prototypes	Virtual	Tangible	Total
A&B	259	312	571
C&D	141	91	232
Total	400	403	803

Prototypes	Virtual	Tangible	Total
A&B	284	287	571
C&D	116	116	232
Total	400	403	803

p= 0,0001

We observed similar results to the response type analysis with the CAT metric for all questions, but CAT showed no statistical significance. However, we observed larger standard deviations across the two groups with this analysis. A final analysis with word count once again revealed similar results, also with no statistical significance, and even larger standard deviations.

Next, we provide quotes that illustrate the range of answers received in response to the different prototype types.

For category A or B answers, Nurse 16 who commented on a 3D-printed prototype, suggested that the device should be made of a non-rigid material: “I would love it if it had been more flexible than this.” The nurse also voiced concerns about the device being disposable, and that a way to prevent repeated use, and the inherent risk of cross-contamination, should be considered by the designers: “Sometimes in our setting, we use it for different patients, so I am thinking that if it will be in such a way that we can use it once for a patient, and that is it. We don’t use it for another patient to put the patient at a risk of infection.”

Student 17, whose response was also categorized as an A answer, suggested that thin patients might not have enough tissue to fill the cavity of the device: “Maybe when someone is very slim, you may not have this space filled. You may not be able to take a great amount of tissue, just take the upper part of the skin and that will make your insertion partial.” The student expanded on this concern and suggested that the device should be made available in different sizes to fit a variety of patients: “Maybe there should be varying sizes for different weight measurement… so maybe from 60–70kg you have this. Then from 50–60, you have a smaller one, so that everyone has his or her appropriate size.” The student also proposed a material change that would alter the procedure and give the provider more control: “Maybe it should be a bit more transparent, so you can see through what you doing, because this, you can’t see what you doing. It should be more transparent.”

Likewise, Doctor 28 commented on the material of the device but was concerned about potential allergic skin reactions: “I think it should be gentle on the skin. If it’s an inert material, then that’s home and prime, then you don’t expect people’s skin to react to the material.” The same doctor suggested durable materials to prevent breakage of the device: “It shouldn’t be too much of a brittle material that can easily give way. Because if it’s in a CHPS [Community-based Health Planning and Services] zone, then you expect this thing to be used, carried from place to place and all of that. If it is something that can easily give way or break, then it may be some minus to it.”

In contrast, the virtual prototypes frequently led to confusion and conflicting information that resulted in C or D answers from the participants. For example, when asked if the concept would work as intended, Student 33, who saw a sketch, first expressed trust in the design: “Yeah it will. Just because of the concept behind it, I think it will,” but later added that it would have been beneficial to see the actual device, undermining the validity of the previous statement: “I would have loved to see the device itself, but it’s nice. It’s really nice. I like the idea. I like everything.”

When participant 25, a nurse who saw a sketch, was asked to comment on the appearance of the device, the input given was: “It would have been better if I have seen it in reality, this drawing; I can’t say much about it.”

Nevertheless, not having enough information did not stop some participants from expressing their opinions. Participant 19, a student, mentioned that the CAD model was “huge,” even though the model shown on the screen afforded no size reference: “I think you have to be very careful, when it goes under the skin and I feel it’s huge so… I still prefer this [free hand insertion method].” However, the student later concluded that additional information would have been required to recommend changes to the device: “I don’t really know the parts and everything well, so I can’t make a comment on that.”

4.2. How does group membership impact stakeholder input?

Considering response type based on stakeholder group for all interview question responses, we found that doctors provided the highest number of category A or B answers, followed by students, then nurses. The response type analysis showed that 78% of the responses provided by doctors were categorized as A or B answers, followed by 69% of A or B answers for students, and 66% of A or B answers for nurses. These findings were opposite for category C or D answers. Here, nurses provided the highest number of category C or D answers, followed by students, and then doctors. The results are shown in Figure 5.

Fig. 5.

Results for response category by stakeholder group for a) Individual answer categories b) Combined answer categories

These findings were significant (p = 0.0029) for the collapsed answer categories (A or B, C or D) among the three stakeholder groups (Table 4).

Table 4.

Chi-squared analysis results for stakeholder groups, Observed values among participants, Expected values among participants

Groups	Nurse	Student	Doctor	Total
A&B	210	124	237	571
C&D	109	56	67	232
Total	319	180	304	803

Groups	Nurse	Student	Doctor	Total
A&B	227	128	216	571
C&D	92	52	88	232
Total	319	180	304	803

p= 0,0029

CAT analysis of usefulness by experts revealed similar results across all interview questions, but none of them were statistically significant. We also observed larger standard deviations for all stakeholder groups with this technique. While word count did not show any statistical significance, it revealed that doctors had the highest average word count. Nurses had higher word counts than students with this analysis, but with a much larger standard deviation.

Next we provide quotes that illustrate the range of answers received by stakeholder groups.

For category A or B for example, Doctor 38 thought that the presented concept was appropriate, but voiced concerns about patients’ perceptions regarding the size of medical devices: “Generally patients are scared when they see big things… So if things are portable, so just… like this. This is small, so I think it’s ok.”

Doctor 11 who saw the cardboard mockup expressed concerns that the tissue might actually not move into the cavity as intended and suggested that the designers might investigate how the skin behaves during the procedure: “You actually have to apply some, a little bit of counter traction on the skin so that the skin is actually not creased or folded. So how sure are we that we don’t get that?”

Doctor 36 stressed the importance of safety and the fact that the device should be disposable to avoid cross-contamination among patients: “If it’s going to be disposable, then I guess it will be safe to use. Because it’s… invasive with the device… a little blood spillage, unless you plan on disinfecting and sterilizing after each patient.”

Students and nurses also provided category A or B answers, but fewer than doctors. For example, Student 20, who reviewed the cardboard mockup, was concerned about the size of the device, but focused more on how the size might influence the procedure: “I kind of think it’s too big… it’s going to be like bulky in between the person’s arm, so if you could have something smaller than this, but with the same concept, I think it’s great.”

Student 23 compared the appearance of the 3D-printed device to an everyday object and posited that it would put a patient at ease: “It looks… seriously, it doesn’t look like something that is used to insert an implant; it’s rather like an opener. Yeah, a bottle opener or something… It does not look like it’s going to be used in the hospital.”

Similarly, Nurse 26 associated the 3D-printed prototype with a writing utensil and concluded that it would be non-threatening: “It’s just like a pen case. It looks like a pen case, so there is no problem with this.”

Nurse 14 thought about how the device would integrate into the implant procedure and stressed the need for training of the service provider to put the patient at ease: “We should [have] adequate training on how the device would be used. Training of the facilitators, and then let the client know how it would be used on them. They would buy into the idea.”

Nurses provided the highest percentage of category C or D answers across all question and prototype types. For example, when asked if the concept would work while reviewing the sketch, Nurse 1 responded: “I think it will be nice, but because I have not seen it, it will be very difficult for me to say. Maybe when it comes out and we’re using it…” When Nurse 5, who reviewed the cardboard mockup, was asked if the concept would work, the answer was referred back to the design team’s earlier description of the device’s intended use: “You said it can do that.”

Members of other stakeholder groups also provided category C or D answers. For example, Doctor 12, who reviewed the cardboard mockup, offered the following insight: “I don’t know, I can’t tell, but I hope it works.” The participant later added: “Ok, well, I really can’t tell, honestly, because I have no idea. I don’t know, I really can’t tell.” Similarly, Student 10, who reviewed a 3D printed model, was not prepared to assess the feasibility of the concept: “Will it work? I have no idea!”

4.3. How does question type impact stakeholder input?

To investigate if the feedback differed among individual questions, we examined the results of our analysis methods for all stakeholder groups (doctors, nurses, and students) and prototype types (virtual and tangible) for questions 3 (Do you think this concept would work?), 6 (How do you think patients would feel about this device being used during the implant procedure?) and 9 (What would you suggest changing about this device?).

4.3.1. Questions and stakeholders

For question 3, 73% of the responses were categorized as A or B answers, for question 6, 89% of the responses were categorized as A or B answers, and for question 9, 54% of the responses were categorized as A or B answers. We found statistically significant differences based on the outcomes of response type analysis (p=0.0000) between questions 6 (most category A or B answers) and question 9 (least category A or B answers) for all stakeholder groups.

For individual stakeholder groups, we found that nurses provided significantly more category A or B answers (p=0.0000) for question 6 (97%) than for questions 3 and 9 (64% and 47%), respectively. Nurses also provided more category A or B answers for question 6 than the other groups. Doctors provided the highest percentages of category A or B answers across the three questions compared to the other stakeholder groups, and offered significantly more category A or B answers (p=0.0011) for both questions 3 and 6 (both 88%) than for question 9 (56%). An analysis of students’ responses showed no significant differences among the three questions, and neither CAT nor word count analyses resulted in any significant findings for questions or stakeholders. The statistically significant findings are shown in Figure 7.

Fig. 7.

Results for category A or B answers for questions 3, 6 and 9 for a) All stakeholders, b) Nurses, c) Doctors

All stakeholders were significantly more likely to provide category A or B answers for question 6 - “How do you think patients would feel about this device being used during the implant procedure?” than for question 9. For example, Nurse 8 thought that not seeing the needle during the procedure would be an asset to the patient: “Once she doesn’t see the needle directly, it will rather make her more relaxed. You explain the procedure to her and how this thing is going to work on her, that will relax her…” Nurse 16 mentioned concerns about the rigidity of the device: “As I said, it’s a bit hard so the patient will feel a bit uncomfortable.” Nurse 24 appreciated what the device would do for the medical provider, but was concerned about the patient: “For us doing the insertion, it will be easy, but thinking about the patient, I think it will be a bit uncomfortable.” Nurse 25 suggested that after a brief explanation, the patients would be fine with the procedure: “Just like you check their BP, you wrap a cuff around their arm, they will be comfortable once you’ve explained the procedure to them.” Nurse 43 voiced concerns about patient comfort and asked if the designers had considered this already: “I don’t know whether there would be some discomfort when the tissues are going there, [do] you anticipate that?”

In contrast, question 9 (What would you suggest changing about this device?) resulted in the lowest number of category A or B answers across all stakeholder groups. Category C or D answers for question 9 included examples like Doctor, who 9 had only this to say: “I think it’s fine,” or Doctor 11, who was uncomfortable commenting on technical details: “Wow you’re talking to me [about] engineering…” Doctor 6 who saw a sketch of the device expressed a need more information to make any recommendations: “I am wondering how it’s going to lift the skin under the cavity, so until I see it, I can’t comment.”

4.3.2. Questions and prototypes

We found statistically significant differences in response type (p=0.0001) between virtual and tangible prototypes for question 9. 78% of the responses to tangible prototypes were categorized as A or B answers, while only 24% of the responses to virtual prototypes were categorized as A or B answers (Figure 8). We also observed a statistically significant differences when assessing for usefulness of the feedback with CAT between virtual and tangible prototypes (p=0.0002), while word count analysis showed similar results but no statistical significance (p=0.3507) combined with large variation.

Fig. 8.

Results for question 9 between prototype types and a) Response type, b) Expert rating of usefulness

Stakeholders who saw tangible prototypes when answering question 9 (“What would you suggest changing about this device?”) responded with more category A or B answers, higher ratings of usefulness by experts, and lengthier answers than those who reviewed virtual prototypes. Examples of category A or B answers included quotes like the following by Nurse 4 who saw the cardboard mockup: “I like it, but I think the size is a little big. Yeah, if it can be a little [more] portable, that will be fine.” Doctor 10 was even more specific about how the size of the device might be critical to a diverse patient population: “Maybe there should be some form of adjustment to take care of thin people because it may accommodate more than the skin in the subcutaneous tissue. It may take some amount of muscle, so maybe some modifications should be made for thin people.” Nurse 16 who commented on a 3D-printed prototype added concerns about the disposable nature of the device: “Yes, it’s enough to know it disposable, but unfortunately, for our setting, sometimes, due to inadequate consumables and all, we turn to reuse it. So if it can be done in such a way that you can’t reuse it…”

In contrast, virtual prototypes resulted in fewer category A or B answers, lower ratings of usefulness by experts, and shorter answers for question 9. Examples for category C or D included answers like “I can’t say much about it until I start using it or something” by Nurse 1 who did not think that the proposed concept was realistic enough to comment on. The nurse later added: “I think for now no because this is just on paper.” This was echoed by Doctor 6 who questioned if the device would actually work and wanted to see the actual device perform: “Ok I haven’t seen it actually been done before, so I am wondering how it’s going to lift the skin under the cavity. So until I see it, I can’t comment.”

5. Discussion

5.1. Influence of prototype format on stakeholder feedback

In our examination of how prototype format influenced the feedback stakeholders provided, we found that tangible prototypes provided more category A or B answers, higher ratings of usefulness by experts, and longer responses than virtual prototypes across all stakeholder groups and questions. These findings echo recommendations that call for tangible prototypes to be used for collecting stakeholder input on products and devices (De Beer et al., 2009; Kelley, 2001; Otto & Wood, 2000; Schrage, 1999), but other researchers have found that, depending on the task, virtual prototypes can be equally beneficial during product development, as long as designers are aware of their benefits and limitations (Rudd et al., 1996; Ulrich & Eppinger, 2015; Walker, Takayama, & Landay, 2016).

Using different types of prototypes might also affect variations of answers. The variations in the types of responses, ratings of usefulness by experts, and word count in our study were smaller for the cardboard mockup and 3D-printed model than for the than the virtual sketch and CAD model. This greater variation within the virtual prototyping categories might suggest greater diversity in participants’ ability to respond to these prototypes and likely makes the process of synthesizing input more difficult for designers. Several participants stated they were not able to obtain enough information from the virtual prototypes, which in some cases, resulted in unjustified or non-useful feedback (category C or D responses). Limited experience with, and exposure to, design processes, medical device development, or the review and critique of virtual prototypes might have contributed to the perceived need for additional information. As a result, participants might have felt overwhelmed by the task, which could have led to frustration and emotional responses rather than analytical processing of information (Frijda, 1989; Scherer, 2003; Winston & Cupchik, 1992).

Analysis of low- vs. high-fidelity showed no statistical significance, but within each category, the more refined prototypes (CAD model for virtual, and 3D-printed for tangible prototypes) were related to more category A or B answers, higher ratings of usefulness by experts, and lengthier responses. These results align with Brandt’s (2007) findings that higher levels of detail within a prototyping category led to smaller variations and more focused conversations between stakeholders and designers. Similarly, studies evaluating stakeholder feedback on new product concepts have found that the highest level of prototype quality correlated with higher ratings by the stakeholders regardless of the criteria, e.g. functionality or creativity of an idea (Häggman, Tsai, Elsen, Honda, & Yang, 2015; Kudrowitz et al., 2012; Sauer & Sonderegger, 2009).

Our findings align with Hannah’s (Hannah, Joshi, & Summers, 2012) conclusion that higher fidelity prototypes lead to more desirable results (more confidence), but also contradict the findings of Viswanathan and Linsey’s (2011) study that investigated the effects of prototypes on designers’ creative process. In this study, situated early in the design process, the researchers found that the physical models that required more time and effort to create led to more design fixation by the designers than the physical models that required less building effort. The researchers also found that low-fidelity prototypes invited more contribution to the design and that higher fidelity prototypes were sometimes seen as “too complete” to warrant more input by participants. Considering the effects of physical models on designers rather than stakeholders, the findings of this study suggest that lower quality physical prototypes might, at least sometimes, be preferable.

On the other hand, some studies found little difference between low- and high-fidelity prototypes, but these tended to focus on non-tangible, two-dimensional products like user interfaces and web sites only (Lim, Youn-kyung et al., 2006; Walker et al., 2016). In our study, the lesser refined prototype categories led to larger variations and more confusion in the stakeholder feedback. For example, one nurse asked which one of the views of the sketch to comment on, not realizing that all views of the sketch depicted the same product.

Similarly, even though participants were made aware they were looking at a prototype, some still voiced concerns about properties specific to a particular prototype, such as the fact that the blood of a patient might stain the cardboard material used for the mockup. This insight might indicate that some participants were not able to look beyond the prototype format and its inherent limitations when assessing lower-fidelity representations.

5.2. Influence of stakeholder group membership on stakeholder feedback

In our examination of group membership, we found differences among stakeholder groups. The feedback doctors provided included the most category A or B answers, the highest ratings of usefulness by experts, and the longest responses. The feedback students provided included more category A or B answers and higher ratings of usefulness by experts than nurses, but nurses provided longer responses than students. There might be several reasons for these differences. First, the introduction of “design thinking” to a clinical environment is a fairly recent development (Kalaichandran, 2017; Roberts, Fisher, Trowbridge, & Bent, 2016) that is often limited to physicians and medical students, and frequently exclude nurses (Rosen & Ku, 2016). Therefore, many healthcare professionals, including those in this study, likely had limited experience with the design and development of medical devices.

Further, nurses in African countries have traditionally been trained with a focus on physician order execution and task completion (Marks, 1994). The mission-style training approach, adopted by many sub-Saharan African countries from the British colonial system (Edwards, 1957), might have introduced a social desirability bias, where nurses are not necessarily accustomed to providing critique and voicing their opinions. More recently, efforts have been made to redefine nursing practices from a more task-oriented approach to one of caring for and caring about patients (Savage, 1995) but without necessarily challenging the hierarchal structure within the healthcare system. These factors might explain why nurses performed better on the patient-centric question than on the design-specific question.

In addition, the training of Ghanaian medical doctors often includes fellowships in the United Kingdom and the United States (Klufio, Kwawukume, Danso, Sciarra, & Johnson, 2003b), introducing them to a culture of innovation and critique. This international experience might explain why doctors provided the most category A or B answers to questions addressing the design of the medical device used in this study.

Nurses frequently provided less critical observations and instead compared the presented device to everyday objects. Leder defines these “looks like” or “feels like” responses as prototypicality, a cognitive way for a reviewer to associate a new object with another and more familiar object. The association of information content with their own situation and emotional state can lead a reviewer to be satisfied with this simple recognition. Parsons (1989) posited that “A naive perceiver might be satisfied with the recognition of the train station in Monet’s La Gare Saint-Lazare because ‘he likes trains because they remind him of a journey.’” This observation is not limited to art, as differing levels of expertise influence how new concepts are perceived in other domains as well. While experts tend to abstract principles when solving a problem, novices often focus on literal features (Chi, Feltovich, & Glaser, 1981). In our study, we saw indications that an emotional evaluation, association with a familiar product, and focus on features might have limited cognitive inquiry by a reviewer. Several participants compared the device concept to everyday objects like a bottle opener or pen case and concluded that since these objects are safe, non-threatening devices, a medical device concept that looks or feels similar must therefore share similar qualities.

Similar to studies that showed that stakeholder input can be contradicting, making it difficult for designers to synthesize information (Mohedas, Daly, & Sienko, 2014; Scott, 2008), we, too, found evidence of sometimes conflicting stakeholder input. Even when participants’ feedback consisted of category A or B answers, high ratings of usefulness by experts, and lengthy responses, their input was sometimes incompatible. For example, one stakeholder asked for the device to be transparent so that practitioners can see what they are doing, while another stakeholder appreciated the fact that an opaque device would hide the needle from the patient during the implant procedure. Both participants provided useful input, yet suggested opposing product qualities (transparent vs. opaque). The fact that these arguments could both be valid underscores that designers cannot simply take stakeholder input at face value. Instead, designers should expect contradicting feedback, especially early in the design process, when seeking a comprehensive understanding of the requirements. Prototypes provide a chance to interact with, and evaluate, proposed solutions and can be used to help uncover “unknown unknowns” (Jensen et al., 2017). Designers need to embrace these findings and use them to inform prototyping strategies and design decisions.

Our results align with studies that have shown that physical prototypes were more widely understood by and accessible to participants, positively affected user emotions, and prompted participants to respond with a high degree of confidence (De Beer et al., 2009; Häggman et al., 2015; Sauer & Sonderegger, 2009). Our results also reflect findings by Björklund (2013) and Simon (1973) that the mental representations of design experts (how design problems are transformed or structured into mental representations) were broader, more detailed, and more geared toward problem solving. Similar to studies that have shown the benefits of exposure to multinational and tangential experiences during training of medical doctors (Klufio, Kwawukume, Danso, Sciarra, & Johnson, 2003a) and biomedical engineers (Sienko, Kaufmann, Musaazi, Sarvestani, & Obed, 2014), we saw indications that stakeholders who might have been exposed to innovation and critique in addition to training in medical practice and patient care might have been better prepared to provide feedback on the design and development of new products, here medical devices.

5.3. Influence of question type on stakeholder feedback

In our examination of how question type influenced stakeholder feedback, we found that the feedback participants provided depended on the question type as well as on participant characteristics. Related to this finding, several studies have shown that the questions designers ask are contingent on the phase of the design process (Christie et al., 2012; Menold et al., 2017). Combined, these findings suggest that designers need to consider the questions they ask as well as whom they are asking in all stages of the design process. Not all stakeholders seem to be equally able to provide input to all questions, and designers may need to rephrase a question, or situate it in a different context, depending on whom they are engaging with. Through the question type, designers can, and need to, enable stakeholders to relate to the design problem and feel comfortable enough to respond.

For example, we found that question 6 “How do you think patients would feel about this device being used during the implant procedure?” resulted in the most category A or B answers across all stakeholder groups. Particularly, nurses provided the most category A or B answers for this patient-centric question that was situated within the participants’ knowledge domain of treating and caring for patients. By no longer asking participants to critique the device directly, this question took them out of the “hot seat” and allowed them to assume the role of caregiver, associating with their patients. This new perspective enabled participants to talk more freely and comment on the experiences both patients and caregivers might have when using the device during the contraceptive implant insertion procedure. Similar to Leder’s example earlier (Leder et al., 2004), question 6 may have enabled stakeholders to pick up on the nuances only an expert could. Familiarity and experience may have allowed participants to move through several stages of information processing for this question, evaluating the device on a much deeper level than before and thinking through the procedure from the patient and caregiver perspectives. For this particular question, participants had become experts, and the highest number of category A or B answers we recorded for this question reflected this level of expertise.

We also found that question 9, “What would you suggest changing about this device?” resulted in the lowest number of category A or B answers for all participants and all prototypes. Two reasons come to mind for why this question might have fared so poorly: First, it was asked last and participants might have exhausted their input on the previous eight questions and simply gotten tired of repeating themselves. Second, this question asked participants directly what they would change about the design. Since some participants had little or no experience with medical device design, this might have caused them to feel uncomfortable and/or overwhelmed. As the findings from other questions indicated, participants were more likely to provide input when they were asked about specific details rather than to give general input. This is another important finding, since novice designers tend to ask more general questions. Our findings suggest that not all stakeholders are equally prepared to do this; designers need to consider their stakeholders and carefully select and frame questions that enable participants to provide feedback on the proposed design.

5.4. Influence of analytical methods on stakeholder feedback

When comparing the findings of the different analytical methods employed during this study, all three techniques identified similar results: Tangible prototypes led to more category A or B answers, higher ratings of usefulness by experts, and longer responses than virtual prototypes. All techniques also revealed that doctors provided more category A or B answers, higher ratings of usefulness by experts, and longer responses than students, who provided more category A or B answers and higher ratings of usefulness by experts than nurses. Nurses provided longer responses than students, but with large standard deviations and no statistical significance.

In addition, there were a few other noteworthy observations. For example, the categorization of responses by type identified the highest number of statistically significant differences among prototypes and stakeholder groups. This finding is not surprising since this method relied on carefully developed codes to analyze the data. The codes provided specific criteria for the analysis and therefore revealed the most precise distinctions among the input categories. The iterative development of codes, in addition to several rounds of coding, required a substantial time commitment from the researchers. Despite these efforts, the results suggest that this method led to the most insightful, significant, and reliable findings.

The CAT relied on criteria individual raters established by themselves (Amabile, 1983), and we observed the same results as with the categorization of responses by type when analyzing the results. At the same time, the larger standard deviations and less significant results of this analytical method make the findings less reliable for eliciting stakeholder input. The small number of expert raters who participated in the analysis likely contributed, and a higher number of experts might improve the results.

Word count used no other criteria than number of spoken words and showed less pronounced, and sometimes even conflicting results with no statistical significance. This analytical method also resulted in the largest standard deviations that occurred where we noticed mismatches among the techniques. In one extreme case, when examining the influence of prototype type on question 9, the standard deviation of 54.75 words even exceeded the average count of 37.79 words for virtual prototypes, large enough to question the validity of this result.

We also found that for all participants, nurses had the highest word count for question 9, but fewer than expected category A or B answers and the lowest average ratings of usefulness by experts for this question. In contrast, doctors had the lowest word count, but the most category A or B answers and second-highest ratings of usefulness by experts for the same question. In these cases, the word count results were in opposition to the findings of the other techniques. These observations contrast with Blumenstock’s study (2008) that found a positive correlation among the length and the quality of articles published on Wikipedia. However, such articles are peer-reviewed and nominated, a process that is absent when collecting stakeholder input. In an earlier study, Weber (Weber, 1983) concluded that content analysis may be the preferred way to generate quantitative indicators, but our findings highlight the need to consider Morgan’s argument (D. L. Morgan, 1993) that it is critical to determine ‘what’ to code for in content analysis and that a range of techniques for analyzing qualitative data might be preferable. In summary, without developing codes for content analysis, how much a person says seems not to be a good indicator of quality of content, making word count the least reliable analytical technique used in this study.

6. Implications

The findings of this study are important for design practitioners planning to use prototypes, and in particular for projects designed at distance when access to stakeholders can be challenging. Specifically in global health design, where geographic distances and time-zone differences can limit and restrict conversations, interactions with stakeholders need to be carefully planned and executed. Here, a successful prototyping strategy is even more critical and should encompass the following elements:

First, designers must select appropriate prototype types. For example, when looking for procedural or in situ feedback, simple prototypes like sketches might not enable stakeholders to address issues that a functional prototype might reveal (Sauer, Franke, & Ruettinger, 2008). The commonly accepted prototyping best practices (quick and simple) used in the United States are not necessarily universally transferable and need to be adjusted to the unique context and background of the design project. It is not enough to consider that different stages in the design process call for different types of prototype (Atman et al., 2007) – designers also need to select the most appropriate prototype types that allow stakeholders to best respond and provide useful input.

Second, designers need to recognize that not all stakeholders are equally prepared to respond to all prototypes – a sketch might work well for an engineer but not resonate with a social worker. When stakeholders have limited domain experience, or feel inadequately equipped to evaluate a new concept, they might not be able to move through the stages of information processing necessary for a comprehensive evaluation. Instead, they might feel overwhelmed and express an emotional response that can be misleading and even harmful, especially when designers do not have experience synthesizing feedback they receive. Designers need to recognize who their stakeholders are and select the types of prototype that best support these groups.

Third, the questions designers ask when using prototypes need to be carefully selected to enable dialog between stakeholders and designers. Designers need to consider the context of the question and develop questions that enable stakeholders to more effectively draw upon their own expertise. A stakeholder who is not well prepared to provide technical input might be an excellent candidate to offer insight into the social or psychological impact a new design concept might have on a community. Having experience with the use of a device is not the same as having experience with the design and development of a device. It is up to the designer to ask the “right” questions and take advantage of individual stakeholders’ expertise.

Fourth, the findings can inform design pedagogy and curriculum development (Authors, 2017), since the application of the results are not limited to medical device design. They can be transferred to other products, services or systems where designers use prototypes to gather stakeholder input. In this study, prototype type, stakeholder group and question type all influenced stakeholder feedback. Educators can capitalize on this insight and guide students to carefully consider the unique circumstances of their design project. In particular, they can encourage students to develop prototyping strategies that optimize their interactions with stakeholders when looking for feedback on new design proposals.

7. Limitations and Future Work

There were several limitations to this study that could be addressed in future work. Only a subset of the answers participants provided was analyzed in detail, and the number of participants could be expanded. No information on participants’ prior design experience was collected and some were likely inexperienced with providing design feedback. The study was limited to one unique setting and stakeholders with limited cultural, geographical and professional background variety. Future studies might explore the extent to which the findings can be transferred to different stakeholder groups, prototypes of products in other arenas, as well as systems and processes. The questions used during the interviews represent typical questions that designers might pose to stakeholders and were not explicitly designed or selected to specifically study the effects of question type on response type. We did not investigate how the individual features of a prototype or the order in which the prototypes were presented influenced the usefulness of the feedback that was elicited from stakeholders. A male researcher who was not a native of Ghana conducted all interviews and, although English is considered an official language, it might not have been the first language for some participants. These factors might have influenced the participants’ responses, specifically the richness and explicitness of their feedback.

8. Conclusion

We found that tangible prototypes resulted in more category A or B answers and higher ratings of usefulness by experts than virtual prototypes, regardless of their fidelity. Designers need to be aware of this tendency and should proactively develop context-specific strategies that complement the “quick and simple” approach to prototyping, since prototype type matters. We also found that participants with more domain experience (here experience in obstetrics and gynecology) provided the most category A or B answers and the highest ratings of usefulness by experts as compared to participants with less domain experience. This was however not true for all questions, and certain groups responded with more A or B answers and higher ratings of usefulness by experts to some questions than others. Questions positioned within a stakeholder’s professional domain resulted in more category A or B answers and higher ratings of usefulness by experts than general and technical questions. It is therefore important for designers to carefully consider what questions they ask, and to whom they are asking them. Specific rather than general, or summative questions that are situated in a participant’s domain have the potential to empower stakeholders to comprehensively evaluate the prototypes with which they are presented.

10. Appendix

A.Interview Questions:

1) In general, what do you like about this concept?

2) What do you dislike about this concept?

3) Do you think this concept will work?

4) Do you think the device would be easy or hard to use?

5) Do you think the device would be safe to use?

6) How do you think patients would feel about this device being used during the implant insertion procedure?

7) What do you think about what the device looks like?

8) Do you think this device would be appropriate for CHPS workers (Community based Health Planning and Services) to use in rural settings?

9) Based on what you know about this concept, what would you change about this device?

B.Chi-squared Analysis of Coding Results