Abstract
Introduction
Data on the effectiveness of strategies for the recruitment of American Indians (AIs) into research is needed. This study describes and compares methods for identifying and recruiting AI tobacco users into a pilot study.
Methods
Community-based strategies were used to recruit smokers (n=35), e-cigarette users (n=28), and dual users (n=32) of AI descent. Recruitment was considered proactive if study staff contacted the individual at a pow wow, health fair, or vape shop and participation on-site or reactive if the individual contacted the study staff and participation occurred later. Screened, eligible, participated and costs and time spent were compared with Chi-square tests. To understand AI descent, the relationship between number of AI grandparents and AI blood quantum was examined.
Results
Number of participants screened via the proactive strategy was similar to the reactive strategy (n=84 vs n=82;p-value=0.8766). A significantly greater proportion of individuals screened via the proactive than the reactive strategy were eligible (77% vs. 50%;p-value=0.0002) and participated (75% vs. 39%;p-value=<.0001). Per participant cost and time estimated for the proactive strategy was $89 and 87 minutes compared to $79 and 56 minutes for the reactive strategy. Proportion at least half AI blood quantum was 32%, 33%, and 70% among those with 2, 3, and 4 AI grandparents, respectively (p=0.0017).
Conclusion
Proactive strategies resulted in two-thirds of the sample, but required more resources than reactive strategies. Overall, we found both strategies were feasible and resulted in the ability to reach sample goals. Lastly, number of AI biological grandparents may be a good, non-invasive indicator of AI blood quantum.
Keywords: American Indians, Electronic Cigarettes, Epidemiologic Studies, Minority Groups, Pilot Projects, Tobacco Use, Recruitment
INTRODUCTION
Although the inclusion of minorities in research is necessary for valid inferences within all sub-populations and mandated in National Institutes of Health funded research (1), minority groups are often underrepresented in research. Reasons for exclusion of minority groups are complex and may stem from planned exclusion, inadvertent exclusion, non-participation, or a combination of these reasons. Participant recruitment can be one of the most challenging aspects of ensuring minority populations are well-represented (2, 3). In addition to conventional recruitment barriers when working with the general population, researchers may also face cultural challenges to research participation with minority populations (3). As a result, there is interest in effective recruitment strategies for minority populations.
Recently, recruitment for a study with American Indian (AI) tobacco users was completed. Although AIs make up less than 2% of the total US population(4), they have some of the highest levels of tobacco use in the US (5, 6). The study was undertaken to estimate the rate of nicotine metabolism, which is one of the most important predictors of nicotine addiction (7), among AI cigarette smokers and electronic cigarette (e-cigarette) users. Prior studies investigating nicotine metabolism have had significant implications for the field of tobacco control, yet none included AI participants.
Some studies with AI participants, most published more than a decade ago, have described their recruitment methods and lessons learned (8–12). A genetic-based study that investigated a molecular basis for differential response to cancer treatment, credited its recruitment of twenty-six AI participants to achieving a trusting partnership with tribal communities and engaging these communities in all aspects of the research endeavor(11). The Strong Heart Study, which is a longitudinal cohort study of cardiovascular disease and its risk factors in more than four thousand AIs (10), attributed their recruitment success to the strong commitment of the participating AI communities and the use of a variety of recruitment techniques. No studies, to our knowledge, have described methods for reaching AI tobacco users. Furthermore, we were unable to identify extensive guidance on the most effective, yet non-invasive, way to assess AI descent for research eligibility. Due to these research gaps, we described the community-based strategies employed to recruit adults of AI descent into a study titled ‘Nicotine Metabolism in American Indian Smokers and Electronic Cigarette Users’.
METHODS
Study design
This study was approved by the University of Oklahoma Health Sciences Center (# 6317) and the Indian Health Service Institutional Review Boards (# P-16-01-OK). Community-based strategies were employed to recruit adults of AI descent who were: 1) exclusive cigarette smokers, 2) exclusive e-cigarette users, and 3) dual users of cigarettes and e-cigarettes. Minimum sample size needed for each of the three user groups was 27. The informed consent process for all participants was the same and included a description of the nature, purpose, risks and benefits of the study and the participant’s right to withdrawal. Only after written consent, participants provided a urine sample, a measurement of carbon monoxide in exhaled breath (eCO), and completed a questionnaire. Participants were compensated $45. Due to the high proportion of AIs in the state (9%) (4), Oklahoma served as an ideal location for this research study. Eligible participants (Table 1) included persons of AI descent who reported having at least two biological grandparents of AI race and were between 18 and 65 years of age. Additional inclusion criteria depended on the individual’s use of cigarettes and/or e-cigarettes. Exclusion criteria are also listed in Table 1.
Table 1.
Inclusion | Exclusion |
---|---|
All participants | |
|
|
Cigarette smokers | |
|
|
E-cigarette users | |
|
|
Dual users | |
|
Assessment of AI descent
Since a large part of the variance in nicotine metabolism is genetic in origin (7), we sought to recruit individuals of AI descent (e.g., individuals with AI ancestors). We were unable to find published guidelines on how to assess AI descent. As a result, we consulted experts in AI studies. We considered asking participants to self-report their degree of Indian blood, known as blood quantum, in the eligibility screener. However, we were advised that it might be too sensitive of information to collect before the informed consent process. In a prior study of nicotine metabolism with Alaska Natives (13), the researchers asked whether the individual was Alaska Native and if they had at least two biological grandparents who were Alaska Native race. As a result, we asked the following questions in the screener: 1) ‘Do you consider yourself AI?’ (yes, no); 2) ‘How many of your biological grandparents are of AI race?’ (0, 1, 2, 3, or 4). Individuals who responded ‘yes’ and ‘2’, ‘3’ or ‘4’ to questions 1 and 2, respectively, were eligible for the study. In the questionnaire, which was completed after the informed consent process, participants were asked the following question to further understand their AI descent: ‘Which best describes your total degree of Indian blood?’ (‘less than 1/4’, ‘1/4 to less than 1/2′, ‘1/2 to less than 3/4’, ‘3/4 to full’).
Recruitment timeline
According to an unpublished analysis of 2014 Oklahoma Behavioral Risk Factor Surveillance System data, 12.8% of self-reported AIs in the state of Oklahoma were exclusive cigarette smokers, 5.0% were exclusive e-cigarette users, and dual use was estimated among 8.5%. We expected recruiting 3 participants per month for each user group during a 9-month period. Due to the relatively small sample size needed per user group (n=27), the proportion of AIs in Oklahoma, and the proportion of AIs estimated to be in each user group, we did not anticipate a problem with reaching the sample size goals.
Recruitment methods
The following recruitment strategies were recommended by the Southern Plains Tribal Health Board (SPTHB), which serves as a unified voice for the 38 federally-recognized tribes in Oklahoma. Through their Tribal Epidemiology Center, the SPTHB has extensive experience recruiting AI participants for research.
Reactive strategies
Study advertisements were sent out via the University of Oklahoma Health Sciences Center (OUHSC) email listserv. Study advertisements were also posted on Oklahoma City’s (OKC) and Tulsa’s Craigslist sites. These advertising methods were zero-cost. Interested individuals who contacted the study personnel were screened for eligibility over the phone. If eligible, individuals screened over the phone were asked to travel to OUHSC in OKC or Tulsa to participate in the study. Since study staff was located in OKC, study staff did not travel to enroll participants in OKC. However, study staff traveled from OKC to Tulsa to enroll eligible individuals in Tulsa. We obtained OUHSC Institutional Review Board (IRB) approval for these reactive strategies.
Proactive strategies
Pow wows
A pow wow is an AI celebration involving singing, dancing and the renewal of friendships. These community events help preserve the heritage and culture of AIs. The pow wow Calendar from PowWows.com features pow wows from across North America and was used to identify upcoming pow wows within Oklahoma (14). We received permission from various pow wow coordinators and tribal leaders to setup a table to recruit and enroll participants into the study at their respective pow wows. OUHSC IRB approval was obtained to attend each event.
Tribal health fairs
To bring health awareness to all generations, AI tribes sponsor health fairs. We received permission from tribal leaders, health clinics, and health fair coordinators to setup a table to recruit and enroll participants. We obtained OUHSC IRB approval for each health fair attended. Since one of the health fairs was held in an Indian Health Service (IHS) clinic, we also obtained IHS IRB approval to attend.
Vape shops
In the state of Oklahoma, there are nearly 400 independent retail specialty shops, known as “vape shops”, which exclusively sell e-cigarettes. Vape shops were selected if they were located within 1-hour of driving distance from OUHSC in OKC and in a city with a high proportion of AI residents. We received permission from vape shop owners and from OUHSC IRB to recruit and enroll on site.
Cost and time
Cost and time spent per strategy were estimated. Total time on behalf of the study staff was calculated to include the time spent traveling and recruiting, conducting the screening, informed consent, and participation processes. We considered total cost to include participant compensation ($45 per participant), travel costs ($0.575 federal mileage reimbursement rate 2016), personnel costs (one person at $20 per hour of travel or per hour of time), and booth or registration cost.
Statistical analysis
A participant was considered to be recruited via the reactive strategy if the individual contacted the study staff and was screened over the telephone and then participated in the study in-person at a later time. A participant was considered to be recruited via the proactive strategy if the study staff contacted the individual at a site or event, and then both screening and participation occurred in-person at the site. Proportion screened, eligible, and consented to participate were calculated for each recruitment strategy and compared with Chi-square test for association. Participant characteristics were also compared across recruitment methods with Chi-square or Kruskal–Wallis tests. A p-value <0.05 was considered statistically significant. Analyses were performed in the Fall of 2016 using SAS software version 9.4 (15).
RESULTS
Having enrolled a total of 95 participants in 8 months, recruitment was completed one month ahead of schedule. In regards to tobacco user groups, the minimum sample size needed per user group (n=27) was exceeded by recruiting 35 cigarette smokers, 28 e-cigarette users, and 32 dual users (Figure 1). We started recruitment by targeting cigarette smokers and dual users in the study advertisements. Roughly 3 months into recruitment, study staff began attending vape shops and advertisements targeting e-cigarette and dual users were disseminated. As a result, the first user group to reach the minimum sample size was cigarette smokers followed by dual users and e-cigarette users.
Reactive strategy
Thirty study advertisements, roughly 1 to 2 weeks apart, were posted April–September 2016 on OKC’s and Tulsa’s Craigslist sites. Additionally, 5 study advertisements were sent out via the OUHSC email listserv. A total of 82 individuals contacted the study staff and were screened over the telephone (Table 2). Of the 82, about half (n=40) were eligible to participant. The primary reasons for ineligibility were not being AI and/or not having at least 2 biological AI grandparents. Of the 40 eligible, 32 participated in the study. The remaining (n=8) did not show up for their scheduled appointment or were unable to schedule a time to participate. Reactive strategies resulted in the recruitment of 9 cigarette smokers, 13 e-cigarette users, and 10 dual users.
Table 2.
Number screened | Number eligible | Number participated
|
||||
---|---|---|---|---|---|---|
Total | Cigarette smokers | E-cigarette users | Dual users | |||
Reactive | 82 | 40 | 32 | 9 | 13 | 10 |
Proactive | 84 | 65 | 63 | 26 | 15 | 22 |
Pow wows | 28 | 21 | 20 | 19 | 0 | 1 |
Health Fairs | 29 | 21 | 20 | 7 | 6 | 7 |
Vape Stores | 27 | 23 | 23 | 0 | 9 | 14 |
All strategies | 166 | 106 | 95 | 35 | 28 | 32 |
Of the 32 participants recruited via reactive strategies, more than half (56%; n=18) participated in OKC. As a result, there were zero costs and time associated with traveling to see these participants (Table 3). Fourteen individuals participated in Tulsa. Study staff traveled 212 miles round trip on 4 occasions to see participants in Tulsa. A total of 16 hours was spent screening and enrolling participants. A total of $1,440 was spent on participant compensation and $600 was estimated for study personnel. As a result, the total time and costs associated with the reactive strategies was estimated at 30 hours and $2,530. Per participant the average costs and time estimated were $79 and 56 minutes.
Table 3.
Travel time | Travel costsa | Misc. costs | Time with partb | Part. compensationc | Personnel costsd | Total time | Total costs | Avg. cost per part. | Avg. time per part. | |
---|---|---|---|---|---|---|---|---|---|---|
Reactive | 14 hrs | $490 | 16 hrs | $1440 | $600 | 30 hrs | $2530 | $79 | 56 mins | |
OKC | 0 | $0 | 9 hrs | $810 | $180 | 9 hrs | $990 | $55 | 30 mins | |
Tulsa | 14 hrs | $490 | 7 hrs | $630 | $420 | 21 hrs | $1540 | $110 | 90 mins | |
Proactive | 23 hrs | $800 | $150 | 68 hrs | $2,835 | $1820 | 91 hrs | $5605 | $89 | 87 mins |
Pow wows | 8 hrs | $265 | $150e | 20 hrs | $900 | $560 | 28 hrs | $1875 | $94 | 84 mins |
Health Fairs | 5 hrs | $180 | 12 hrs | $900 | $340 | 17 hrs | $1420 | $71 | 51 mins | |
Vape Shops | 10 hrs | $355 | 36 hrs | $1035 | $920 | 46 hrs | $2310 | $100 | 120 mins | |
All | 37 hrs | $1290 | $150 | 84.5 hrs | $4275 | $2420 | 121 hrs | $8135 | $86 | 76 mins |
$0.575 per mile;
Includes participant recruitment, screening and participation;
Each participant was compensated $45 for completing the study;
One person at $20 per hour
Cost to register as a vendor for pow wows; part.=participant; avg.=average
Proactive strategy
Eighty-four individuals were screened for eligibility at one of three proactive recruitment sites. The vast-majority (80%; n=65) were eligible to participant in the study. Sixty-three participated, which included 26 cigarette smokers, 15 e-cigarette users, and 22 dual users. Study staff traveled a total of 1,390 miles to attend proactive strategies. A total of 68 hours was spent recruiting, screening and enrolling participants. We also estimate spending $2,835 on participant compensation and $1,820 on study personnel. As a result, the total time and costs associated with the proactive strategies was estimated at 91 hours and $5,605. Per participant the average costs and time were $89 and 87 minutes.
Comparison of reactive versus proactive recruitment methods
A similar number of participants were screened via the proactive (n=84) and reactive (n=82) recruitment strategies (p=0.8766). More than three–fourths (77%; n=65) of individuals screened via proactive strategies were eligible at screening, and this was significantly higher than the proportion (50%; n=40) eligible via the reactive strategies (p=0.0002). Three-fourths (75%; n=63) of individuals screened via proactive methods participated, and this was significantly higher than the proportion (39%; n=32) who participated via reactive strategies (p=<.0001). Thus, nearly 4 participants were added for every 5 screened via proactive strategies; whereas, 2 participants were added for every 5 screened with reactive recruitment.
In terms of participant characteristics, there was no difference in age (p=0.3587), the number of biological grandparents of American Indian race (p-value=0.1377), or American Indian blood quantum (p=0.0707) when comparing across recruitment strategies. A greater proportion of participants recruited by reactive strategies than proactive strategies were female (71.4 versus 50.0; p-value=0.0395).
Comparison of methods for assessing AI descent
All participants had complete information on number of biological grandparents of AI race assessed in the screener and blood quantum assessed in the questionnaire. There was a statistically significant association between the number of biological grandparents of AI race and blood quantum (p=0.0017). Proportion of at least half AI blood quantum was 32%, 33% and 70% among participants with 2, 3, and 4 biological grandparents of AI race, respectively.
DISCUSSION
This research described strategies for identifying and recruiting AI cigarette smokers and e-cigarette users into a pilot study. Proactive recruitment at pow wows, tribal health fairs, and vape shops resulted in the highest number of participants but required more resources than reactive strategies. Ultimately, we found using both reactive and proactive strategies resulted in the ability to reach recruitment goals ahead of schedule. This study is timely given that AIs are a minority population with a high prevalence of tobacco use and often underrepresented in research.
In order to reach the recruitment goals, study staff attended an array of events and sites that were known to be attended by AIs and/or tobacco users. Pow wows were found to be the primary source for cigarette smokers recruited via proactive strategies; while vape stores were most effective for the recruitment of dual users followed by e-cigarette users. Attending health fairs resulted in a similar number of smokers, e-cigarette users and dual users. Since pow wows and health fairs were sponsored by tribes and the majority of attendees were AI, ineligibility at pow wows and health fairs was largely due to tobacco use status; while ineligibility at vape stores was largely due to patrons not being AI. Together, these proactive recruitment strategies resulted in two-thirds of the sample. The success of the proactive strategy reinforces findings from prior studies, especially the Strong Heart Study (10), which found face-to-face (i.e., proactive) interaction between study staff and AI communities to be an effective method of participant recruitment.
An essential component in determining the effectiveness of one recruitment method over another is to consider the costs and time spent. The average cost per participant recruited by proactive and reactive strategies was $89 and $79, respectively. It is important to note, that this was not a multi-site study. Study staff was located in OKC, so any participants recruited via reactive strategies in the Tulsa area resulted in the need for study staff to travel from OKC to Tulsa. Thus, the travel cost associated with reactive strategies was due to study staff traveling to a second study location. Other studies may consider having personnel at multiple sites to reduce the costs and time spent traveling.
In terms of average time spent per participant, the reactive strategy (56 minutes) was more efficient than proactive strategy (87 minutes). In addition to time spent traveling, the proactive strategy required study staff to spend time at the event recruiting, screening, and enrolling participants; whereas, reactive strategies involved those previously identified as being eligible to travel to the study staff and then participation was immediate. Also, we spent time obtaining permission from IRB and site coordinators in order to attend each of the proactive sites. Thus, though proactive strategies were more effective in obtaining participant numbers, reactive recruitment strategies required less time and costs per participant (especially if this had been a multi-site study). Researchers should consider this tradeoff in the planning phase of a study. We found using both reactive and proactive strategies were feasible given our study’s budget and resources, and resulted in reaching the sample size goal of 27 per user group ahead of schedule.
The objective of this study was to recruit tobacco users of AI descent. Some, but not all, tribes require a specific degree of Indian blood, referred to as blood quantum, to enroll as a tribal member (16). Degree of Indian blood is computed from lineal ancestors of Indian blood who were enrolled with a federally-recognized Indian tribe or whose names appear on the designated base rolls of a federally-recognized Indian tribe (17). We did not require a specific blood quantum for our study, nor did we require documentation of tribal enrollment or blood quantum. Rather, to qualify for our study, individuals self-reported AI race and having at least two biological grandparents of AI race. We chose these questions since they are perceived as less invasive than ascertaining blood quantum in the eligibility screener. Moreover, these questions had been previously used (13).
We were interested in whether participants were willing to report their blood quantum level and if responses to the question assessing number of AI grandparents predicted blood quantum level in our sample. We observed a significant, positive association where the proportion at least half AI blood quantum was 32%, 33% and 70% among participants with two, three, and four AI biological grandparents, respectively. These findings will be informative for guiding future studies of biologic or genetic nature in the development of survey instruments for the assessment of AI descent.
Conclusion
Minority participation in research is limited and data on the effectiveness of community-based strategies for the recruitment of AIs is lacking. This paper described community-based strategies used to recruit AI tobacco users for a study of nicotine metabolism. Though proactive strategies resulted in two-thirds of our participants, they required more time and money than reactive strategies. Overall, we found both reactive and proactive strategies were feasible given our budget and timeline, and using both resulted in the ability to reach sample size goals ahead of schedule. Lastly, we showed that asking participants to report their number of AI biological grandparents may be a good, non-invasive indicator of their AI blood quantum.
Acknowledgments
Study support was also provided by the Oklahoma Shared Clinical and Translational Resources (OSCTR, U54GM104938). We are deeply thankful for the support of the Southern Plains Tribal Health Board, the IHS IRB, and the many tribal communities which helped make this research a success. We are also thankful for the several vape shops that allowed the study staff to recruit and enroll participants on-site.
Funding
This project titled ‘Nicotine Metabolism in AI Smokers and Electronic Cigarette Users’ was supported by the National Institute on Drug Abuse (project number 1R36DA042208-01).
Footnotes
Conflict of Interest
The authors declare that they have no conflict of interest.
Contributor Information
Dana Mowls Carroll, Department of Psychiatry, University of Minnesota, 717 Delaware St SE, Minneapolis, MN.
Lacy S. Brame, Department of Biostatistics and Epidemiology, College of Public Health, University of Oklahoma Health Sciences Center.
Lancer D. Stephens, Department of Health Promotion Sciences, College of Public Health, University of Oklahoma Health Sciences Center.
Theodore L. Wagener, Oklahoma Tobacco Research Center. Department of Pediatrics, University of Oklahoma Health Sciences Center.
Janis E. Campbell, Department of Biostatistics and Epidemiology, College of Public Health, University of Oklahoma Health Sciences Center.
Laura A. Beebe, Department of Biostatistics and Epidemiology, College of Public Health, University of Oklahoma Health Sciences Center.
References
- 1.NIH Revitalization Act, Public Health Service Act sec. 492B, 42 U.S.C. sec. 289a-(1993)
- 2.Blanton S, Morris DM, Prettyman MG, McCulloch K, Redmond S, Light KE, et al. Lessons learned in participant recruitment and retention: the EXCITE trial. Physical Therapy. 2006;86(11):1520–33. doi: 10.2522/ptj.20060091. [DOI] [PubMed] [Google Scholar]
- 3.George S, Duran N, Norris K. A systematic review of barriers and facilitators to minority research participation among African Americans, Latinos, Asian Americans, and Pacific Islanders. American journal of public health. 2014;104(2):e16–e31. doi: 10.2105/AJPH.2013.301706. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.US Census Bureau. QuickFacts UNITED STATES. Dervived from Population Estimates, American Community Survey, Census of Population and Housing, Current Population Survey, Small Area Health Insurance Estimates, Small Area Income and Poverty Estimates, State and County Housing Unit Estimates, County Business Patterns, Nonemployer Statistics, Economic Census, Survey of Business Owners, Building Permits. 2015 [cited 2016 October 28]. Available from: https://www.census.gov/quickfacts/table/PST045215/00.
- 5.US Department of Health Human Services. The health consequences of smoking—50 years of progress A report of the Surgeon General. 2014 [Google Scholar]
- 6.Espey DK, Wu XC, Swan J, Wiggins C, Jim MA, Ward E, et al. Annual report to the nation on the status of cancer, 1975–2004, featuring cancer in American Indians and Alaska Natives. Cancer. 2007;110(10):2119–52. doi: 10.1002/cncr.23044. [DOI] [PubMed] [Google Scholar]
- 7.Benowitz NL, Hukkanen J, Jacob P., 3rd Nicotine chemistry, metabolism, kinetics and biomarkers. Handbook of experimental pharmacology. 2009;(192):29–60. doi: 10.1007/978-3-540-69248-5_2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Satter DE, Veiga-Ermert A, Burhansstipanov L, Pena L, Restivo T. Communicating respectfully with American Indian and Alaska natives: Lessons from the California health interview survey. Journal of Cancer Education. 2005;20(1):49–51. doi: 10.1207/s15430154jce2001_14. [DOI] [PubMed] [Google Scholar]
- 9.Burhansstipanov L, Schumacher SCSA. Lessons learned from community-based participatory research in Indian country. Cancer control: journal of the Moffitt Cancer Center. 2005;12(Suppl 2):70. doi: 10.1177/1073274805012004s10. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Stoddart ML, Jarvis B, Blake B, Fabsitz RR. Recruitment of American Indians in epidemiologic research: The strong heart study. American Indian and Alaska native mental health research (Online) 2000;9(3):20. doi: 10.5820/aian.0903.2000.20. [DOI] [PubMed] [Google Scholar]
- 11.Petereit DG, Burhansstipanov L. Establishing trusting partnerships for successful recruitment of American Indians to clinical trials. Cancer control: journal of the Moffitt Cancer Center. 2008;15(3):260. doi: 10.1177/107327480801500310. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Jacobson SF, Booton-Hiser D, Moore JH, Edwards KA, Pryor S, Campbell JM. Diabetes research in an American Indian community. Image: the Journal of Nursing Scholarship. 1998;30(2):161–5. doi: 10.1111/j.1547-5069.1998.tb01273.x. [DOI] [PubMed] [Google Scholar]
- 13.Renner CC, Lanier AP, Lindgren B, Jensen J, Patten CA, Parascandola M, et al. Tobacco use among southwestern Alaska Native people. Nicotine & tobacco research: official journal of the Society for Research on Nicotine and Tobacco. 2013;15(2):401–6. doi: 10.1093/ntr/nts137. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.PowWows.com LLC. Welcome to Pow Wows Calendar. 2016 [cited 2016 October] Available from: http://calendar.powwows.com/
- 15.SAS Institute Inc. version 9.4 Cary. NC, USA: [Google Scholar]
- 16.U.S. Department of the Interior, Office of Public Affairs-Indian Affairs. A Guide to Tracing American Indian & Alaska Native Ancestry. 2016 [cited 2016 October 24]. Available from: http://www.bia.gov/cs/groups/xois/documents/text/idc-002619.pdf.
- 17.Bureau of Indian Affairs. CERTIFICATE OF DEGREE OF INDIAN OR ALASKA NATIVE BLOOD INSTRUCTIONS. 2017 [cited 2016 October 24]. Available from: http://www.bia.gov/cs/groups/xraca/documents/text/idc1-029262.pdf.