Development and validation of a product acceptability questionnaire for intranasal Q-Griffithsin COVID-19 prophylaxis (SPRAY PAL)

Abstract

Objectives

Patient experiences are critical when determining the acceptability of novel interventional pharmaceuticals. Here, we report the development and validation of a product acceptability questionnaire (SPRAY PAL) assessing feasibility, acceptability and tolerability of an intranasal Q-Griffithsin (Q-GRFT) drug product designed for COVID-19 prophylaxis.

Design

SPRAY PAL validation was undertaken as part of an ongoing phase 1 clinical trial designed to test the safety, pharmacokinetics and tolerability of intranasally administered Q-GRFT for the prevention of SARS-CoV-2 infection.

Setting

The phase 1 clinical trial took place at a University Outpatient Clinical Trials Unit from November 2021 to September 2023.

Participants

The initial SPRAY PAL questionnaire was piloted among healthy volunteers ages 25 to 55 in phase 1a of the clinical trial (N=18) and revised for administration in phase 1b for participants ages 24–59 (N=22).

Results

Spearman correlations tested convergent and discriminant validity. Internal consistency was assessed using Cronbach’s alpha, and test–retest reliability was assessed using intraclass correlation coefficients of responses collected from three repeated questionnaire administrations. The initial version demonstrated excellent internal consistency. The revised version demonstrated very good internal consistency after removal of one item (alpha=0.739). Excellent test–retest reliability (intraclass coefficient=0.927) and adequate convergent (r’s=0.208–0.774) and discriminant (r’s=0.123–0.392) validity were achieved. Subscales adequately distinguished between the constructs of acceptability, feasibility and tolerability.

Conclusions

The SPRAY PAL product acceptability questionnaire is a valid and reliable patient-reported outcomes measure that can be considered a credible tool for assessing patient-reported information about product acceptability, feasibility of use, tolerability of product and side effects and cost of product for novel intranasal drug formulations. The SPRAY PAL is generalisable, and items may be readily adapted to assess other intranasal formulations.

Trial registration numbers

NCT05122260 and NCT05437029.

STRENGTHS AND LIMITATIONS OF THIS STUDY.

• We examined the reliability and validity of a novel questionnaire designed to assess acceptability, feasibility and tolerability of a novel intranasal spray formulation.

• The questionnaire can be readily adapted and generalisable for use with other intranasal formulations.

• The study is limited by the small sample size, precluding a more sophisticated principal components analysis, and relatively short period of follow-up in which to assess retest reliability.

Introduction

Over the past two decades, three coronaviruses of the Betacoronavirus genus have emerged as serious human pathogens, with the COVID-19 pandemic causing over 700 million infections globally¹ and over 1 million deaths to date in the USA.²

The virus that causes COVID-19, SARS-CoV-2, replicates efficiently in the upper respiratory tract—the nasopharynx and oropharynx.³ High viral replication in the nasopharynx in the early stages of infection, prior to symptom onset, accounts for the high transmissibility of SARS-CoV-2. Respiratory aerosols and droplets are the most frequent sources of human transmission events.^{4 5} Consequently, the development of an intranasal spray that prevents the establishment of infection is an effective strategy to curb virus spread. This strategy will be synergistic to vaccine approaches and biomedical interventions, such as personal protective equipment and measures like social distancing and frequent hand washing, in eliminating the pandemic.

Due to the limited long-term durability of antibody response to vaccines, and the requirement of booster doses to maintain effective immunity to SARS-CoV-2,^{6 7} an additional level of protection of the kind likely to be offered by an intranasal spray product is critical in infection prevention. Topical delivery of drugs by the nasal route is cost-effective and eliminates or reduces potential drug–drug interactions.^{8 9} Additionally, it is a convenient, easy-to-use approach and is a widely accepted method of drug administration for a variety of patients,^{9 10} especially for prolonged daily dosing periods.

As such, the PREVENT-CoV (PRe-Exposure prevention of Viral ENTry of CoronaViruses) study was designed based on the potential utility of the intranasal drug delivery approach as a technology to prevent the establishment of upper respiratory infection. This is the first-in-human intranasal application of Q-Griffithsin (Q-GRFT), an oxidation-resistant variant of GRFT, a lectin initially extracted from red sea algae.^{11 12} The PREVENT-CoV phase 1 clinical trial evaluated the safety, tolerability and pharmacokinetics of the novel intranasal spray in healthy male and female volunteers, as the primary endpoint. Secondary endpoints included user perceptions, acceptability and the impact of product use on participants’ olfactory sensation and quality of life.¹² The phase 1 clinical trial is ongoing to collect a final assessment of the levels of antidrug antibodies 1 year after final dose administration.

Compliance with intranasal formulations is key to effectiveness, and this depends largely on patient preference, as seen in prior work on intranasal corticosteroid formulations.^{13 14} Daily use of intranasal formulations may be impacted by product sensory attributes, such as smell and aftertaste, intranasal sensations of the product, as well as ease of product use and cost.¹⁵ Questionnaires are often used to assess these product features. However, there is no readily available instrument assessing the acceptability, feasibility and tolerability of an intranasal formulation. This prompted the development of the product acceptability questionnaire, SPRAY PAL. Here, our objective is to report on the development and reliability, defined by psychometric properties, of a novel questionnaire measuring key components of key intranasal product features.

Methods

Study design

This study consisted of two separate phases of a randomised, single-site trial (ClinicalTrials.gov identifiers NCT05122260 and NCT05437029). Details regarding trial design, drug product, and participant eligibility, recruitment and informed consent have been previously reported.¹² Briefly, participants were prescreened using online questionnaires and telephone interviews to determine eligibility. Selected volunteers were invited for a screening visit at the clinical trials unit where eligibility was confirmed and written informed consent was obtained. Participants were generally healthy, aged 16–85, screened negative for SARS-CoV-2, able to attend all study visits, participating in no other concurrent drug trials, not pregnant or breastfeeding and/or were using contraception. Individuals with acute or chronic upper respiratory or pulmonary issues/illnesses, smokers, recreational drug users and those taking intranasal medications or systemic steroids were excluded. Participants retained their right to withdraw from the study at any time for any reason.

The phase 1a study (SAMPLE 1) was performed in a double-blind fashion, with 18 participants randomly assigned 2:1 to either the study product arm or the placebo arm after stratification by race and gender. After participants received either a single dose of study product or a single dose of placebo, follow-up assessments were performed at 1 hour, 6 hours, 24 hours (visit 2) and 72 hours (visit 3) post initial dose administration. A follow-up safety review was completed by phone approximately 2 weeks later (visit 4). The SPRAY PAL was administered at visits 2, 3 and 4.

The phase 1b study (SAMPLE 2) was an open-label design conducted in two separate groups stratified by race and gender. Group 1 participants administered the study product once daily for 7 days and were evaluated at multiple visits over the subsequent 9 days. The SPRAY PAL was administered at visit 3 (midway through study product administration; study day 4), visit 4 (the final day of product administration; study day 7), and visit 6 (48 hours following the final dose; study day 9). One participant withdrew from the study due to contracting COVID-19 and completed the SPRAY PAL at an early termination visit after having received one dose of the study product.

Group 2 participants administered the study product twice daily, approximately every 12 hours, for 7 days and were evaluated over the subsequent 9 days. The SPRAY PAL was administered at visit 4 (midway through study product administration; study day 5), visit 5 (the final day of product administration; study day 8) and visit 7 (48 hours following the final dose; study day 10).

A 1-year follow-up assessment of antidrug antibodies in both groups is ongoing.

Measure: product acceptability questionnaire

Participants evaluated product acceptability, feasibility and tolerability. Because there was no readily available questionnaire assessing these aspects for existing intranasal formulations, questionnaire items were derived from existing, validated questionnaire items with adaptation for the current study.¹⁶ Participant experience and opinion of efficacy, sensory perceptions, spray characteristics, administration process, applicator design and use regimen were assessed. Items are rated on 5-point Likert scales coded from 1 to 5 (most negative to most positive), with an option of ‘prefer not to answer’ included on each item to allow participants the opportunity to opt out of a question if desired. The SPRAY PAL also included open-ended items to allow participants to comment on other characteristics of the nasal spray not assessed by the questionnaire, and to allow comment on the questionnaire items themselves. The subscale and total scale scores are calculated by summing all items in each subscale and all questionnaire (including cost) items, respectively.

Analyses

Responses were collected from participants on paper forms and were double-entered into a REDCap database hosted at the University of Louisville.^{17 18} Entries were compared and, when mismatches occurred, data accuracy was confirmed against paper records. In SAMPLE 2 group 2, one participant skipped an item about the comparability of the spray to the COVID vaccine on each administration of the product acceptability questionnaire. The mean score of all other items from that subscale for that participant was imputed to replace the three missing responses. Otherwise, all SPRAY PAL items were answered completely. Item responses for all participants were summarised using descriptive statistics.

Item revision

Open-ended responses from participants in SAMPLE 1 were reviewed to assess for any participant comments on questionnaire item construction. SPRAY PAL items were also discussed with SAMPLE 1 participants who voluntarily provided feedback. The suggestions were incorporated, and a revised questionnaire was employed with SAMPLE 2.

Group comparisons

Statistical comparisons of demographic data between SAMPLES 1 and 2 were performed using independent samples t-tests and Fisher’s exact tests. SPRAY PAL summary scores between SAMPLE 2 group 1 and group 2 were compared using independent samples t-tests.

Reliability and validity tests

Internal consistency was assessed using Cronbach’s coefficient based on responses from the first administration of the SPRAY PAL for each SAMPLE. Test–retest reliability was assessed by calculating the intraclass correlation coefficients of responses collected three times over a span of 5 (SAMPLE 2) to 12 (SAMPLE 1) days during study participation; at least 48 hours had elapsed between each administration of the SPRAY PAL. We assessed the Spearman correlation of each item with its own scale (with the overlapping item removed) to determine convergent validity, and the Spearman correlation of each item with other scales to assess discriminant validity. All analyses were conducted using SPSS V.27 with alpha set at 0.05 (IBM).

Patient and public involvement

None.

Results

Sample demographics

Sample demographics are provided in table 1. There were no significant differences in demographic characteristics across samples except that SAMPLE 2 had a significantly higher vaccination rate than SAMPLE 1 due to updates made to guidelines for booster shot administration during the data collection period.

Table 1.

Sample demographics and baseline characteristics

	Sample 1 (N=18)	Sample 2 (N=22)	Total (N=40)	P value
	N (%)	N (%)	N (%)
Gender				0.761
Male	8 (44.0)	11 (50.0)	19 (47.5)
Female	10 (66.0)	11 (50.0)	21 (52.5)
Race				0.111
White	6 (33.3)	14 (63.6)	20 (50.0)
African American	0 (0.0)	1 (4.5)	1 (2.5)
Asian	10 (55.6)	5 (22.7)	15 (37.5)
Hispanic	0 (0)	2 (9.1)	2 (5.9)
Mixed race	2 (11.1)	0 (0.0)	2 (5.0)
Fully vaccinated with booster	6 (33.3)	20 (90.9)	26 (65.0)	<0.001
Age, years, M (SD, range)	32.6 (8.1, 25–55)	35.6 (11.8, 23–59)	34.3 (30.5, 23–59)	0.335
BMI, M (SD)	25.5 (3.8)	27.5 (7.6)	26.6 (6.2)	0.315

BMI, body mass index.

Item revision

After administration to participants in SAMPLE 1, who received a single nasal spray administration, internal consistency was calculated for each subscale and the total scale. Internal consistency was above the acceptable range (alpha>0.7) for all subscales and for the total scale, excluding the Acceptability subscale, where Cronbach’s alpha=0.514. Based on feedback from participants in SAMPLE 1, one acceptability item was rephrased from inquiring about whether use of the spray would be acceptable versus not acceptable to inquiring about likelihood of use. Tolerability items were rephrased from how much the participant liked versus disliked a spray characteristic to how much each characteristic encouraged versus discouraged product use.

Internal consistency

Internal consistency (Cronbach’s alpha) was calculated for each subscale and for the total scale score from the initial SPRAY PAL administration for SAMPLE 2. For the feasibility subscale, alpha was initially 0.346. Reliability estimates after individual item removal suggested removal of one item which, when removed, improved Cronbach’s alpha to 0.651 for the acceptability subscale. Alpha was acceptable for all other subscales: 0.618 for the feasibility subscale, 0.789 for the tolerability subscale and 0.739 for the total scale score.

Test–retest reliability

The full SPRAY PAL was administered three times over the course of study participation for the purposes of calculating test–retest reliability. For all responses collected from participants in SAMPLE 2, intraclass coefficients were well above the acceptable threshold (>0.7) at 0.951 for three acceptability subscale scores, 0.888 for the feasibility subscale scores, 0.870 for the tolerability subscale scores, 0.971 for the cost item, and 0.927 for the total scale score.

Convergent validity

No significant differences were noted in subscale scores between SAMPLE 2 groups 1 and 2, so SAMPLE 2 responses were pooled for validity and reliability tests. All but two items correlated highly with their own subscale; an item assessing likelihood of using the spray as many days as needed achieved a small correlation with the remaining items in the feasibility subscale (r=0.040), and an item assessing whether the product ran down the back of the throat achieved a small correlation with the remaining items in the tolerability subscale (r=0.134). Otherwise, items demonstrated convergent validity that was within the accepted range based on a correlation with their own subscale between 0.2 and 0.7 (table 2).

Table 2.

Convergent and discriminant validity

Subscale	# Items	Acceptability	Feasibility	Tolerability
Acceptability	6	0.208–0.630	-0.303–0.132	0.084–0.507
Feasibility	7	-0.375–0.202	0.040–0.576	-0.252–0.311
Tolerability	9	0.060–0.440	-0.171–0.201	0.134–0.774

Correlation coefficients on the diagonal (italicised) represent the range of correlation coefficients obtained for each item with its own subscale after removal of the overlapping item (ie, convergent validity). All other coefficients represent divergent validity. Some negative correlations were obtained due to the varying nature of items (ie, asking about self vs asking about friends/family).

Discriminant validity

In the accessibility subscale, an item comparing effectiveness of the spray to vaccine did not meet criterion for discriminant validity (r>0.4) from the tolerability subscale. Similarly, in the tolerability subscale, an item assessing likeability of the spray bottle itself did not meet the discriminant validity criterion from the acceptability subscale. Some negative correlations were obtained due to the varying nature of items (ie, asking about self vs asking about friends/family). Otherwise, all items correlated more highly with their own subscale score than other subscales, demonstrating good discriminant validity. The correlations between subscale scores ranged from 0.123 to 0.392, indicating adequate distinction between subscale constructs. The final SPRAY PAL questionnaire is provided in online supplemental material.

Discussion

Acceptability is an important consideration for the successful design and implementation of novel pharmaceutical products. Adherence to drug regimen may be greatly impacted by patient acceptance of study product and treatment regimen, including feasibility of use, tolerability of treatment and side effects, and product cost. Our objective was to develop the SPRAY PAL product acceptability questionnaire to provide evidence for all these factors in efforts to better inform the development and commercialisation of a novel intranasal formulation designed for COVID-19 prophylaxis. Item development was based on existing, validated questionnaires, with adjustments made based on qualitative feedback from study participants.

Initial tests of internal consistency indicated that one item, ‘How easy or difficult would it be to carry a spray bottle like the one used in this study around with you if you needed to?’ should be removed to improve Cronbach’s alpha to an acceptable level. This was possibly due to the item asking the participant to speculate about future use, rather than ask about current experiences, in addition to inconsistencies in ratings when compared with other items (eg, participants who rated this item as less feasible rated other items as more feasible). After this, item was removed, we observed adequate indices of internal consistency as well as test–retest reliability on the revised version of the SPRAY PAL.

While tests of convergent and discriminant validity were generally acceptable, there were two items that fell just below conventional thresholds for each construct. This is, in part, related to the diversity of themes across items that fall under the broader theme of each subscale, such as assessments about the nature of physical spray characteristics versus impressions of efficacy, and inquiring about administration for one’s self versus others. However, tests of internal consistency for the full scale did not suggest that removal of any one item would improve the overall alpha score achieved. Taken together with the observation of low correlations between subscales, this suggests that the single full scale sum score may be the most appropriate indicator of overall product acceptability.

Because the SPRAY PAL was implemented as part of a phase 1 clinical trial, the sample size was small, precluding the use of more sophisticated analytic procedures, such as factor analysis, for tests of item validity. Confirmation of item validity should be further tested in a larger, and more diverse, sample of patients. Similarly, assessments of test–retest reliability were designed to fit within the existing study appointments necessary for determining safety and tolerability of the study product. As such, the retest time frame was limited to 12 days. Retest stability over longer treatment periods will need to be addressed in future trials. The SPRAY PAL items were generated with respect to a novel intranasal COVID-19 prophylactic formulation; the generalisability of items to other applications may therefore be limited. Finally, while the SPRAY PAL was created based on a sound conceptual framework and tested using commonly used psychometric methods for validation and reliability assessment of a new questionnaire, it should be employed with caution until the results are confirmed among larger samples and in different clinical settings.

Conclusions

Compliance with intranasal formulations can be impacted by product administration schedules, sensory attributes, ease of use and cost. The lack of a readily available instrument to assess these features in an intranasal formulation has challenged accurate assessment of patient perception. This prompted the development of the SPRAY PAL among a small sample of patients participating in a phase 1 clinical trial. The SPRAY PAL product acceptability questionnaire was found to be psychometrically sound with adequate validity and reliability, though further psychometric validation steps should be performed. It can be considered a credible tool for assessing patient-reported information about product acceptability, feasibility of use, tolerability of product and side effects, and cost of product for novel intranasal drug formulations. The SPRAY PAL is generalisable, and items may be readily adapted to fit modified study designs and different dosing regimens for other nasal spray product formulations as necessary.

Data availability statement

No data are available. The results reported here use data that were collected as part of a phase 1 clinical trial. These data will be made available as part of the data sharing plan that accompanies the forthcoming report of the phase 1 trial outcomes.

Ethics statements

Patient consent for publication

Consent obtained directly from patient(s).

Ethics approval

This study involves human participants and approval to conduct this study was granted by the University of Louisville Institutional Review Board (IRB), (Phase 1a IRB# 21.0704 and Phase 1b IRB# 22.0224). Participants gave informed consent to participate in the study before taking part.