Protocol for the validation of sensitivity and specificity of the Cow’s Milk-related Symptom Score (CoMiSS) against open food challenge in a single-blinded, prospective, multicentre trial in infants

Abstract

Introduction

The symptoms of cow’s milk protein allergy (CMPA) in infancy can be non-specific which may delay a correct diagnosis and cause adverse clinical outcomes. The diagnosis of non-IgE-mediated CMPA is particularly complex as it involves a 2 to 4 week elimination diet followed by oral food challenge (OFC). The Cow’s Milk-related Symptom Score (CoMiSS) is a clinical resource for primary healthcare providers which aims to increase awareness of CMPA symptoms to facilitate an earlier diagnosis. The aim of the present study is to assess if the CoMiSS can be used as a potential diagnostic tool in infants with suspected CMPA.

Methods and analysis

Exclusively formula-fed infants aged 0–6 months presenting with symptoms suggestive of CMPA will be included in this prospective, multicentre trial which will be conducted in 10 centres in China. All infants will commence a 2-week trial of an amino acid-based formula (AAF) while eliminating all cow milk protein from their diets. After the AAF treatment period, infants will undergo an open OFC in hospital with standard cow’s milk formula, followed by an open home challenge for another 2 weeks. Clinical symptoms will be documented on standardised symptom scorecards. The CoMiSS will be determined at study entry (CoMiSS 1, before the start of the AAF), after 2 weeks (CoMiSS 2, before the OFC) and after a further period of 2 weeks or when symptoms suggestive of CMPA reappear (CoMiSS 3). Weight and length will be measured at each visit. The difference between CoMiSS 1 and 2 as a predictor of the OFC outcome will also be assessed. The diagnostic accuracy of the baseline CoMiSS will be calculated.

Ethics and dissemination

The study was approved by the Hunan Children’s Hospital Medical Ethics Committee, Hunan, China. The findings of this trial will be submitted for publication in a peer-reviewed journal in paediatric nutrition or gastroenterology. Abstracts will be submitted to the relevant national and international conferences.

Trial registration number

NCT03004729; Pre-results.

Strengths and limitations of this study.

• The CoMiSS was developed by an expert panel as an awareness tool for cow’s milk protein allergy (CMPA) in infants and is based on a panel of common allergic symptoms.

• The findings of this trial are likely to increase the awareness for cow’s milk-related symptoms and may facilitate an earlier diagnosis of CMPA in infants in the primary care setting.

• A limitation of the study is that the CoMiSS will be validated against open oral food challenges rather than double-blind, placebo-controlled food challenges.

• In addition, the study design may not allow to reliably distinguish between IgE-mediated CMPA, non-IgE-mediated CMPA and non-allergic cow’s milk-related symptoms.

Introduction

Cow’s milk protein allergy (CMPA) is one of the major food allergies in infants with an estimated prevalence of 2%–5%.¹ The diagnosis of CMPA remains a clinical challenge as many symptoms are non-specific.^{1 2} Non-IgE mediated CMPA is particularly difficult to diagnose as the time between contact with the offending protein and onset of allergic symptoms is delayed, while IgE-based testing with cow’s milk-specific IgE or skin prick testing is usually negative.³ This often leads to misdiagnosis or a delay in obtaining the correct diagnosis and appropriate management.⁴

The Cow’s Milk-related Symptom Score (CoMiSS; figure 1) was developed to increase awareness of mainly non-IgE mediated CMPA.⁵ Expert clinicians experienced in managing children with gastrointestinal (GI) problems and/or atopic diseases reviewed the literature to determine whether a clinical score derived from symptoms associated with the ingestion of cow’s milk protein (CMP) could increase the awareness of primary healthcare providers for cow’s milk-related symptoms. The CoMiSS tool provides a score that considers general allergic manifestations, as well as dermatological, GI and respiratory symptoms (total score range 0–33). Some allergic symptoms, such as vomiting, rectal bleeding or failure to thrive, are not included in the score. The score was conceived as an awareness tool for cow milk allergy (CMA). The CoMiSS may also be used to monitor the evolution of symptoms in response to a therapeutic intervention.⁵ At present, the recommended diagnostic approach for CMPA relies on a 2 to 4 week elimination diet followed by an oral food challenge (OFC).³ While the double-blind, placebo-controlled food challenge (DBPCFC) is the gold standard for the diagnosis of food allergy, in clinical practice open challenges are generally considered sufficient, particularly in infants and young children.^{3 6} Parents are often reluctant to proceed with a food challenge as CMPA symptoms may recur during a positive OFC.⁷ A validated score which could replace or reduce the need for food challenges would therefore be of great clinical value.

Figure 1.

CoMiSS. All parameters included in the CoMiSS (evaluation of regurgitation: see ref 11; evaluation of stool composition: see ref 12). CoMISS, Cow’s Milk-related Symptom Score.

Methods and analysis

This single-blinded, prospective, multicentre study will be conducted in formula-fed infants with symptoms suggestive of CMPA. The trial design is based on current best clinical practice, that is, a cow’s milk protein-free elimination diet followed by OFC.³ All dietary interventions of the study are open label, while investigators involved in the OFC will be blinded to the response to amino acid-based formula and CoMiSS results. Thus, the study remains single-blinded for the comparison of the OFC and CoMiSS results. The CoMiSS will be determined by an investigator (not involved with the OFC) at study entry, after 2 weeks of an elimination diet (before the OFC) and 2 weeks after the reintroduction of cow’s milk or whenever symptoms reoccur during the open-home challenge.

Study objectives

The primary objective of this trial is to validate the change in CoMiSS from baseline to week 2 (Δ=CoMiSS 1–CoMiSS 2) against the reference standard, the OFC to cow’s milk formula as a predictor of CMPA.

Secondary objectives are:

1. To determine the sensitivity and specificity of the change in CoMiSS from baseline to week 2, based on the optimal cut-off on the receiver operating characteristic (ROC) curve.

2. To determine if the CoMiSS 1 (baseline) is predictive of OFC outcome.

3. To compare the visit 2 CoMiSS following exclusive treatment with amino acid formula (AAF) with historical CoMiSS data sets of healthy infants.

4. To determine if CoMiSS decreases over time in subjects on an AAF diet.

5. To determine if a high CoMiSS at baseline predicts a greater absolute decrease in CoMiSS following an AAF diet (change from baseline to visit 2)

6. To analyse the symptom components of CoMiSS (crying, regurgitation, stool consistency, eczema, urticaria and respiratory symptoms).

7. To assess the impact of a family history of atopy on baseline CoMiSS and the percentage of subjects with confirmed CMPA.

8. To assess the impact of delivery mode (vaginal vs Caesarean section) on baseline CoMiSS and the percentage of subjects with confirmed CMPA.

9. To monitor the subjects’ growth.

10. To evaluate the therapeutic effect of an elimination diet.

11. To measure compliance with an elimination diet.

Clinical setting and study population

The recruitment of the study subjects will take place in 10 paediatric departments throughout China (for list of clinical sites please refer to Acknowledgements section). Exclusively formula-fed infants under 6 months of age with symptoms suggestive of CMPA are eligible for inclusion in the study. Patients will be consecutively enrolled according to the inclusion and exclusion criteria, without regard to the severity of symptoms (table 1; figure 2). Symptoms need to have been present for at least 1 week and have developed within the first 2 months of commencing an infant formula containing intact CMP. Infants considered for inclusion should, in the opinion of the investigator, benefit from a 2-week elimination diet with an AAF, followed by an OFC.

Table 1.

Schedule of events

Assessments and visits	Screening visit	Visit 1 baseline	Elimination diet with AAF	Visit 2	Home OFC (subjects with no reaction to CMP at visit 2)	Visit 3 completion	Early withdrawal visit
	Day 0 (or maximum 2 days before day 0)	Day 0	Day 1 to day 14	Day 15 ±4 days	Day 15 until reaction; or day 15 to day 28 if no reaction	Day 29±4 days (or within 4 days following reaction)	0 to 4 days after stopping study procedures
Eligibility/Informed Consent	X
Medical history, including subject’s results for allergen-specific IgE and skin prick test (if available)	X
Document delivery mode	X
Document demographic data		X
Infant length and head circumference		X
Infant weight		X		X		X	X
Investigator VAS score		X		X
CoMiSS (including physical examination; performed by designated site staff member)		X		X		X	X
Two-week elimination diet with Alfamino formula		X
Provide daily subject intake record		X		X*
In-hospital open OFC (including physical examination)				X
Two-week home OFC with NAN formula				X*
Prescribe and dispense Alfamino formula				X†		X‡
Prescribe and dispense cow’s milk formula						X§
Concomitant medications	X	X		X		X	X
Adverse events (including SAEs)		X		X		X	X
Collect subject intake record				X		X	X
Record allergy status (physical examination)						X

*For subjects with no immediate reaction to cow’s milk formula at visit 2.

†For subjects with reaction to cow’s milk at visit 2. These subjects should be commenced on Alfamino formula (with follow-up by treating medical team).

‡For subjects with reaction to cow’s milk at visit 3. These subjects should be commenced on Alfamino formula (with follow-up by treating medical team).

§For subjects with no reaction to cow’s milk formula at visit 3.

AAF, amino acid-based formula; CMP, cow’s milk protein; CoMISS, CoMISS, Cow’s Milk-related Symptom Score; OFC, oral food challenge; SAEs, serious adverse events; VAS, Visual Analogue Scale.

Figure 2.

Study design. CMPA, cow’s milk protein allergy; CoMISS, Cow’s Milk-related Symptom Score.

Inclusion criteria for participation in the study defined

1. Male or female infants aged up to 6 months.

2. Infant exclusively fed with cow’s milk-based formula for at least 1 week.

3. Symptoms suggestive of CMPA have been present for at least 1 week and have developed within the first 2 months of starting cow’s milk-based formula.

4. Infant likely to benefit from a 2-week trial of AAF (as per investigator’s assessment).

5. Gestational age > 37 to<42 weeks.

6. Birth weight between 2500 g and 4500 g.

7. Informed consent signed by either the parent or the legal guardian.

Exclusion criteria defined

1. Subject has received extensively hydrolysed or amino acid-based formula prior to enrolment.

2. Fever>38.5°C at time of enrolment.

3. History of serious allergic reaction, suggestive of anaphylaxis (medically diagnosed).

4. Use of antibiotics at enrolment (may participate if antibiotics discontinued for at least 7 days prior to enrolment).

5. Infant with medical condition or family situation which would preclude participation in the study (according to assessment of the investigator).

6. Infant or infant’s parent unable to comply with trial procedures.

7. Participation in another clinical trial within 4 weeks prior to enrolment.

8. Inability to obtain informed written consent from parent or legal guardian.

Interventions

All study materials, including the CoMiSS tool were translated into Chinese language and approved by the local investigator team. At study entry, the investigator will determine the baseline CoMiSS (CoMiSS 1). Allergy testing (total IgE, milk-specific IgE and skin prick testing) may be performed, as clinically indicated, but is not a protocol requirement. The study consists of the following phases: a screening visit (visit 0), a baseline visit (visit 1) and a 2-week elimination diet during which patients exclusively receive AAF. A study visit (visit 2) will take place at the end of these 2 weeks. At visit 2, the investigator will assess if the elimination diet was clinically effective, or not, and CoMiSS 2 will be determined. This will be followed by a hospital-based, open OFC using a cow’s milk-based infant formula with intact CMP. Subjects who do not react during the hospital-based challenge will continue the OFC at home for up to 2 weeks (visit 3). Subjects who react to CMP during the home-based OFC will stop the OFC before 2 weeks and will return to the study centre for visit 3. Infants reacting during the OFC will be examined by the investigator, either when the parents make contact or, at the latest, after 2 weeks. The investigator will determine the CoMiSS (CoMiSS 3) at visit 3. Subjects with a positive OFC during the hospital-based challenge will complete visit 3 assessments on the same day as visit 2.

Elimination diet trial with AAF

The 2-week elimination diet will be performed using an AAF (Alfamino, Nestlé SA, Vevey, Switzerland). Alfamino is a hypoallergenic, amino-acid based, nutritionally complete powdered specialty infant formula containing amino acids, carbohydrates, fats, vitamins, minerals, trace elements and the long-chain polyunsaturated fatty acids, arachidonic acid and docosahexaenoic acid. The planned duration of the elimination diet is based on published guidelines on the diagnosis and treatment of CMPA.^{3 8} Formula will be prepared by the infant’s caregiver according to the standard instructions provided on the tin.

OFC

The open OFC will be conducted in hospital by a medically qualified study investigator. All investigators will be trained in the food challenge procedure, including watching a standardised training video in Chinese language. The doctor supervising the OFC is blinded to the CoMiSS scores. The cow’s milk-based infant formula NAN1 (Nestlé SA, Vevey, Switzerland) will be administered during the OFC. The OFC involves a typical dose escalation while observing for clinical symptoms. The OFC will be performed in hospital on day 1 and continued at home for a full week, as tolerated. The following doses will be administered during of the OFC: initial test dose of one drop on the lip of the infant; if no reaction is observed after 15 min, 0.5 mL are given orally. If no reaction is observed after 30 min, the following oral doses are administered at 30 min intervals, as tolerated: 1 mL, 3 mL, 10 mL, 30 mL, 50 mL and 100 mL (maximum cumulative dose 194.5 mL). Symptoms during OFC will be carefully documented. The OFC will be categorised as ‘positive’ or ‘negative’ according to symptoms and the assessment by the investigator. The OFC is considered positive in the presence of immediate symptoms (vomiting, hives, facial angioedema, wheeze, stridor) during day 1 in hospital or delayed-onset symptoms (vomiting, increased regurgitation, persistent diarrhoea, increased eczema, irritability/persistent crying) during the remaining week. In case of a positive OFC, infants will remain on AAF and will be followed up by their treating clinician.

Discontinuing study interventions and patient withdrawal

Subjects may discontinue the study intervention prematurely for the following reasons: (1) the subject’s parents or legal guardian request to leave the trial; (2) the subject is lost to follow-up; (3) in the investigator’s opinion, continuation in the trial would be detrimental to the subject’s well-being; (4) a major protocol violation occurred (in case of a minor protocol violation, the sponsor’s clinical project manager will evaluate with the investigator, if the subject has to be withdrawn); (5) a prohibited diet/treatment/medication defined in the protocol is used during the study (in the event a subject is found to be taking such a diet/treatment/medication during the trial, the site should immediately contact the clinical project manager; the decision to withdraw the subject will be made in conjunction with the investigator); (6) the sponsor or Independent Ethics Committee decides to terminate trial.

If a subject discontinues the intervention or chooses to discontinue the intervention prematurely, an early withdrawal visit will be completed without delay but no later than 4 days after discontinuation. The reason(s) for premature discontinuation will be documented accordingly. Every effort will be made to follow-up subjects who terminate prematurely to determine the final clinical outcome.

Patient and public involvement statement

No parents or patient advocate groups were involved in the planning of this study. Patients were recruited via the investigators at medical clinics in China. Once the study results have been published in a peer-reviewed journal, a summary of the main study findings will be distributed in Chinese language via the local investigator teams to the participating families.

Statistical analysis

The main objective of the trial is to validate the CoMiSS (index test) as a predictor of CMPA, as confirmed by a positive OFC (reference test). The ROC curve for the CoMiSS will be evaluated based on this prediction and the area under the curve (AUC) of the ROC curve be calculated. The ROC curve is the curve of true positive rate (sensitivity) versus false positive rate (1-specificity) for different thresholds of the CoMiSS (change from baseline). The hypothesis to be tested is:

$${\displaystyle H_0:AUC=0.75vs.H_1:AUC>0.75}$$

The CoMiSS instrument scores range from 0 to 33. An AUC >0.75 corresponds to a 1-point ROC curve with 90% sensitivity and 60% specificity. Based on this threshold, we aim to identify a meaningful CoMiSS cut-off that would provide a high sensitivity (around 80%–90%) to pick-up all infants suspected of CMPA and minimise the number of infants that have a CoMiSS below the proposed threshold and would have a positive OFC and moderate specificity in the range of 60%–70% for predicting a positive OFC (ie, diagnosis of CMPA).

All statistical analysis will be performed using the statistical software package R V.3.2.2. Statistical significance will be tested at the two-sided 5% level, unless otherwise specified. Continuous safety and effectiveness parameters will be summarised by presenting the number of patients, mean, SD, median, minimum, maximum by formula group (test or control). Tabulation of categorical parameters by formula group will include counts and percentages;95% CIs will be provided, as appropriate.

Randomised analysis plan

For the validation process, the study subjects will be randomised 1:1 into a training set and a holdout set. A logistic regression model will be fitted to the training set (50% of the overall sample), regressing the odds of having a positive OFC on the change in CoMiSS from baseline to week 2. This model will then be used to predict the CMPA status (positive or negative challenge test) of the subjects in the holdout set based on their change in CoMiSS and a ROC curve will be derived based on these predictions.

The following effects will be estimated:

• AUC of the ROC with 95% CI.

• Sensitivity, specificity, negative and positive predictive values, as well as negative and positive likelihood ratios with 95% CIs for the optimal threshold of the change in CoMiSS.

Different logistic regression models based on using different CoMiSS change cut-offs will be tested within the training set using cross-validation methods, adjusting covariates to choose a model with the highest predictive accuracy (using cross-validation) and with the best goodness of fit. The selection of the best predictive model based on the cross-validation will provide the basis for predicting the OFC outcomes in the test infants. The prediction will be obtained as the probability (between 0 and 1) of a positive OFC after 2 weeks. Using different thresholds on this probability, several 2×2 contingency tables will be obtained. For each 2×2 table, the sensitivity (true positive) is defined as the proportion of infants positive on the OFC who were also predicted as positive using the particular cut-off on the predicted probabilities. Specificity (true negative) is defined as the proportion of infants negative on the OFC who were also predicted as negative using the particular probability cut-off. In addition to specificity and sensitivity, positive and negative predictive values, as well as positive and negative likelihood ratios will be calculated. The AUC of each ROC will be used to determine the overall accuracy of the prediction and to choose an optimal probability threshold for the use of CoMiSS in clinical practice.

Sample size calculation

The hypothesis of interest is that the AUC > ${\displaystyle A_0}$ (=0.75) and is tested using the test-statistic

$${\displaystyle Z=\frac{\hat{A}-A_0}{\sqrt{Va{r}_0(AUC)}}}$$

where ${\displaystyle \hat{A}}$ is the estimated AUC on the holdout set, ${\displaystyle A_0=0.75}$ is the benchmark or minimum clinically relevant AUC of the ROC curve and ${\displaystyle Va{r}_0(A)}$ is the variance of the AUC under the null hypothesis ${\displaystyle H_0}$. The variance of the AUC at a given value ${\displaystyle A}$ is given by Hajian-Tilaki⁹

$${\displaystyle Va{r}_A(AUC)=0.0099e^{-\frac{a^2}{2}}\left(\frac{5a^2+8}{n_d}+\frac{a^2+8}{n_h}\right),}$$

where ${\displaystyle a=\sqrt{(2)}{\mathrm{\Phi }}^{-1}(A)}$ ${\displaystyle \mathrm{\Phi }}$ is the cumulative distribution function of the standard normal distribution ${\displaystyle n_{-}}$ and ${\displaystyle n_{+}}$ and are the number of subjects in the holdout set who test negative and positive in the OFC, respectively.

The corresponding sample size for the required number of subjects who would test positive on the OFC test when the assumed AUC is A is given by:

$${\displaystyle n_{+}=\frac{{\left[z_{\alpha /2}\sqrt{V_0(AUC)}+z_{\beta }\sqrt{V_A(AUC)}\right]}^2}{(A-A_0{)}^2}}$$

where ${\displaystyle V_A(AUC)=n_{+}Va{r}_A(AUC)}$.

The pooled prevalence from three recent CoMiSS trials of children with a positive OFC, that is, confirmed CMPA was 83%.¹⁰ Entry criterion for these studies was a CoMiSS ≥12. For the present study, patients will be recruited based on symptoms and the estimated prevalence of CMPA in the study population will therefore lower (estimated prevalence of challenge-proven CMPA 70%). Based on the above asymptotic sample size formula and prevalence estimate for CMPA, around 80 subjects with a positive OFC and 35 subjects with a negative OFC will be required in the holdout set to test the above hypothesis regarding the AUC with 90% power and at 5% level of significance and 2.5% for one-sided testing. Thus, the overall sample size accommodating the training and holdout set splits is N=2×115=230. Assuming a 15–20% dropout rate, 300 subjects will need to be enrolled.

Interim analysis

An interim analysis will is planned after 146 subjects have completed their OFC. At this point, there will be data from 73 subjects in the training set and 73 subjects in the holdout set. The interim analysis will be carried out in an unblinded fashion by an independent statistical centre and the interim decision will be taken by an independent data monitoring committee (iDMC) in a closed session. The sponsor will remain blinded to the interim data and will only be informed about the iDMC decision resulting from the interim analysis.

Based on an interim analysis at, the following decisions may be made:

1. Stopping for futility if the AUC is less than the desired 0.75 benchmark.

2. Stopping for efficacy if the AUC is greater than 0.9.

3. Continuing the trial, as planned, if the AUC at the interim is close to 0.9.

4. Increase the sample size by a required amount (see below) if the interim results are promising but observed prevalence, actual AUC or dropout rate at the interim are slightly different than the corresponding assumed design values.

Datasets to be analysed

The intent-to-treat (ITT) population is defined as all randomised subjects who have attempted the OFC. Efficacy and safety analyses will be based on the ITT population. The per-protocol (PP) population is defined as all subjects who complete the study with no major protocol violations. Subjects who ingest CMP during the elimination diet and those who require treatment with antibiotics, antipyretics or corticosteroids during the study will be removed from the PP population but will be included in the full-analysis population. Every attempt will be made to collect final study data from subjects who withdraw before completing the study. No imputation for missing data is planned for this study.

Ethics and dissemination

A written informed consent form will be signed by a parent or legal guardian before the study enrolment. Any modifications to the protocol, which may affect the conduct of the study, potential benefits to the patients or patient safety, including changes to the study design, will be reported to the ethics committee for all necessary amendments. All study-related information will be stored securely in the study sites in locked cabinets, in an area with limited access (databases will be secured with a password-protected access system). The findings of this validation trial will be submitted to a peer-reviewed journal (paediatric, allergy, nutrition and/or gastroenterology). All Chinese investigators will be involved in the development of the final manuscript. Abstracts will be submitted to relevant national and international conferences.

Study sponsorship and oversight

The Contract Research Organisation, George Clinical, Sydney, Australia will be overseeing the overall conduct of the study, including adverse event monitoring and data management. Adverse events will be reported in the patient file. They will be assessed and managed according to good clinical practice. Adverse events are part of the secondary outcomes and will be listed in the final study report. The independent statistical analysis will be performed by Cytel, Cambridge, Massachusetts, USA.

Timelines

The study started recruitment in January 2017, and completion of data collection is anticipated by June 2018.