Reporting of participant compliance in randomized controlled trials of nutrition supplements during pregnancy

Abstract

Randomized controlled trials (RCT) are widely considered to be the gold standard for demonstrating intervention effects. Adequacy of reporting of participant compliance in RCTs affects the interpretation of study results. Our aims were two‐fold: first, to assess the adequacy of reporting of participant compliance in RCTs investigating the effect of maternal nutritional supplements on infant outcomes; and second, to examine authors' adherence to the Consolidated Standards of Reporting Trials (CONSORT) guidelines on participant flow. Papers reporting trials of nutritional supplementation during pregnancy, and published after revision of the CONSORT recommendations, were identified using a search of medical databases. Two researchers systematically reviewed the papers to assess the reporting of participant compliance according to specified criteria, and the presentation of participant flow data recommended in the CONSORT guidelines. The literature search identified 58 papers. Almost a third (n = 18) did not describe how participant compliance was assessed. Nearly half of the papers (n = 27) failed to report participant compliance numerically (absolute numbers or percentage) and differences in compliance data between treatment arms were not reported in 52% of papers (n = 28). The majority (83%) gave no information on whether the study protocol included any researcher input aimed at maximizing compliance. In addition to inadequate reporting of compliance, two of the CONSORT requirements (eligibility criteria and numbers discontinuing the intervention) were inadequately reported in 69% and 60% of papers, respectively. We conclude that participant compliance in nutrition trials is frequently inadequately reported. ‘False negative’ results from RCTs with poor compliance could wrongly influence policy and inhibit further research concerned with nutritional supplementation for women of child‐bearing age. We suggest that changes to the CONSORT guidelines may improve RCT reporting.

Introduction

Randomized controlled trials (RCT) are considered to provide the most conclusive evidence of treatment effects within health intervention research (Altman et al. 2001; Jones 2002). Most RCTs involving nutritional supplementation take place over a sustained period of time. ‘Pregnancy’ trials present a particular challenge because supplementation may begin preconceptionally and continue until lactation ceases, and so participants may be required to consume supplements for several months or years. It is intuitive that longer duration of supplementation presents a greater participant burden and will be associated with reduced rates of participant compliance with the trial protocol. Participant compliance with nutritional supplements is conventionally defined as ‘ingesting the prescribed intervention as per the trial protocol’.

Low levels of participant compliance in one or more of the trial groups in an RCT may lead to a lack of power to detect treatment effects (Campbell et al. 2000). False negative findings may then be reported (Moher et al. 2001a; Moher 2007) and cause rejection of potentially effective therapies. Poor compliance can also reduce the external validity of the study (how generalizable the results are) if a particular subgroup is less compliant.

Several international nutrition journals specify that RCTs of nutritional supplementation should be reported according to the Consolidated Standards of Reporting Trials (CONSORT) (Altman et al. 2001). The CONSORT guidelines comprise three tools designed to guide the author in writing a report of an RCT: flow diagram, checklist and an elaboration document. CONSORT suggest that authors produce a flow diagram to account for the progression of participant numbers through a trial (Fig. 1).

Figure 1.

CONSORT (Consolidated Standards of Reporting Trials) Diagram showing flow of participants through each stage of a randomised trial.

In 2004, a systematic review of 27 health behaviour studies found that only 30% reported data on participant compliance (Dzewaltowski et al. 2004). To our knowledge, no previous study has reviewed the reporting of participant compliance in maternal nutritional intervention studies. We carried out a systematic review of papers reporting such studies. Our main objective was to assess the reporting of participant compliance data using criteria developed by us for the purposes of the review. Our second objective was to examine the authors' adherence to the CONSORT guidelines with respect to aspects of participant flow as depicted in the flow diagram.

Method

We searched MEDLINE (Medical Literature Analysis and Retrieval System Online), EMBASE (Excerpta Medica Database), CINAHL (Cumulative Index to Nursing and Allied Health Literature) and AMED (Allied and Complimentary Medicine) databases for papers reporting RCTs that investigated the effects of nutritional supplementation on infant outcomes. We carried out subject heading searches of 31 nutrient‐related terms (obtained from keywords found in pregnancy supplementation studies published before 2001). All yielded terms were then entered in a keyword search. We repeated this process for the terms ‘supplement’, ‘pregnancy’, ‘birth’ and ‘randomised controlled trial’. Papers containing one of each of these four groups of terms were included in the search results.

Papers were included in the review if they were published 1 year or more after the revised CONSORT statement (after 31/12/2001) and if they reported trials of single or multiple macronutrient or micronutrient supplements in the form of tablets, drinks or foods, administered before or during pregnancy. Infant outcomes included were mortality, birth size and/or weight, congenital abnormalities, metabolic function, nutrient status, atopy, growth and cognitive development.

Two reviewers independently classified the studies according to the reporting of compliance data by answering the following five questions:

• 1Was the method of measuring compliance reported? (Such that another researcher could carry out the same measurement).
• 2Was the method of measuring compliance objective? Participant self‐report was considered not to be objective. Pill counts, biochemical measures or being observed were considered to be objective methods.
• 3Was a compliance rate given for participants whose data were used in the final analysis?
• 4Were differences in compliance rate between the treatment groups reported in statistical terms?
• 5Were attempts made to maximize compliance? For example, by providing incentives or encouragement.

The reviewers also categorized the papers according to whether they reported the data specified in the CONSORT guidelines at each stage of the participant flow. One of these guidelines states that the number of participants who ‘received treatment’ should be reported. This is an ambiguous term, and could mean anything from, for example, the participant being given a week's supply of tablets with an instruction to take one daily, to the participant being observed by the researcher to ingest a tablet daily. For the purpose of this part of the review (adherence to CONSORT guidelines on reporting participant flow), we defined ‘received treatment’ as ‘the participant was given the treatment as allocated’ rather than ‘the participant ingested the treatment according to the study protocol’. We chose this less rigorous definition because: (1) the guidelines could reasonably be interpreted by authors in this way when referring to the participant flow diagram and the elaboration of the CONSORT statement (Altman, et al. 2001); and (2) the above five questions on compliance should adequately capture whether a study monitored ‘ingestion of’ as opposed to simply ‘receiving’ the treatment. Differences of opinion between reviewers were resolved by discussion.

Results

Study characteristics

The literature search yielded a total of 1184 papers published after 1st January 2002 of which 62 reporting results of RCTs that assessed the effect of nutritional supplementation during pregnancy on infant outcomes were identified from study titles. Four of these were rejected, one because the intervention consisted of a single‐dose supplement and three because they reported interventions administered to infants. The remaining 58 were included in our study. The number of participants in each study ranged from 16 to 8312, with a median [inter‐quartile range (IQR)] of 259 (83–921). Twenty‐seven studies were based in North America, Europe or Australia and the remaining 31 were in Africa, Asia or Latin America.

Study participants were all pregnant or pre‐pregnant women; 21 trials excluded women with high‐risk pregnancies or other health conditions while 17 studies specifically recruited women with a particular health condition or in an at‐risk group for pregnancy complications. The remaining 20 did not specify inclusion criteria with respect to pregnancy. Twenty‐three studies investigated the effects of single nutrients and 35 were multiple micronutrient supplementation trials. Thirty‐nine papers reported the exact length of supplementation (19 did not), the range being 20–730 days with a median (IQR) of 158 (117–179).

Reporting of ‘compliance’ data

Almost a third of papers (n = 18) did not report how the researchers assessed participant compliance (Table 1). Twelve per cent (n = 7) used participant self‐report as the only means of assessment and only one study used observation by a researcher. All other studies used pill counts, with one study using a computerized chip in the pill bottle to record the number of times the bottle was opened.

Table 1.

Number and percentage of studies reporting compliance data (n = 58)

	YES	NO	Not reported
	n (%)
Was the method of measuring compliance reported?	40 (69)	18 (31)	–
Was the method of measuring compliance objective?	34 (59)	6 (10)	18 (31)
Was the compliance rate of participants included in the analysis reported?	31 (53)	27 (46)	–
Did the paper state whether there was a significant difference in the number of compliers between treatment groups?	28 (48)	30 (52)	–
Were attempts made to maximize compliance?	10 (17)	0 (0)	48 (83)

Thirty‐one (53%) studies reported the compliance rate among participants whose data were included in the final analysis. Of these, 28 (48%) reported whether there was a significant difference in compliance rate between the experimental and control arms of the study. One paper did not report the compliance rate numerically but described it as ‘good’. Seventeen per cent (n = 10) stated that attempts were made by researchers to influence or maximize participant compliance. The remainder did not report whether such efforts were made.

Adherence to CONSORT guidelines

Approximately, a third of papers reported the number of persons assessed for eligibility to take part in the RCT (Table 2). All but two of the 58 included studies reported the number of participants allocated to each arm of the intervention. Eighty‐six per cent reported the number that received the allocated intervention while 79% gave the number that did not receive the intervention. The majority of papers (88%) quoted the number of participants lost to follow up and excluded from analysis. The number of participants whose data were included in the analysis was reported in 97% of studies while only 40% of papers reported how many participants discontinued the intervention.

Table 2.

Number of studies reporting CONSORT recommended data (n = 58)

CONSORT criteria	Number (%) of studies reporting recommended data
Assessed for eligibility	18 (31)
Allocated to intervention	56 (97)
Received allocated intervention	50 (86)
Did not receive allocated intervention	46 (79)
Lost to follow up	51 (88)
Discontinued intervention	23 (40)
Analysed	56 (97)
Excluded from analysis	51 (88)

CONSORT, Consolidated Standards of Reporting Trials.

Discussion

We carried out a systematic review of RCTs written and published since the CONSORT guidelines were revised in 2001 in order to assess the adequacy of reporting of information on compliance and participant flow. The main focus of the review, the reporting of information relating to compliance, was frequently inadequate. Almost a third of papers did not report the method used for assessing participant compliance, and approximately half of the papers did not state the compliance rate in the study as a whole or differentiate between treatment groups. This suggests a need for improvement in the reporting of nutritional trials, and possibly modification to the CONSORT guidelines. In addition, two items in the CONSORT participant flow diagram, eligibility criteria and numbers of subjects that discontinued the intervention, were not reported in 69% and 60% of papers, respectively. Other aspects of participant flow recommended by CONSORT were well presented in the majority of papers.

The use of a wide search strategy and multiple databases maximized the number of papers within the inclusion criteria for the review. Review of the studies by two independent parties allowed a good degree of objectivity. The study was limited to one area of RCT research and findings may not be applicable to research on other topics.

Of the studies reviewed that did report the method of assessing compliance, a small number employed self‐report as a measure. This method may be the only option in large‐scale trials but it is open to false reporting for social desirability reasons (Paulhus 1991). Pill counts were the most frequently used way to assess compliance; a disadvantage of this approach is that the study participants may not always ingest the supplement themselves. Researchers in one study observed participants consuming the supplement (Cox et al. 2005). This approach has advantages over pill counts and self‐reports in that there is no doubt that the participant is consuming the intervention, however, it may adversely affect compliance and willingness to participate in the study because being observed is likely to constrain participants in some way (e.g. by having to visit a distribution centre or being regularly visited in their home) and is likely to require greater staffing resources. Whichever method is used, it is helpful for the reader to be aware of how compliance was assessed in an RCT in order to judge how accurate the compliance data are likely to be.

Numerical compliance rates were frequently omitted from trial reports. Without these figures, the reader cannot fully interpret the results of the study. In the case of a negative finding, it may be that poor compliance led to the study being underpowered to demonstrate an intervention effect. Poor reporting of compliance data could be because of the researchers not collecting the information or not being aware that this data is important to the interpretation of the RCT results. The CONSORT checklist does specify that the number of participants that complied with the study protocol should be reported and in the elaboration of the CONSORT guidelines, it is pointed out that attrition because of follow‐up losses should be distinguished from investigator‐determined exclusion for reasons such as poor adherence to the trial protocol. However, this guidance implies that the reader should report how many participants are excluded from analysis because of poor compliance and it does not stress the importance of reporting the level of compliance in those that are included in the analysis. This is particularly pertinent to long‐term intervention studies where ‘intention to treat analysis’ is employed and participants are included in the analysis regardless of the extent to which they have complied with the trial protocol.

The CONSORT flow diagram states that the number of participants receiving treatment should be reported. This can be interpreted in two ways: obtaining the treatment or fully complying with the treatment protocol. In long‐term studies such as these, unless the participants in the trial are observed consuming the supplements, it is not possible to know whether they have fully complied with the protocol; only one of the 58 studies we reviewed observed the participants consuming the supplements. In RCTs such as these, as well as knowing whether the participants ‘received the treatment’ as allocated, it is useful to have some measure of the extent to which the participants complied with the protocol.

The majority of papers did not report whether efforts were made by researchers to maintain compliance and it is possible that attempts were made but this information was not available to the reader. Thus, it is often unclear whether the trial tested the efficacy or effectiveness of the supplement (whether it has a biological effect or whether it is effective at modifying an outcome in a ‘real‐life’ situation). Lower rates would be expected in effectiveness trials as efforts to maximize compliance would not usually be made (Gartlehner et al. 2006). Policy makers are likely to require this information when deciding whether a particular intervention is suitable for implementation in a public health context.

Eligibility figures are included in the participant flow diagram because they have relevance to external validity of trials. However, CONSORT point out that researchers will not always know the number of persons eligible for a trial and consider this to be less important than other aspects of participant flow (Altman et al. 2001). The number of participants discontinuing treatment may have been rarely reported because participants are more likely to display intermittent or poor compliance with a nutritional intervention than to stop taking it completely. Authors may consider that by including the number of participants lost to follow up, they are adequately reporting the number of ‘discontinuers’. This is unlikely to be a valid assumption as data on outcome parameters may be collected from participants without them necessarily having complied with the protocol in terms of the intervention. Our study suggests that there is room for improvement in reporting eligibility criteria, and losses because of discontinuation of the intervention in studies reporting results of nutrition intervention RCTs.

Suggestions for changes to guidelines and to practice

There is no basis for assuming that compliance will be high in any trial that requires a participant to change their behaviour on a regular basis over a sustained period of time. Therefore, it is important that information about participant compliance in an RCT of a sustained intervention be reported. A study by Moher et al. (2001b) concluded that authors generally adhere well to CONSORT guidelines when reporting their findings and evidence suggests that the inclusion of reporting criteria in the CONSORT recommendations leads to an improvement in trial reporting (Devereaux et al. 2002). Incorporation of participant flow criteria into a flow diagram has been found to improve reporting still further (Egger et al. 2001). We recommend that compliance rates in RCTs involving sustained interventions should always be reported numerically and as a percentage within each treatment group. It should be clear exactly what the number or percentage relates to, e.g. number or percentage of tablets taken or the number or percentage of ‘compliers’, with a comprehensive definition. We suggest that the CONSORT checklist and flow diagram specify that compliance figures are reported so that it is made clear to authors that these aspects of a trial are crucial to its evaluation and that the term ‘received treatment’ in the flow diagram (Fig. 1) is clarified. We also recommend that the elaboration and explanation of the CONSORT guidelines explain the importance of reporting the method of compliance assessment and whether efforts were made by researchers to maximize compliance.

Funding

This work was funded by the Medical Research Council (UK).

Conflict of interest

No conflicts of interest declared.

Key messages

• • Information on participant compliance in RCTs is important for interpretation of the trial results.
• • Compliance data is frequently inadequately reported in trials assessing the effect of nutritional supplements during pregnancy.
• • Incorporating specific instructions for authors in the CONSORT guidelines with respect to compliance may improve reporting of RCTs investigating the effects of long‐term interventions.