. 2020 Dec 3;20:293. doi: 10.1186/s12874-020-01179-5

Table 7.

Consensus reached on standards for preferred statistical methods for measurement error (agreement)

Statistical methods	very good	adequate	doubtful	inadequate
7	For continuous scores: was the Standard Error of Measurement (SEM), Smallest Detectable Change (SDC), Limits of Agreement (LoA) or Coefficient of Variation (CV) calculated? Relevance: 29/38 (76%)(R2^a); wording: 22/32 (69%)(R3^b); add CV: 22/30 (73%)(R3)	SEM, SDC, LoA or CV calculated; the model or formula for the SEM/SDC is described; it matches the study design^c and the data 32/36 (89%)(R2)	SEM, SDC, LoA or CV calculated, but the model or formula is not described or does not optimally match the study design and evidence provided that no systematic difference has occurred 25/34 (72%)(R2)	SEM_consistency SDC_consistency or LoA or CV calculated, without knowledge about systematic difference or with evidence provided that systematic difference has occurred	SEM calculated based on Cronbach’s alpha 22/31 (71%)(R3) OR using SD from another population 27/34 (79%)(R2)
8	For dichotomous/ nominal/ ordinal scores: Was the percentage specific (e.g. positive and negative) agreement calculated? 24/35 (69%)(R2)	% specific agreement calculated	% agreement calculated

Statistical methods

very good

adequate

doubtful

inadequate

For continuous scores: was the Standard Error of Measurement (SEM), Smallest Detectable Change (SDC), Limits of Agreement (LoA) or Coefficient of Variation (CV) calculated?

Relevance: 29/38 (76%)(R2^a); wording: 22/32 (69%)(R3^b); add CV: 22/30 (73%)(R3)

SEM, SDC, LoA or CV calculated; the model or formula for the SEM/SDC is described; it matches the study design^c and the data

32/36 (89%)(R2)

SEM, SDC, LoA or CV calculated, but the model or formula is not described or does not optimally match the study design and evidence provided that no systematic difference has occurred

25/34 (72%)(R2)

SEM_consistency SDC_consistency or LoA or CV calculated, without knowledge about systematic difference or with evidence provided that systematic difference has occurred

SEM calculated based on Cronbach’s alpha

22/31 (71%)(R3)

OR using SD from another population

27/34 (79%)(R2)

For dichotomous/ nominal/ ordinal scores: Was the percentage specific (e.g. positive and negative) agreement calculated?

24/35 (69%)(R2)

% specific agreement calculated

% agreement calculated

^a R2: consensus reached in round 2; ^b R3: consensus reached in round 3; ^c Based on panelists’ suggestions the steering committee decided after round 3 to use the word ‘study design’ instead of ‘reviewer constructed research question’