Figure 4. Quantifying the agreement between expert scenarios of metastasis.

(A) A hypothetical example of scenarios produced by two experts (experts 1 and 2). Expert 1 suggested two alternative scenarios (ABC and AFDC), while expert 2 suggested only one (ACBD). We define the agreement/similarity between experts as twice the number of stage pairs they share, of all possible ordered pairs of stages from each scenario. We define the disagreement/dissimilarity as the agreement minus the sum of unmatched pairs; see the example. (B) Cumulative probabilities that two experts agree at least at level x (similarity of x or greater), and that two experts disagree at level x or less. The probability of agreement drops rapidly as the number of statements from each expert increases, while the probability of disagreement grows gradually to a rather large number of pairwise disputes. (C) A hypothetical four-expert “regulatory deadlock” could occur if any one of the experts insisted on his disagreement with all others.