. 2019 May 5;9(5):e022105. doi: 10.1136/bmjopen-2018-022105

Table 3.

Summary of PTTT validation study outcomes

PTTT	First author, year	Country	Study population	Study design	Number of centres	PTTT used in practice?	Internal/ external validation study?	Outcome measures	Sample size	Score or trigger?	Score tested/ maximum score	Which score used (frequency of scoring)?^*	AUROC	Sensitivity	Specificity	PPV	NPV	Notes on accuracy/ reliability of scoring and missing data	Quality score (max=24)
Paediatric early warning system (PEWS) score	Duncan 2006⁴⁷	Canada	All inpatients	Case-control study (retrospective)	1	No	Int	Code blue call for actual or impending cardiopulmonary arrest	215 (87 cases)	S	5/26	Max 24 hours before event (hourly)	0.90	78.0	95.0	4.2†		No details on data abstraction. 13% of eligible cases and 84% of eligible controls excluded due to incomplete clinical data.	14
	Robson 2013²⁸	USA	All inpatients	Case-control study (retrospective)	1	No	Ext	Code blue call	192 (96 cases)	S	5/32	Max 24 hours before event (six hourly)	0.85	86.6	72.2			Four researchers scored PTTT from 20 charts, inter-rater reliability of 0.95. No details on extent of missing data.	8
	Chapman 2017²⁹	UK	All inpatients	Case-control study (retrospective)	1	No	Ext	Death, arrest or unplanned PICU transfer	608 (297 cases)	S	7/32	Max 48 hours before event (per usual practice)	0.82	70.0	75.0	72.6	72.0	Data abstraction by single researcher. 36% of observation sets contained HR, RR, O2 Sats, systolic BP, temperature and assessment of consciousness.	17
Bedside PEWS	Parshuram 2009¹¹	Canada	All inpatients	Case-control study (retrospective)	1	No	Int	Urgent PICU transfer (without code blue call)	180 (60 cases)	S	8/26	Max 24 hours before event (hourly)	0.91	82.0	93.0			Availability of scoring items in medical records varied from 27% (cap refill time) to 93% (oxygen therapy).	21
	Parshuram 2011⁴⁴	Canada and UK	All inpatients	Case-control study (prospective)	4	No	Ext	Urgent PICU transfer or immediate call to resuscitation team	2074 (686 cases)	S	7/26	Max 24 hours before event (hourly)	0.87	64.0	91.0			PTTT scores calculated electronically after abstraction by research nurse. 5.1% of records had all seven items recorded, 31% had at least five items.	22
	Robson 2013²⁸	USA	All inpatients	Case-control study (retrospective)	1	No	Ext	Code blue call	192 (96 cases)	S	7/26	Max 24 hours before event (six hourly)	0.73	56.3	78.1			See above.	8
	Zhai 2014¹⁹	USA	All inpatients	Case-control study (retrospective)	1	No	Ext	Urgent PCU transfer within 24 hours of admission	6352 (53 cases)	S	7/26	Max 24 hours before event (hourly)	0.82	73.6	71.7	2.1†		Data extracted from electronic health records. Excluded two items of Bedside PEWS (oxygen therapy and respiratory effort) due to difficulty abstracting.	17
	Gawronski 2016⁴⁶	Italy	Stem Cell Transplant Unit	Case-control study (retrospective)	1	No	Ext	Unexpected death, urgent consult with RRT or urgent PICU transfer	99 (19 cases)	S	6/26	Score 4 hours before event	0.90	79.0	97.5			Data abstracted by research nurses. No details on extent of missing data. Conflicting/missing observations resolved by interviews with clinical staff.	15
	Chapman 2017²⁹	UK	All inpatients	Case-control study (retrospective)	1	No	Ext	Death, arrest or PICU transfer	608 (297 cases)	S	6/26	Max 48 hours before event (per usual practice)	0.88	72.0	89.0	86.0	77.0	See above.	17
Modified Bedside PEWS (a)	Fuijkschot 2015³⁰ (study 1)	The Netherlands	Oncology ward	Case-cohort study (retrospective)	1	Yes	Int	Emergency medical intervention or reviewed by PICU staff or staff concern	118 (15 cases)	S	8/28	Unclear (minimum eight hourly)				73.0		41% of admissions excluded from study due to incomplete PTTT scores.	10
	Fuijkschot 2015³⁰ (study 2)	The Netherlands	All inpatients	Case-cohort study (retrospective)	1	Yes	Int	PICU transfer	Unclear (24 cases)	S	8/28	Score 2–6 hours before event (minimum eight hourly)		66.6				High rate of exclusions reported due to missing data.	10
	Fuijkschot 2015³⁰ (study 3)	The Netherlands	All inpatients	Case-cohort study (prospective)	1	Yes	Int	Emergency medical intervention	Unclear (14 cases)	S	8/28	Unclear (minimum eight hourly)		100				No details on missing data.	10
	Chapman 2017²⁹	UK	All inpatients	Case-control study (retrospective)	1	No	Ext	Death, arrest or PICU transfer	608 (297 cases)	S	7/28	Max 48 hours before event (per usual practice)	0.87	69.0	91.0	87.9	79.0	See above.	17
Modified Bedside PEWS (b)	Ross 2015⁴⁹	USA	All inpatients	Case-control study (retrospective)	1	No	Int	Urgent PICU transfer	4628 (848 cases)	S	8/26	Max during admission		70.0	84.0			No details on data abstraction. Respiratory effort category excluded due to difficulty abstracting. No details on missing data.	9
Modified Brighton PEWS (a)	Tucker 2009³¹	USA	General medical unit	Cohort study (prospective)	1	Yes	Int	PICU transfer	2979 (51 cases)	S	3/11	Max during admission (four hourly)	0.89	90.2	74.4	5.8	99.8	Intraclass coefficient of 0.92 reported for two bedside nurses scoring 55 patients. No details on missing data.	14
	Zhai 2014¹⁹	USA	All inpatients	Case-control study (retrospective)	1	No	Ext	Urgent PCU transfer within 24 hours of admission	6352 (53 cases)	S	2/11	Max 24 hours before event (hourly)	0.74	68.4	81.6	2.3		Data extracted from electronic health records. Only included records with complete PEWS score: 64% of eligible cases and 51% of eligible controls excluded.	17
	Fenix 2015³⁹	USA	PICU transfers among all inpatients (excluding haematology oncology, surgical and cardiac wards)	Case-control study (retrospective)	1	Yes	Ext	Non-elective PICU transfer followed by deterioration event	97 PICU transfers (51 cases of PICU transfer followed by ‘deterioration event’)	S	3/11	Max during admission		80.0	43.0	61.0	67.0	No details on missing data.	15
Modified Brighton PEWS (b)	Akre 2010⁴⁵	USA	All inpatients	Chart review study (retrospective)	1	No	Int	Rapid response team call or code blue call	186 cases (170 RRT calls, 16 code calls)	S	4/13	Max 24 hours before event (minimum four hourly)		85.5				Scores abstracted from charts by single nurse, having calibrated with advanced nurse practitioner. Categories scored missing if any items missing. 25% of charts missing behavioural state, 26% cardiovascular colour.	14
Modified Brighton PEWS (b)	Chapman 2017²⁹	UK	All inpatients	Case-control study (retrospective)	1	No	Ext	Death, arrest or PICU transfer	608 (297 cases)	S	4/13	Max 48 hours before event (per usual practice)	0.79	61.0	84.0	78.4	69.0	See above.	17
Modified Brighton PEWS (d)	Skaletzky 2012⁴⁸	USA	Medical surgical wards	Case-control study (retrospective)	1	No	Int	PICU transfer	350 (100 cases)	S	2.5/9	Max 48 hours before event (four hourly)	0.81	62.0	89.0			Data abstracted from medial charts and notes. Behaviour category abstracted from LOC. No details on missing data.	15
Modified Brighton PEWS (d)	Chapman 2017²⁹	UK	All inpatients	Case-control study (retrospective)	1	No	Ext	Death, arrest or PICU transfer	608 (297 cases)	S	4/9	Max 48 hours before event (per usual practice)	0.74	46.0	90.0	81.3	63.0	See above.	17
Children’s Hospital Early Warning Score	McLellan 2014⁵⁰	USA	All inpatients	Case-control study (retrospective)	1	Yes	Int	Arrest or unplanned PICU transfer	1136 (360 cases)	S	4/12	Max in admission (four hourly)	0.90	84.2	80.9			No details on missing data.	10
Children’s Hospital Cardiac Early Warning Score	McLellan 2013²³	USA	Cardiovascular unit	Case-control study (retrospective)	1	Yes	Int	Arrest or unplanned PICU transfer	312 (64 cases)	S	3/12	Max 18 hours before event (four hourly)	0.86	95.3	76.2	50.8	98.4	Study nurse and bedside nurses assessed scores for 37 patients, 67% agreement. No details on missing data.	9
	Agulnik 2016⁴¹	USA	Oncology unit	Case-control study (retrospective)	1	Yes	Ext	Unplanned PICU transfer	330 (110 cases)	S	4/12	Max 24 hours before event (four hourly)	0.96	86.0	95.0			PTTT scores abstracted by researcher. Did not abstract if vital signs were present but no PTTT score calculated by nurse. No details on missing data.	14
	Agulnik 2017⁴²	Guatemala	Oncology unit	Case-control study (retrospective)	1	Yes	Ext	Unplanned PICU transfer	258 (129 cases)	S	4/12	Max 24 hours before event (three hourly)		91.0	88.0			Researcher evaluated charts and calculated scores, reporting 14% error rate (PTTT score calculated incorrectly) and 3% omission rate (vital signs recorded but no PTTT score calculated). One out of 130 cases excluded due to missing PTTT documentation.	16
Children’s Hospital Los Angeles PEWS	Mandell 2015⁴⁰	USA	Inpatients discharged from PICU to ward	Case-control study (retrospective)	1	Yes	Int	Early unplanned re-admission to PICU (within 48 hours of discharge from PICU to ward)	189 (38 cases)	S	2/10	First score assigned on ward, post-PICU discharge	0.71	76.0	56.0			No details on missing data.	12
Melbourne Activation Criteria	Tume 2007³	UK	Inpatients with an unplanned PICU transfer	Chart review study (retrospective)	1	No	Ext	Unplanned PICU transfer	33 cases	T	N/A	Unclear		87.8				Data abstracted by two reviewers. Reference to ‘large number of missing records and observation charts’.	11
	Tume 2007³	UK	Inpatients with an unplanned PHDU transfer	Chart review study (retrospective)	1	No	Ext	Unplanned PHDU transfer	32 cases	T	N/A	Unclear		87.5				See above.	11
	Edwards 2011³³	UK	All inpatients	Cohort study (retrospective)	1	No	Ext	Death or unplanned PICU or HDU transfer	1000 (16 cases)	T	N/A	Any trigger over admission (per usual practice)	0.79	68.3	83.2	3.6	99.7	Observation charts altered to include all PTTT parameters. 56% of records missing at least one component. Missing data assumed to be normal.	17
	Chapman 2017²⁹	UK	All inpatients	Case-control study (retrospective)	1	No	Ext	Death, arrest or PICU transfer	608 (297 cases)	T	N/A	Max 48 hours before event (per usual practice)	0.71	93.0	49.0	64.0	88.0	See above.	17
Cardiff and Vale PEWS	Edwards 2009³²	UK	All inpatients	Cohort study (prospective)	1	No	Int	Death or unplanned PICU or HDU transfer	1000 (16 cases)	S	2/8	Max score during admission (per usual practice)	0.86	69.5	89.9	5.9	99.7	Observation charts altered to include all PTTT parameters. 56% of records missing at least one component. Missing data assumed to be normal.	18
Cardiff and Vale PEWS	Chapman 2017²⁹	UK	All inpatients	Case-control study (retrospective)	1	No	Ext	Death, arrest or PICU transfer	608 (297 cases)	S	3/8	Max 48 hours before event (per usual practice)	0.89	80.0	86.0	84.0	82.0	See above.	17
Bristol paediatric early warning tool (PEWT)	Tume 2007³	UK	Inpatients with an unplanned PICU transfer	Chart review (retrospective)	1	No	Ext	Unplanned PICU transfer	33 cases	T	N/A	Unclear		87.8				See above.	11
	Tume 2007³	UK	Inpatients with an unplanned PHDU transfer	Chart review (retrospective)	1	No	Ext	Unplanned PHDU transfer	32 cases	T	N/A	Unclear		84.4				See above.	11
	Wright 2011³⁵	UK	All inpatients	Chart review (retrospective)	1	Yes	Ext	Cardiac arrest	55 cases	T	N/A	If triggered 24 hours before event		49.1				One case excluded due to missing notes. No details on missing data.	11
	O’Loughlin 2012³⁴	UK	All inpatients	Cohort study (prospective)	1	Yes	Ext	PICU transfer	331 (7 cases)	T	N/A	Triggered during admission (12 hourly)	0.91	100	81.0	11.0		No details on missing data.	6
	Robson 2013²⁸	USA	All inpatients	Case-control study (retrospective)	1	No	Ext	Code blue call	192 (96 cases)	T	N/A	Triggered 24 hours before event (6 hourly)	0.75	76.3	61.5			See above.	8
	Chapman 2017²⁹	UK	All inpatients	Case-control study (retrospective)	1	No	Ext	Death, arrest or PICU transfer	608 (297 cases)	T	N/A	If triggered 48 hours before event (per usual practice)	0.62	96.0	28.0	56.0	88.0	See above.	17
Modified Bristol PEWT (b)	Clayson 2014³⁸	UK	Cardiac ward	Cohort study (prospective)	1	Yes	Int	‘A deteriorating patient’	126 (unclear number of cases)	T	N/A	Unclear				12.5	97.0	No details on missing data.	5
NHS Institute for Innovation and Improvement PEWS	Mason 2016¹⁴	UK	All inpatients	Cohort study (retrospective)	1	No	Ext	Death or unplanned PICU or HDU transfer	1000 (16 cases)	S	2/7	Max score over admission (per usual practice)	0.88	80.0	81.0	4.3	99.7	Observation charts altered to include all PTTT parameters. 56% of records missing at least one component. Missing data assumed to be normal.	15
NHS Institute for Innovation and Improvement PEWS	Chapman 2017²⁹	UK	All inpatients	Case-control study (retrospective)	1	No	Ext	Death, arrest or PICU transfer	608 (297 cases)	S	2/7	Max 48 hours before event (per usual practice)	0.82	83.0	65.0	69.6	80.0	See above.	17
Logistic regression algorithm	Zhai 2014¹⁹	USA	All inpatients	Case-control study (retrospective)	1	No	Ext	Urgent PICU transfer within 24 hours of admission	6352 (53 cases)	S	>0.5	Max 24 hours before event (hourly)	0.91	84.9	85.9	4.8		Data extracted from electronic health records. No details on extent of missing data but authors report that ‘missing data were a major cause of incorrect prediction’.	17
Burton Paediatric Early Warning Score	Ahmed 2012³⁶	UK	PICU admissions only	Chart review (retrospective)	1	Yes	Int	PICU admission	23	S	4/19	Max 24 hours before event (unclear)		93.0				Data extracted from case notes by two reviewers. No details on missing data.	4
‘Between the Flags’ PEWS	Blackstone 2017⁴³	UK	Urgent PICU admissions only	Chart review (retrospective)	1	Yes	Ext	Urgent PICU admission	100	T	N/A	Unclear		91.0				Data extracted from health records. No details on missing data.	8

All studies conducted in a specialist/tertiary centre.

PPV and NPV values in italics represent results from case-control studies—these values are misleading in isolation because they assume that the wider prevalence rate of the adverse event is equal to the case to control ratio used in the research study (eg, if the researchers studied 300 cases and 300 controls, the prevalence rate of adverse events for the calculation of PPV is 50%). As per the cohort studies, prevalence rates of critical events are typically far lower among hospitalised paediatric populations than the case-control ratios used in studies, and so PPV values would be considerably lower in clinical practice.

Studies classified as internal validation if the setting for the study was the same hospital and same research team as those who developed the score. Studies classified as external validation if the score was tested in a different centre and by a different research team to those who developed it.

*Typically, study researchers collected or abstracted multiple PTTT scores for each patient at different time points, but can only use one score per patient for the analysis of the tool’s predictive ability. This column specifies which score the researchers used. In most cases, the study team used the maximum PTTT score recorded for each patient in a given study window, eg, 24 hours prior to a critical event for case patients. The text in parentheses describes the frequency with which scores were assessed or abstracted for each patient, if this information was described in the paper.

†Case-control study, but PPV value calculated based on clinical prevalence of event as measured at local centre during the study.

AUROC, area under the receiver operating characteristic curve; Ext, external validation; HFNC, high flow nasal cannula; Int, internal validation; Max, maximum; N/A, not applicable; NPV, negative predictive value; PHDU, paediatric high-dependency unit; PICU, paediatric intensive care unit; PPV, positive predictive value; PTTT, paediatric track and trigger tool; RRT, rapid response team; S, score; T, trigger.