Abstract
Understanding clinical workflow is critical for researchers and healthcare decision makers. Current workflow studies tend to oversimplify and underrepresent the complexity of clinical workflow. Continuous observation time motion studies (TMS) could enhance clinical workflow studies by providing rich quantitative data required for in-depth workflow analyses. However, methodological inconsistencies have been reported in continuous observation TMS, potentially reducing the validity of TMS’ data and limiting their contribution to the general state of knowledge. We believe that a cornerstone in standardizing TMS is to ensure the reliability of the human observers. In this manuscript we review the approaches for inter-observer reliability assessment (IORA) in a representative sample of TMS focusing on clinical workflow. We found that IORA is an uncommon practice, inconsistently reported, and often uses methods that provide partial and overestimated measures of agreement. Since a comprehensive approach to IORA is yet to be proposed and validated, we provide initial recommendations for IORA reporting in continuous observation TMS.
BACKGROUND AND RATIONALE
Understanding the complexity of clinical workflow provides researchers and administrators with the necessary knowledge to assess some of the most critical issues in healthcare, such as: increasing patient safety by detecting medication errors1,2, improving quality of care by assessing timeliness of treatments3,4 and procedures5,6, productivity7,8 and efficiency9,10, integrating health information technologies and data management platforms11,12, and optimizing clinicians workload and time allocation13,14. Clinical workflow is commonly studied using qualitative approaches (ethnographic studies and interviews15), while quantitative methods use variations of time motion studies16,17. Time motion studies (TMS) were originally developed in industrial engineering and focused on the analysis of movements in a task with an emphasis on the amount of time required to perform the task18. Several different techniques have been compared for collecting quantitative workflow data19,20,21 (external observers, self-reports or databases analysis; work sampling vs. continuous observation), defining the use of an external observer as the gold standard to quantify clinical workflow22,23.
It is a common practice among workflow researchers to use a combination of both qualitative and quantitative methods. Typically, researchers interpret and model qualitative data onto theoretical activity diagrams24, and then enrich the abstracted workflow with time-data25 by having observers record the duration of expected ordered milestones26. For example, a common practice in medication administration studies is to develop an idealized medication administration flowchart, and then have observers time the duration of each predicted step27,28. However, that practice does not take into account interruptions and the intrinsic variability of nursing workflow29.
A more direct and exhaustive approach towards clinical workflow has been recently introduced within continuous observation TMS: workflow time studies. In this variation of continuous observation TMS, observers continuously follow a subject for a predefined period of time and record tasks as they occur, producing a data schema of time-stamped sequences of tasks30,14. This technique allows observers to track unexpected instances of tasks, accounting for task fragmentation, interruptions, and the real-world variability of clinical workflow. However, this approach substantially increases the burden on observers, raising concern on observers’ reliability. While the introduction of electronic time capture tools has facilitated the recording process by allowing observers to direct their attention on the subjects being studied31, the benefits of this methodology to workflow studies might be impeded by the complexity of the data capture process, producing unreliable data due to overburdened observers. The potential contributions of continuous observation TMS to clinical workflow studies could be meaningfully exploited only if the concerns regarding observer reliability can be overcome: the quantitative assessment of observers’ reliability should be methodically assessed and reported in a standardized fashion.
OBJECTIVE
In this report, we aim to contribute to the validation of clinical workflow continuous observation time-motion studies by analyzing the diverse practices to inter-observer reliability as found in a representative sample of reports describing such studies, and further, by assessing their suitability and appropriateness. Specifically, we intend to:
Describe the current approaches in inter-observer reliability assessment (IORA) by researchers studying clinical workflow using continuous observation time motion studies in the last five years [2008–2012].
Assess the features and limitations of the methods reported.
Synthesize the preceding analyses and propose a set of recommendation concerning how to address gaps in knowledge and practice pertaining to the use of IORA in workflow studies.
METHODS
Search Strategy
We concentrated our search effort on PubMed since the focus of our research question is restricted to the biomedical domain. We included articles published in English between January 2008 and December 2012, and selected the initial query terms based on a subject matter expert driven heuristics. After doing iterative refinement of the search strategy, the main query terms included were “Time and Motion Studies”, “Workflow”, “Observation” and “Activity,” We filtered out sports related publications because of a decrease in the specificity of the query by retrieving mostly motion studies using tracking devices. The query was run on March 2013 and returned 152 citations [Figure 1].
Figure 1:
Search strategy
Study Selection
Our study focused on clinical workflow studies using a continuous observation time motion methodology: observers having the ability to capture instances of occurring tasks and time-stamping or recording their duration. Thus, we excluded non-clinical studies, studies using other time motion approaches (work sampling, self-reports, database time-stamps), studies using a unit of timing other than seconds, and studies focusing exclusively on timing predefined unique instances of milestones in a confined process. The exclusion process was conducted by an MD (ML) and an RN/PhD in biomedical informatics (PY), both with extensive experience with time motion studies. A final manual search for potential articles not detected by our query and that met the inclusion criteria was also conducted.
Data extraction
An MD (ML) and a PhD candidate in biostatistics (SB) fully assessed the included articles and extracted data regarding the presence of an inter-observer reliability assessment instance, the method(s) used (if any) and the value(s) reported.
Data analysis
Descriptive statistics of the relative occurrences of IORA and the methods used were summarized. When IORA was reported, a deep analysis of the implemented method was attempted.
RESULTS
The 152 titles/abstracts were reviewed excluding 62 articles. The remaining 90 articles were fully reviewed, excluding 50 additional articles. A manual search returned an additional 9 articles meeting our inclusion criteria but not detected by our query. These 49 articles studying clinical workflow using continuous observation TMS were fully assessed.
Of the 49 reviewed articles, 47%(23) did not report having conducted any kind of inter-observer reliability assessment. Of those, 6%(3) recognized the lack of IORA as a limitation in their discussion. Among those who reported an IORA, 50%(13) did not specify the method used to calculate the reported value, while 23%(6) used the kappa-coefficient, and the rest used Pearson product-moment correlation, Spearman correlation, interclass correlation coefficient, percentage agreement or Bland-Altman. Two articles used a combination of intraclass correlation and kappa [Table 1].
Table 1:
Inter-observer reliability assessments current practices.
| IORA reported | n | % | |
|---|---|---|---|
| YES | single method reported | ||
| Kappa-coefficient | 6 | 12% | |
| Pearson product-moment correlations | 1 | 2% | |
| Spearman correlation | 1 | 2% | |
| interclass correlation coefficient | 1 | 2% | |
| percentage agreement | 1 | 2% | |
| Bland-Altman | 1 | 2% | |
| not specified | 13 | 27% | |
| multiple methods reported | |||
| intraclass correlation & Kappa | 2 | 4% | |
|
|
|||
| NO | 23 | 47% | |
|
|
|||
| Total | 49 | 100% | |
When assessing articles who reported IORA, the values reported are systematically high. Within the same methods, authors do not follow a pattern for reporting values. For example, among those who reported a kappa coefficient, some reported a point value, while others reported a lower boundary, or a range, rarely accompanied by a confidence interval [Table 2].
Table 2:
Type and magnitude of values reported for each method used
| Method used | value reported | details |
|---|---|---|
|
| ||
| Kappa-coefficient | 0.86 | p<0.001 |
| 0.94 | -- | |
| 0.78, 0.66 | -- | |
| 0.94–0.96 | -- | |
| >0.89 | -- | |
| 0.97 | 95%CI[0,91;1] | |
| 0.82 | 95%CI[0,61;1] | |
| 0.71 | t=41.6, p=0.00 | |
|
| ||
| Pearson product-moment correlations | 0.93, 0.89, 0.96 | -- |
|
| ||
| Spearman correlation | r=0.89 and 0.86 (frequency) r=0.85 and 0.84 (duration) |
-- |
|
| ||
| interclass correlation coefficient | >0.95 | 95%CI[0.85;1] |
|
| ||
| percentage agreement | >=85% | -- |
|
| ||
| Bland-Altman | −0.06 | 95%CI[-0.284;0.164] |
|
| ||
| intraclass correlation | 0.99 | |
| 0.96 | ||
|
| ||
| not specified | 85% | range 85%–98% |
| 79% | -- | |
| >80% | -- | |
| 78% | -- | |
| 82% | -- | |
| >88% | -- | |
| 88% | -- | |
| >85% | -- | |
| 0.96 | -- | |
| >85% | -- | |
| >85% | 95%CI[72%;89%] | |
| >95% | -- | |
| 73% | -- | |
DISCUSSION
We found that the reporting of inter-observer reliability is often incomplete and inadequate. Our results bring awareness of several potential limitations of the current practices for conducting IORA: the lack of consistency of conducting and reporting IORA appropriately, the intrinsic limitations of some of the methods used, and partial data integrity assessment by only evaluating one dimension of the data captured.
Lack of consistency of conducting and reporting IORA
Every data collection method requiring a human interface is subject to variability and error in the data capture process. In order to minimize and handle that variability, first it has to be recognized, taken into account and measured, thus allowing researchers to conduct a meaningful analysis of the data. TMS are not an exception to the rule: the value that continuous observation TMS could add to workflow studies depends entirely on the raters’ experience, focus, and ability to capture reliable information of the environment observed. Also, reliability and agreement are not fixed properties of measurement tools, but rather the product of interactions between the tools, subjects and the environment under study.32 Hence, IORA should be a methodic practice in every TMS.
In our review, 23 out of 49 articles (47%) did not report any form of assessment of IORA, and 13 out of the 26 articles (50%) that reported having conducted an IORA did not specify the method used to calculate the values declared.
The report of IORA statistics were found to be in various formats, including the average percentage agreement, the minimum of agreement, a single agreement statistic, or accompanied by a 95% confidence interval for the statistics. Even for the authors who specified a method, important information such as the data used for calculating reliability and a brief mention of the implementation of the statistical method were systematically underreported. For example, the kappa coefficient is an appropriate measure of reliability, but it is known to be influenced by the prevalence of the attribute and the number of categories33,34. For two pairs of observers that have the same percentage agreement, the values of the kappa coefficients can be different depending on the specific numbers of paired data that agree and disagree34. Therefore, it is difficult for readers to interpret and rely on the validity of the studies when not knowing the details of the calculation.
It is also important to point out that researchers should be aware of two types of observer reliability: the intra-observer and inter-observer reliability. The intra-observer reliability (also known as test-retest) assesses the variability of a single observer’s measurements over time, and thus requires repeated or replicated data within each observer. Given the nature of the inherent fluctuating and changing nature of clinical processes, intra-rater reliability assessments are impractical in continuous observation TMS. The inter-observer reliability assesses agreement between different observers following the same set of subjects, thus is an appropriate measure for the reliability in TMS. Nevertheless, we found that most studies only conducted inter-rater reliability assessments prior to data capture in a pilot study. We feel that researchers should conduct and report “in-study” IORA as well in order to control for observers’ drift: observers changing rating practices over time35.
Intrinsic limitations of methods used
Among the studies that reported an IORA statistics, the most common ones are kappa coefficient36, intraclass correlation coefficient37 Spearman or Pearson’s correlation coefficient, and the Bland-Altman plot38. One study39 reported interclass correlation coefficient, though we believe that it could be a typo. The intraclass correlation coefficient is the appropriate definition and measure of reliability, not interclass correlation coefficient.
The kappa coefficient and the intraclass correlation coefficient are the traditional approaches for assessing agreement in categorical data and continuous data, respectively41. The kappa coefficient is a measure of correlation between categorical variables. It has meaning beyond percentage agreement corrected for chance. In addition, kappa coefficient is designed to measure correlation between nominal data (e.g. task frequency, number of task categories), but not for ordinal data (task order)42. The intraclass correlation coefficient is a popular reliability measure for continuous data (mean duration time/task), defined as the ratio of the variance of between-observer values to the variance of the total (within + between) observed values. It relies on the correct assumption of the analysis-of-variance models where the variance components are estimated from. In our review, we found that when a statistic was reported, most studies used kappa coefficient for task frequency, and intraclass correlation coefficients for mean duration time, but only 9 articles (18%) reported such appropriate statistics.
The Bland-Altman plot and Limit-of-Agreement estimates (LoA) is the most popular agreement tool used by medical researchers in clinical studies. It is a simple approach that is based on the pairwise difference between measurements made by the two observers on the same subject, accompanied by the Bland-Altman plot. The Bland-Altman plot is a scatter plot of the difference versus the average of the readings made by the two observers. It provides a visual examination of the outliers and the magnitude of the disagreement, and it is also used to check the assumption of the LoA estimates. For proper inference, the LoA estimates should be compared with a pre-specified clinically acceptable range. In TMS studies, the usual setting involves the comparison of two observers, therefore the LoA estimates is an appropriate tool for assessing IORA. However, in our review, only one article used a Bland-Altman plot, but it did not provide the Bland-Altman plot or report LoA estimates, nor did it mention the pre-specified acceptable range40.
The percentage agreement, Spearman’s correlation and Pearson’s correlation coefficients are inappropriate for assessing IORA. The percentage agreement ignores the multiple dimensions of the TMS data. The simplification of TMS data to calculate percentage agreement can create arbitrarily high agreement statistics, as with the example we discussed earlier with kappa coefficient. The Pearson’s correlation coefficient only assesses the linear association of measurements between two observers, but not their agreement43. The Spearman’s correlation is a nonparametric measure of the association between paired measurements. Similar to Pearson’s correlation, it also does not quantify the agreement. Despite the fact that these statistics are not recommended by statisticians in agreement studies, we found that these methods are still being used when assessing IORA in TMS.
Multidimensional data
As workflow researchers, we are interested in every aspect of the clinical workflow: the sequence (order in which tasks occur), occurrence (time when tasks occur), count (number of tasks occurrences) and duration of tasks required to accomplish a goal. The multi-dimensionality of the data produced by continuous observation TMS provides the required information for such comprehensive analyses. Thus, each aspect should be taken into account when conducting inter-observer reliability to ensure that we are maintaining the integrity of data produced by TMS. To our knowledge, there is no study that has yet looked at the multiple dimensions of TMS for IORA: the appropriate methodology for IORA for the rich data of continuous observation TMS needs to be developed. In our review, 6%(3) were aware of this issue, and in an initial approach to a more comprehensive approach, they attempted to use a combination of 2 methods44,45 (intraclass correlation for time and kappa for categorization) or used Spearman’s correlation for both frequency and duration46.
Recommendations
Since a comprehensive approach to IORA is yet to be proposed and validated, researchers currently conducting TMS should at least report the methods implemented in an appropriate fashion. By following the Guidelines for Reporting Reliability and Agreement Studies (GRRAS)32, we encourage researchers in TMS to report the following information for IORA:
Identify in the abstract that IORA was assessed.
Assess the intra-observer reliability before and during the study to control for observer drift
Describe the measurement/data collection process
Describe the observer population, number, and duration of the observations
Describe the statistical analysis for IORA in details
Report estimates of IORA statistics including measures of statistical uncertainty (standard error, 95% CI)
Include citation if the magnitude of the IORA statistics is compared with a guideline
Provide detailed results and explanation of IORA in context if possible
Make the data available for method development of IORA in TMS
LIMITATIONS
The use of a single database (PubMed/MEDLINE) might have limited the scope of the included articles for review, thus any study not indexed in MEDLINE or with insufficient methods description in the abstract might have been missed by our query.
CONCLUSION
Inter-observer reliability assessments is not a common practice among clinical workflow TMS. When conducted, it is underreported, utilizes methods with limited applicability, and usually focuses only on one dimension of the data. Future directions of our work include the development and validation of a comprehensive method for IORA in TMS. The establishment of a comprehensive inter-observer reliability assessment method presents as the next crucial milestone in validating TMS’ contributions to clinical workflow studies, finally aiming at a meaningful and comprehensive time motion driven workflow analysis methodology.
REFERENCES
- 1.Barker KN, Flynn EA, Pepper GA. Observation method of detecting medication errors. American journal of health-system pharmacy : AJHP : official journal of the American Society of Health-System Pharmacists. 2002;59:2314–6. doi: 10.1093/ajhp/59.23.2314. [DOI] [PubMed] [Google Scholar]
- 2.Keohane CA, et al. Quantifying nursing workflow in medication administration. The Journal of nursing administration. 2008;38:19–26. doi: 10.1097/01.NNA.0000295628.87968.bc. [DOI] [PubMed] [Google Scholar]
- 3.Rebmann T, Clements BW, Bailey JA, Evans RG. Organophosphate antidote auto-injectors vs. traditional administration: a time motion study. The Journal of emergency medicine. 2009;37:139–43. doi: 10.1016/j.jemermed.2007.09.043. [DOI] [PubMed] [Google Scholar]
- 4.Wolfrum S, Pierau C, Radke PW, Schunkert H, Kurowski V. Mild therapeutic hypothermia in patients after out-of-hospital cardiac arrest due to acute ST-segment elevation myocardial infarction undergoing immediate percutaneous coronary intervention. Critical care medicine. 2008;36:1780–6. doi: 10.1097/CCM.0b013e31817437ca. [DOI] [PubMed] [Google Scholar]
- 5.Wiese CHR, et al. Using a laryngeal tube during cardiac arrest reduces “no flow time” in a manikin study: a comparison between laryngeal tube and endotracheal tube. Wiener klinische Wochenschrift. 2008;120:217–23. doi: 10.1007/s00508-008-0953-1. [DOI] [PubMed] [Google Scholar]
- 6.Zalaudek I, et al. Time required for a complete skin examination with and without dermoscopy: a prospective, randomized multicenter study. Archives of dermatology. 2008;144:509–13. doi: 10.1001/archderm.144.4.509. [DOI] [PubMed] [Google Scholar]
- 7.Were MC, et al. Patterns of care in two HIV continuity clinics in Uganda, Africa: a time-motion study. AIDS care. 2008;20:677–82. doi: 10.1080/09540120701687067. [DOI] [PubMed] [Google Scholar]
- 8.Schiller B, Doss S, DE Cock E, Del Aguila MA, Nissenson AR. Costs of managing anemia with erythropoiesis-stimulating agents during hemodialysis: a time and motion study. Hemodialysis international International Symposium on Home Hemodialysis. 2008;12:441–9. doi: 10.1111/j.1542-4758.2008.00308.x. [DOI] [PubMed] [Google Scholar]
- 9.Fisher J, Lotery H, Henderson C. Time in motion--testing efficiency in the dermatology procedure setting. Dermatologic surgery : official publication for American Society for Dermatologic Surgery [et al] 35:437–44. doi: 10.1111/j.1524-4725.2009.01076.x. discussion 445 (2009). [DOI] [PubMed] [Google Scholar]
- 10.Amusan AA, Tongen S, Speedie SM, Mellin A. A time-motion study to evaluate the impact of EMR and CPOE implementation on physician efficiency. Journal of healthcare information management : JHIM. 2008;22:31–7. [PubMed] [Google Scholar]
- 11.Morgan MB, et al. Just-in-time radiologist decision support: the importance of PACS-integrated workflow. Journal of the American College of Radiology : JACR. 2011;8:497–500. doi: 10.1016/j.jacr.2011.01.003. [DOI] [PubMed] [Google Scholar]
- 12.Vishwanath A, Singh SR, Winkelstein P. The impact of electronic medical record systems on outpatient workflows: a longitudinal evaluation of its workflow effects. International journal of medical informatics. 2010;79:778–91. doi: 10.1016/j.ijmedinf.2010.09.006. [DOI] [PubMed] [Google Scholar]
- 13.Trotter MJ, Larsen ET, Tait N, Wright JR. Time study of clinical and nonclinical workload in pathology and laboratory medicine. American journal of clinical pathology. 2009;131:759–67. doi: 10.1309/AJCP8SKO6BUJQXHD. [DOI] [PubMed] [Google Scholar]
- 14.Westbrook JI, Ampt A, Kearney L, Rob MI. All in a day’s work: an observational study to quantify how and with whom doctors on hospital wards spend their time. The Medical journal of Australia. 2008;188:506–9. doi: 10.5694/j.1326-5377.2008.tb01762.x. [DOI] [PubMed] [Google Scholar]
- 15.Malhotra S, Jordan D, Shortliffe E, Patel VL. Workflow modeling in critical care: piecing together your own puzzle. Journal of biomedical informatics. 2007;40:81–92. doi: 10.1016/j.jbi.2006.06.002. [DOI] [PubMed] [Google Scholar]
- 16.Clinical rounds: Nursing activities: how paperwork eats up your time. Plastic surgical nursing : official journal of the American Society of Plastic and Reconstructive Surgical Nurses. 30:123. doi: 10.1097/PSN.0b013e3181ebc7b7. [DOI] [PubMed] [Google Scholar]
- 17.Hendrich A, Chow MP, Skierczynski BA, Lu Z. A 36-hospital time and motion study: how do medical-surgical nurses spend their time? The Permanente journal. 2008;12:25–34. doi: 10.7812/tpp/08-021. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Time and Motion Studies - MeSH - NCBI. at < http://www.ncbi.nlm.nih.gov/mesh?term=Time+and+Motion+Studies>.
- 19.Wirth P, Kahn L, Perkoff GT. Comparability of two methods of time and motion study used in a clinical setting: work sampling and continuous observation. Medical care. 1977;15:953–60. doi: 10.1097/00005650-197711000-00009. [DOI] [PubMed] [Google Scholar]
- 20.Gordon BD, Flottemesch TJ, Asplin BR. Accuracy of staff-initiated emergency department tracking system timestamps in identifying actual event times. Annals of emergency medicine. 2008;52:504–11. doi: 10.1016/j.annemergmed.2007.11.036. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Ampt A, Westbrook J, Creswick N, Mallock N. A comparison of self-reported and observational work sampling techniques for measuring time in nursing tasks. Journal of health services research & policy. 2007;12:18–24. doi: 10.1258/135581907779497576. [DOI] [PubMed] [Google Scholar]
- 22.Burke TA, et al. A comparison of time-and-motion and self-reporting methods of work measurement. The Journal of nursing administration. 2000;30:118–25. doi: 10.1097/00005110-200003000-00003. [DOI] [PubMed] [Google Scholar]
- 23.Bratt JH, et al. A comparison of four approaches for measuring clinician time use. Health policy and planning. 1999;14:374–81. doi: 10.1093/heapol/14.4.374. [DOI] [PubMed] [Google Scholar]
- 24.Kersting M, Hauswaldt J, Lingner H. [Modeling the requirements on routine data of general practitioners from the health-care researcher’s point of view with the help of unified modeling langauge (UML)] Gesundheitswesen (Bundesverband der Ärzte des Öffentlichen Gesundheitsdienstes (Germany)) 2012;74:e68–75. doi: 10.1055/s-0032-1314824. [DOI] [PubMed] [Google Scholar]
- 25.De Carvalho ECA, et al. Standardizing clinical trials workflow representation in UML for international site comparison. PloS one. 2010;5:e13893. doi: 10.1371/journal.pone.0013893. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Fischman D. Applying Lean Six Sigma Methodologies to. 2010;19:201–210. doi: 10.1097/QMH.0b013e3181eece6e. [DOI] [PubMed] [Google Scholar]
- 27.Tschannen D, Talsma A, Reinemeyer N, Belt C, Schoville R. Nursing medication administration and workflow using computerized physician order entry. Computers, informatics, nursing : CIN. 2011;29:401–10. doi: 10.1097/NCN.0b013e318205e510. [DOI] [PubMed] [Google Scholar]
- 28.Elganzouri ES, Standish CA, Androwich I. Medication Administration Time Study (MATS): nursing staff performance of medication administration. The Journal of nursing administration. 2009;39:204–10. doi: 10.1097/NNA.0b013e3181a23d6d. [DOI] [PubMed] [Google Scholar]
- 29.Westbrook JI, Woods A, Rob MI, Dunsmuir WTM, Day RO. Association of interruptions with an increased risk and severity of medication administration errors. Archives of internal medicine. 2010;170:683–90. doi: 10.1001/archinternmed.2010.65. [DOI] [PubMed] [Google Scholar]
- 30.Mache S, et al. Cardiologists’ workflow in small to medium-sized German hospitals: an observational work analysis. Journal of cardiovascular medicine (Hagerstown, Md) 2011;12:475–81. doi: 10.2459/JCM.0b013e328347db8f. [DOI] [PubMed] [Google Scholar]
- 31.Lopetegui M, Yen P, Lai AM, Embi PJ, Payne PRO. Time Capture Tool (TimeCaT): Development of a Comprehensive Application to Support Data Capture for Time Motion Studies. AMIA Annual Symposium Proc; 2012. [PMC free article] [PubMed] [Google Scholar]
- 32.Kottner J, et al. Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed. Journal of clinical epidemiology. 2011;64:96–106. doi: 10.1016/j.jclinepi.2010.03.002. [DOI] [PubMed] [Google Scholar]
- 33.Bakeman R, McArthur D, Quera V, Robinson BF. Detecting sequential patterns and determining their reliability with fallible observers. Psychological Methods. 1997;2:357–370. [Google Scholar]
- 34.Gwet K. Inter-rater reliability: dependency on trait prevalence and marginal homogeneity. Statistical methods for inter-rater reliability …. 2002:1–9. [Google Scholar]
- 35.Leckie G, Baird J-A. Rater Effects on Essay Scoring: A Multilevel Analysis of Severity Drift, Central Tendency, and Rater Experience. Journal of Educational Measurement. 2011;48:399–418. [Google Scholar]
- 36.Cohen J. A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement. 1960;20:37–46. [Google Scholar]
- 37.Fisher RA. Statistical Methods for Research Workers. oliver and boyd; London: 1925. [Google Scholar]
- 38.Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet. 1986;1:307–10. [PubMed] [Google Scholar]
- 39.Tipping MD, et al. Where did the day go?--a time-motion study of hospitalists. Journal of hospital medicine : an official publication of the Society of Hospital Medicine. 5:323–8. doi: 10.1002/jhm.790. [DOI] [PubMed] [Google Scholar]
- 40.Abbey M, Chaboyer W, Mitchell M. Understanding the work of intensive care nurses: A time and motion study. Australian critical care : official journal of the Confederation of Australian Critical Care Nurses. 2011 doi: 10.1016/j.aucc.2011.08.002. [DOI] [PubMed] [Google Scholar]
- 41.Lin L. Overview of agreement statistics for medical devices. Journal of biopharmaceutical statistics. 2008;18:126–44. doi: 10.1080/10543400701668290. [DOI] [PubMed] [Google Scholar]
- 42.Chmura Kraemer H, Periyakoil VS, Noda A. Kappa coefficients in medical research. Statistics in medicine. 2002;21:2109–29. doi: 10.1002/sim.1180. [DOI] [PubMed] [Google Scholar]
- 43.Altman D, Bland J. Measurement in medicine: the analysis of method comparison studies. The statistician. 1983;32:307–317. [Google Scholar]
- 44.Cady R, et al. Exploring the translational impact of a home telemonitoring intervention using time-motion study. Telemedicine journal and e-health : the official journal of the American Telemedicine Association. 2010;16:576–84. doi: 10.1089/tmj.2009.0148. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 45.Lindquist R, et al. Time-motion analysis of research nurse activities in a lung transplant home monitoring study. Progress in transplantation (Aliso Viejo, Calif) 2011;21:190–9. doi: 10.7182/prtr.21.3.u267v51327m276l2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46.Dasgupta A, et al. Descriptive analysis of workflow variables associated with barcode-based approach to medication administration. Journal of nursing care quality. 2011;26:377–84. doi: 10.1097/NCQ.0b013e318215b770. [DOI] [PubMed] [Google Scholar]