Table 3.
Item by Item Test-Retest & Inter-rater Reliability Showing % Agreement and Kappa Statistics (N = 24 Files)
| Section | Element | Response type | SP | AS | ES | Test-re-test | Inter-rater | ||
|---|---|---|---|---|---|---|---|---|---|
| % agree | kappa | % agree | kappa | ||||||
| History | Demographic Data Complete | subjective | S | 87.50 | 0.3739 | 84.72 | 0.1410 | ||
| History | Privacy Form present and complete | subjective | E | 83.33 | 0.7467 | 59.72 | 0.3737 | ||
| History | New Patient Form present and complete | subjective | S | 66.67 | 0.3725 | 48.61 | 0.0111 | ||
| History | Health Status Survey/Pain Diagram reviewed by clinician | objective | S | 75.00 | 0.4000 | 66.67 | 0.2334 | ||
| History | Outcome Measures appropriate | objective | S | A | 79.17 | 0.5699 | 72.22 | 0.4198 | |
| History | Past Health History | subjective | S | A | 62.50 | −0.0385 | 66.67 | 0.1111 | |
| History | Family Health History | subjective | S | A | 62.50 | 0.4114 | 47.22 | 0.1582 | |
| History | Social History | subjective | S | A | 87.50 | 0.6066 | 66.67 | 0.0759 | |
| History | Flags | subjective | A | 54.17 | 0.2000 | 37.50 | −0.0679 | ||
| History | Primary Complaint Complete | subjective | S | A | 83.33 | 0.5556 | 50.00 | −0.2101 | |
| History | Secondary Complaint (if applicable) | subjective | S | 62.50 | 0.3684 | 45.83 | 0.1063 | ||
| History | Consent to Physical Examination obtained and documented | subjective | S | A | 100.00 | 1.0000 | 97.22 | 0.9434 | |
| History | Differential Diagnoses rendered | subjective | S | A | 83.34 | 0.6129 | 75.00 | 0.4311 | |
| History | Complexity Completed | objective | A | 87.50 | 0.6538 | 75.00 | 0.3069 | ||
| Physical Examination | Observation/posture | subjective | S | 95.84 | 0.8710 | 90.28 | 0.7483 | ||
| Physical Examination | Vitals | subjective | S | A | 75.00 | 0.4947 | 58.33 | 0.3411 | |
| Physical Examination | Range of Motion | subjective | E | 79.17 | 0.5219 | 68.06 | 0.3196 | ||
| Physical Examination | Palpation | subjective | E | 100.00 | 1.0000 | 100.00 | 1.0000 | ||
| Physical Examination | Orthopaedic Procedures | subjective | S | A | 95.84 | 0.7857 | 88.89 | 0.5590 | |
| Physical Examination | Correlates with history | subjective | A | 95.84 | 0.7857 | 91.67 | 0.6364 | ||
| Physical Examination | Neurologic Procedures | subjective | S | A | 95.84 | 0.8909 | 94.44 | 0.8633 | |
| Physical Examination | Procedures sufficient to: Exclude differential diagnoses | subjective | A | 87.51 | 0.5714 | 86.11 | 0.4865 | ||
| Physical Examination | Procedures sufficient to: Render clinical diagnosis | subjective | S | A | 100.00 | 1.0000 | 91.67 | 0.5917 | |
| Physical Examination | Complexity Completed | objective | A | 91.67 | 0.7612 | 86.11 | 0.5740 | ||
| Diagnosis | Diagnoses rendered | subjective | S | A | 95.84 | 0.8182 | 97.22 | 0.8786 | |
| Diagnosis | Appropriate and supported by findings | subjective | A | 95.84 | 0.7857 | 80.56 | 0.3429 | ||
| Diagnosis | Complexity Completed | objective | A | 95.84 | 0.8806 | 91.67 | 0.7690 | ||
| Plan of Management | Further Evaluation | subjective | S | 87.51 | 0.5294 | 80.56 | 0.4661 | ||
| Plan of Management | Frequency, duration | subjective | S | A | 95.84 | 0.8182 | 90.28 | 0.5758 | |
| Plan of Management | Therapy details | subjective | S | A | 83.34 | 0.4353 | 83.33 | 0.4368 | |
| Plan of Management | Active program - planned/ detailed | subjective | A | 79.17 | 0.5604 | 62.50 | 0.1914 | ||
| Plan of Management | Evidence-based | objective | A | 95.84 | 0.73333 | 86.11 | 0.3182 | ||
| Plan of Management | Complexity Completed | objective | A | 87.50 | 0.6418 | 88.89 | 0.6883 | ||
| Plan of Management | Dx & POM - Verified by clinician | objective | E | 91.67 | 0.6364 | 83.33 | 0.4525 | ||
| Goals & Outcomes | Appropriate Specific & measurable goals & outcomes present | objective | A | 75.00 | 0.4217 | 63.89 | 0.0517 | ||
| Goals & Outcomes | Repeat OM measures administered | subjective | A | 66.67 | 0.3786 | 51.39 | 0.1286 | ||
| Prognosis | Prognosis appropriately defined | subjective | S | A | 91.67 | 0.7209 | 80.56 | 0.4540 | |
| Prognosis | Short term & long term prognosis, support by positive and negative prognostic factors | subjective | S | A | 75.00 | 0.5826 | 58.33 | 0.3463 | |
| Consent / ROF | ROF complete | subjective | S | A | 87.50 | 0.6539 | 80.56 | 0.5512 | |
| Consent / ROF | Risks – major/minor & benefits | subjective | S | A | 91.67 | 0.7176 | 91.67 | 0.7512 | |
| Consent / ROF | Recording of patient Questions in ROF | objective | S | 87.51 | 0.7714 | 97.22 | 0.9428 | ||
| Consent / ROF | ROF signed by patient | objective | S | 95.83 | 0.8571 | 94.44 | 0.8093 | ||
| Consent / ROF | ROF signed by clinician | objective | S | 95.83 | 0.8571 | 94.44 | 0.8093 | ||
| Consent / ROF | ROF/IC current | objective | S | A | 95.83 | 0.8571 | 94.44 | 0.8093 | |
| Consent / ROF | IC signed by patient: | objective | S | 95.83 | 0.8571 | 91.67 | 0.7323 | ||
| Consent / ROF | IC signed by clinician | objective | S | 95.83 | 0.8571 | 94.44 | 0.8093 | ||
| Consent / ROF | IC signed by intern | objective | E | 95.83 | 0.8571 | 94.44 | 0.8093 | ||
| Case Report / Physicians letter | Clear concise narrative | subjective | E | 70.83 | 0.4167 | 63.89 | 0.1628 | ||
| Case Report / Physicians letter | Professional format | subjective | E | 75.00 | 0.3077 | 76.39 | 0.2842 | ||
| Documentation | Collaborative care/correspondence (if applicable) | subjective | A | 75.00 | 0.4667 | 70.83 | 0.2738 | ||
| Documentation | Exchange of Medical Info form present & complete | objective | E | 87.50 | 0.8059 | 34.72 | −0.0349 | ||
| Documentation | Dashboard (brown) boxes current & complete (Social Hx, Med.Hx, Ongoing Concerns, Reminder Section) | subjective | S | A | 70.84 | 0.4894 | 66.67 | 0.3410 | |
| Documentation | Time to conditional case sign-off < 7 days | subjective | E | 50.00 | 0.3043 | 30.56 | −0.0164 | ||
| Documentation | Case Signed off | objective | E | 87.50 | 0.5714 | 73.61 | 0.2392 | ||
| Documentation | SOAP notes: - Complete | subjective | S | 87.50 | −0.0588 | 77.78 | −0.0159 | ||
| Documentation | Clinician verification of SOAP | objective | S | 95.83 | 0.0000 | 94.44 | −0.0286 | ||
| Documentation | Response to care documented in SOAP | subjective | S | 79.16 | 0.5200 | 66.67 | 0.1787 | ||
| Documentation | Timely uploading of documents | objective | S | A | 62.50 | 0.2703 | 63.89 | 0.2034 | |
| Documentation | Compliance documented in file | subjective | S | A | 66.66 | 0.4037 | 48.61 | −0.0238 | |
| Re-evaluation | Completed as scheduled in POM | subjective | S | A | 87.50 | 0.8059 | 29.17 | −0.0588 | |
| Re-evaluation | Includes relevant details - Thorough & Complete | subjective | S | A | 74.99 | 0.6308 | 29.17 | −0.0217 | |
| Totals | N = 38 | N = 37 | N = 10 | N = 61 | N = 61 | N = 61 | N = 61 | ||
| > 70% agreement per Standards: test-retest | 32/38 (84%) | 31/37 (84%) | 9/10 (90%) | 52 > 70% | k > 0.6 32/61 | 37 > 70% | k > 0.6 16/61 | ||
| > 70% agreement per Standards: IRR | 23/38 (60%) | 23/37 (60%) | 5/10 (50%) | ||||||
ROF report of findings. SP Standards of practice. AS Accreditation standard. ES Educational standard. IRR interrater reliability