Effect of the current state of marking errors on national physical therapy examination scores

Abstract

[Purpose] In this study, we aimed to evaluate marking errors made by candidates who took trial National Physical Therapy Examination and examine their association with the national examination results. [Participants and Methods] We analyzed transcripts and mark sheets of 76 students who took three trial examinations conducted by Company M in the Department of Physical Therapy at International University of Health and Welfare. After each trial, the students submitted their marked answers using Google Forms. The total trial examination scores, the number of marking errors, and the number of incorrect answers were analyzed. Students were categorized into two groups: those who made no marking error in all three trials and those who made one or more marking errors. The total, practical question, and general question scores of the national examination were compared between the two groups. [Results] Some students made marking errors during the trial examinations, with the number of errors increasing over the three successive trial examinations. Furthermore, in the national examination, students who made no marking error in the trials scored significantly higher on practical questions than those who made errors. [Conclusion] The study findings highlight the importance of providing guidance to prevent marking errors during the national examination.

INTRODUCTION

In 1999, the educational curriculum was broadened following revisions to regulations for designating physical therapist and occupational therapist training facilities, leading to the establishment of national examination question standards. Subsequently, the content has been revised multiple times¹⁾. The National Physical Therapist Examination is held annually in late February. It comprises 100 questions in the morning and 100 in the afternoon, totaling 200 questions. The examination includes 160 general questions (1 mark each) and 40 practical questions (3 marks each), with a total score of 280 points. Candidates must score ≥168 points (60%) overall and ≥43 points in practical questions to pass. The two primary question formats are Type A, which requires selecting one correct answer from five options, and Type X2, which requires selecting two correct answers. The multiple-choice questions (MCQs) are used to assess the knowledge required for physiotherapists, and responses are recorded on a multiple-choice sheet.

Most students preparing for physical therapy have prior experience with multiple-choice tests and do not find them difficult. However, discrepancies have been observed between self-assessment scores and scores on submitted mark sheets. These discrepancies are attributable to two types of errors: mechanical and human errors. Marking errors owing to human mistakes can occur while answering questions or when transcribing responses onto the mark sheet²⁾. Mechanical errors result from reading failures by the Optical Mark Reader (OMR), which has a reported readability rate of <0.01%³⁾. Human errors occur when answers are placed in the wrong position, responses exceed the required number of marks, or marks are applied carelessly or too lightly to be read. As OMR errors are rare, most marking errors are attributed to human error.

The National Physical Therapist Examination requires a full day to complete 200 questions covering many topics. Because responses include A and X2 types, unexpected marking errors may occur. While these errors rarely increase scores, they frequently cause candidates to lose points⁴⁾. Failing the national examination necessitates waiting 1 year to retake it, potentially impacting career progression. Therefore, marking errors in trial examinations should be avoided. A study on errors in the national medical examination revealed that prolonged thinking and stress induce mental strain, increasing human error⁵⁾. Furthermore, Fukami et al.⁶⁾ reported that early individual instruction on marking errors, such as forgetting to mark or marking incorrectly in the national speech-language-hearing therapist examination, helped students review and prevent further errors, effectively reducing errors. A similar approach is expected to benefit National Physical Therapist Examination candidates, making repeated practice essential during the preparation stage.

Therefore, in this study, we aimed to clarify the circumstances under which National Physical Therapist Examination candidates made marking errors in trial examinations and assess how these errors influenced national examination results.

PARTICIPANTS AND METHODS

We analyzed the mark sheets and transcripts of 76 students who took the three trial examinations conducted by Medical, Dental, and Pharmaceutical Publishing Co., Ltd. (below: Company M) from a cohort of 78 students in the Department of Physical Therapy at International University of Health and Welfare who sat for the 2023 National Physical Therapist Examination. Then, national examination scores were obtained for 59 students who received their results after the national examination was completed (Fig. 1).

Fig. 1.

Participant selection.

The trial examinations were conducted by Company M in mid-November, early December, and late January. Each test comprised 200 MCQs of Type A and X2. The examination lasted 160 min in the morning and afternoon sessions. Students were not permitted to leave the room during the test. Candidates gathered in one location and recorded their answers on answer sheets provided by the company. After completing the test, students used Google Forms to transcribe the content of their test answer sheets. After distributing the trial examination answers, we asked all participants to enter their answers into the Google Forms. When entering the data, we instructed them to place their answer sheets in their bags so that they could not be seen and to take out only their workbooks and transcribe them.

For analysis, the median total score and first and third quartiles were calculated from vendor generated report cards. Similarly, we determined the number of marked incorrect answers and score of incorrectly answers. In this paper, a marking error is defined as the discrepancy between the trial examination answer and the transcribed self-assessment result. The Friedman test was used to analyze total scores, the number of marked incorrect answers and score of incorrectly answers across the three trial examinations. In addition, students checked the report cards sent by the vendor for each session. However, the teacher did not verbally warn individual students about how many grading errors they had made but only reminded the entire student body to pay attention to grading errors.

In order to collect data on total scores, practical question scores, and general question scores of the national examination, a questionnaire survey was conducted using Google Forms after the national examination results were published. The number and percentage of participants who made marking errors in each of the three trial examinations were calculated. Students were categorized into two groups: those who made no marking errors in all three trial examinations (the no-mistake group) and those who made one or more marking errors (the marking error group). After the national examination results were announced, the median, first quartile, and third quartile were calculated for total, practical question, and general question scores. A Mann–Whitney U test was conducted on the results of the three trial examinations and the national physical therapy examinations for the total scores, practical questions, and general questions of the no-mistake and marking error groups. Statistical analysis was performed using SPSS Statistics version 29 (IBM, Armonk, NY, USA), and the significance level was set at 5%.

Regarding ethical considerations, we applied to the Ethics Review Committee of the International University of Health and Welfare, which determined that the study did not fall under ethical guidelines for life sciences and medical research involving human participants and was not subject to ethical review. Therefore, following the Declaration of Helsinki, the study’s purpose was explained to participants orally and in writing, and informed consent was obtained before data collection.

RESULTS

For the 76 participants, the median scores of the three trial examinations are given in Table 1. The difference among the three trial examinations was statistically significant (p<0.01). In addition, a significant difference was found between the number of marking errors and number of points lost owing to marking errors (p<0.05).

Table 1. M Company’s trial examination results and current situation of marking errors.

	1st time	2nd time	3rd time
Median (points)**	177.5	176.0	188.5
First quartile (points)	160.0	157.8	176.0
Third quartile (points)	195.3	187.3	203.0
Number of marked incorrect answers (questions)*	37	83	84
Score of incorrectly answers(points)*	53	121	124
Maximum points conceded (points)	6	13	15

^*p<0.05, ^**p<0.01.

Fifty-nine students received their national examination scores. Marking errors were made by 15 students (25.4%) in the first mock exam, 29 (49.2%) in the second, and 22 (37.3%) in the third. The no-mistake group included 20 students (33.9%), while the marking error group comprised 39 (66.1%). The median scores (first to third quartile) for the no-mistake and marking error groups are listed in Table 2. A significant difference was observed in the practical questions of the first trial examination (p<0.05). No significant differences were found in total or general question scores of the national examination between the two groups. However, total and practical question scores were significantly higher in the no-mistake group than in the marking error group (p<0.05).

Table 2. Median, first quartile, and third quartile for the no-mistake and marking error group.

		No-mistake group	Marking error group
First trial examination	Total scores	190.0 (163.0–205.8)	178.0 (160.0–194.0)
	Practical question scores*	96.0 (85.5–99.8)	87.0 (81.0–93.0)
	General question scores	94.0 (79.0–102.8)	91.0 (81.0–101.5)
Second trial examination	Total scores	184.0 (158.3–194.0)	175.0 (158.5–187.0)
	Practical question scores	78.0 (67.5–84.8)	75.0 (63.0–82.5)
	General question scores	102.5 (96.0–109.3)	97.0 (94.0–108.0)
Third trial examination	Total scores	195.5 (177.0–204.8)	188.0 (174.0–203.0)
	Practical question scores	81.0 (74.3–85.5)	78.0 (72.0–84.0)
	General question scores	109.5 (105.8–123.0)	111.0 (103.5–119.5)
The national Physical Therapy examination	Total scores	225.5 (214.3–231.5)	215.0 (207.0–226.0)
	Practical question scores*	100.5 (98.3–105.8)	96.0 (91.5–102.0)
	General question scores	123.5 (119.0–126.3)	120.0 (113.0–125.0)

The median scores (first quartile–third quartile).

^*p<0.05.

DISCUSSION

We investigated marking errors in three trial examinations and found that through three successive tests. Furthermore, in the national examination, students who made no marking errors during the trial scored significantly higher on practical questions than those who made errors. This is the first study to suggest that avoiding marking errors may more reliably help students achieve a passing grade, highlighting the novelty of this study.

More marking errors occurred in the second and third trials than in the first trial examination. In the current study, unintentional errors in marking the answer sheets and transcription errors were possible causes. However, because participants were instructed to double-check their responses before submitting the Google Forms, transcription errors are unlikely to have occurred. Humans cannot maintain prolonged attention or tension and unconsciously release tension over time⁷⁾. This may explain why students became accustomed to the test and marked answers without thoroughly checking them, causing errors owing to decreased concentration. Company M records the results of trial examinations to identify marking errors. Even if students look at the results, they cannot be sure whether they have correctly understood the content. However, although many students check their marks, very few seem to check for marking errors. Furthermore, this study was unable to alert individual students each time a marking error occurred but only alerted the student body to the numerous marking errors. In psychology, human error is defined as failure to achieve the intended result in a planned psychological or physical process⁸⁾. Therefore, we believe that these mistakes may have occurred because some students were unaware of the detailed grading results or did not think that the grading error was their error.

In the national examination, the no-mistake group scored significantly higher on practical questions than the marking error group. Hitosugi et al.⁹⁾ reported no significant difference in test scores based on the presence or absence of human error; however, students with lower scores were more likely to make frequent errors. A 4.5-point difference in practical question scores suggests that students answered two questions incorrectly. The National Physical Therapy Examination can be passed by achieving a minimum required score rather than a near-perfect one; however, earning even one additional point may be crucial for success. The higher practical question scores in the no-mistake group underscore the significance of minimizing marking errors in national examinations, which determine whether students pass or fail. Reviewing and correcting errors, whether in trial or national examinations, is a critical skill even for practicing physical therapists. Identifying students prone to frequent marking errors early and implementing preventive strategies is essential. Hitosugi et al.²⁾ interviewed students who had experienced human errors and provided them with monthly reminders to read questions carefully, follow instructions, and review answers. Their findings showed a significant reduction in errors, demonstrating the educational benefits of this approach. In addition to identifying marking errors, analyzing the nature of human errors may help prevent marking errors.

A limitation of this study is that we examined data from a single year and a single university. Future studies should include multiple years and institutions to improve result accuracy. Additionally, the possibility that Company M’s mark sheet format contributed to errors cannot be ruled out. Therefore, further studies involving multiple companies are necessary. In addition, in the trial examination, for questions that students are unsure of their answers, the question booklet contains numerous sample answers, and it is expected that students will mark self-grading items that differ from those in the actual examination. Because we are unable to track the status of these students, future studies should explore how many questions students are unable to answer confidently.

In conclusion, in this study, we examined marking errors across three trial examinations and revealed that they were prevalent and influenced national examination results. As passing the national examination is essential for becoming a physical therapist, early measures to reduce marking errors are crucial.

Conflict of interest

There are no conflicts of interest to declare.