An automated online proctoring system using attentive-net to assess student mischievous behavior

Abstract

In recent years, the pandemic situation has forced the education system to shift from traditional teaching to online teaching or blended learning. The ability to monitor remote online examinations efficiently is a limiting factor to the scalability of this stage of online evaluation in the education system. Human Proctoring is the most used common approach by either asking learners to take a test in the examination centers or by monitoring visually asking learners to switch on their camera. However, these methods require huge labor, effort, infrastructure, and hardware. This paper presents an automated AI-based proctoring system- ‘Attentive system’ for online evaluation by capturing the live video of the examinee. Our Attentive system includes four components to estimate the malpractices such as face detection, multiple person detection, face spoofing, and head pose estimation. Attentive Net detects the faces and draws bounding boxes along with confidences. Attentive Net also checks the alignment of the face using the rotation matrix of Affine Transformation. The face net algorithm is combined with Attentive-Net to extract landmarks and facial features. The process for identifying spoofed faces is initiated only for aligned faces by using a shallow CNN Liveness net. The head pose of the examiner is estimated by using the SolvePnp equation, to check if he/she is seeking help from others. Crime Investigation and Prevention Lab (CIPL) datasets and customized datasets with various types of malpractices are used to evaluate our proposed system. Extensive Experimental results demonstrate that our method is more accurate, reliable and robust for proctoring system that can be practically implemented in real time environment as Automated proctoring System. An improved accuracy of 0.87 is reported by authors with the combination of Attentive Net, Liveness net and head pose estimation.

Introduction

The outbreak of the COVID-19 pandemic has severely affected different sectors of society, especially education. The education sector is recovering from the pandemic with the usage of digital devices to avoid a generational catastrophe in teaching, learning, and evaluation. Every component of education — including but not limited to classroom teaching and teaching methods, student learning and exposition, reciprocation and review, examination and testing, and assessment and evaluation — has undergone paradigm shifts to meet the expectations in the realms of academia, research, and innovation at large. The considerable shift in the expected skills, training, and acumen of graduates has initiated these changes, particularly in the domains of science, technology, and engineering. Information and Communication Technology supports knowledge transmission to remote places. Various technological platforms have been introduced into the education sector to decrease the transitional distance among the learners. The orientation towards blended learning is highly promoted as a combination of both conventional and online teaching. Online education offers the learners to attend the classes either in synchronous or asynchronous mode. The challenges of the virtual learning world demand an array of strategies to gauge the attentiveness of the learners in the e-classrooms. The pandemic made the teachers, institutions and systems to imbibe victorious strategies to usher in a learning system to “make your way by unexpected routes and tackle unguarded spots - problems that are novel in virtual learning. In addition, it facilitates the instructors to evaluate the learners also in both modes. Asynchronous mode of evaluation is not preferred by all the instructors for summative assessment. For synchronous mode of examinations, a human proctor manually examines the examinee and alerts if any mischievous behavior is observed. But, this physical proctoring does not identify all types of malpractice behavior of the examinee [31]. In line with the present scenario revisiting the existing/past practices to re-visualize, recast, and redesign the leading conventions and evolving as “competent” individuals, who can examine the students and online proctoring is the switch required. As per the current scenario, various Software industries have introduced proctoring software like ProctorU [40] to help the instructors in the conduction of examination in online mode, though there is still a need for trained proctors [10] or sensors [50] to do online monitoring of the examines to avoid malpractices. The authors in [2] state that the percentage of students indulging in mischievous behavior has been increased drastically during these online modes. Semi-automated proctoring software restricting the examinee to not to open any other browsers during the exam and also take the snaps of the students in random intervals to capture the behavior through various video Processing techniques like dominant Color [36], SURF, HARRIS, BRISK features [35], Color and shape features [37]. Automated Online proctoring systems uses Artificial Intelligence and Deep Learning techniques to identify the attentiveness of learner/examinee in virtual learning. Fully automated online proctoring facilitates an instructor to identify the misbehavior of the examinee with minimum requirements like internet connectivity, webcam, etc. using various deep learning models [15] and various convolutional neural networks [60]. The proposed automated proctoring identifies different physical activities of the examinee through the use of a camera. The proctoring system alerts the examiner if the examinee is not found or if multiple persons are detected or if face spoofing is identified. Furthermore, these proctoring systems are also used to assess the attentiveness of the examinee during the examination and as well as in the regular classrooms.

The proposed automated proctoring system – Attentive System helps to examine the attentiveness of the examinee with minimum false alarms. In this study, the authors proposed Attentive-Net along with an Inception-Resnet to detect various activities of the examinees during an examination. The proposed system aims to detect the faces from the live video captured through the camera. To reduce the complexity of the model, the proposed system checks for the aligned faces in the frame and stops further processing and alerts the proctor if no aligned face is detected. For aligned faces, the proposed system extracts the facial features and detects the liveliness of the face. Our system also employs these features to estimate the head pose of the examinee.

In this work, a real-time online proctoring system – Attentive System is proposed with the following objectives:

• Face detection system using an Attentive Network to assess the attentiveness of the examinee.

• FaceNet approach to extract features using Inception-Resnetv1 blocks to carryout landmark localization and Affine Transformation to check the face alignment.

• Evaluation of the proposed method on customized and CIPL Datasets to estimate the accuracy in Head Pose and Face Spoofing respectively.

In Chapter 2, a thorough survey on literature published on various online proctoring systems and face detection systems is given. Chapter 3 illustrates various activities including facial detection, multiple face detection, no person identification, face alignment, liveness detection, head-pose estimation, and other mischievous actions performed by the examinee. The performance of the proposed work with discussions is presented in Chapter 4. Chapter 5 contains the conclusion and the future Scope.

Related work

Over the recent years, researchers have developed a wide spectrum of proctoring methodologies for online examinations. These methodologies are mainly divided into four categories: (i) No proctoring (ii) Manual Proctoring (iii)Semi-automated Proctoring and (iv) Fully Automated Proctoring. In the absence of online proctoring, mischievous behavior of the examinees is identified through eight online exam control procedures like offering the online exam only at one set time, exam to be accessible only for a brief period, to randomize questions, etc. [11]. In [51], network security issues were considered to prevent cheating of examinee. In manual proctoring, well-trained proctors are employed but this needs more manpower. Guo et al. [19] in their study took snapshots at regular intervals and observed the behavior of the examinee. In Semi-automated proctoring, 360⁰ cameras are employed to get the view of the examinee from all angles. Some manual proctoring systems [25] uses group cryptography for controlling of ports and inputs. In desktop Robo [42] robots are also used to monitor the examinee’s behavior. In [27], both front and rear camera are used to identify the examinee behavior. These methods are very costly and do not identify all types of malpractice behavior. In [24], AI based techniques are used to identify examinee behavior. In (https://towardsdatascience.com/automating-online-proctoring-using-ai-e429086743c8) some fully automated online proctoring systems, eye-gaze tracking is used to monitor the examinees which requires high-end infrastructure. In [38], by using Dlib’s facial keypoint detector and OpenCV, eye-gaze tracking is implemented. Object detection techniques (https://towardsdatascience.com/automating-online-proctoring-using-ai-e429086743c8) are used to identify various objects used for cheating proctor. This system is limited to identifying only one type of misbehavior. Pratish et al. [38] used audio-visual traits to identify the mischievous behavior of the examinee. Atoum et al. [2] used two cameras to do behavior profiling to capture both what the examinee sees and what their behavior is. In this work, the authors proposed a semi-automated system in which they used four-vision-based capabilities like gaze tracking, mouth open or close, face accounting, and mobile phone detection.

Face Detection

Facial detection plays a prominent role in the online proctoring application. Despande et al. [14] used Region-Based Convolutional Neural Networks (R-CNN) for feature extraction from the faces detected. Histogram Intersection Kernel with Support Vector Machines (SVM) classifier is used for the classification. Sun et al. [47] used a faster R-CNN framework including feature concatenation, hard negative mining, multi-scale training for face detection. This framework needs to be boosted for different patterns of the face and has a computational burden. Resnet50 based Caffe model (https://towardsdatascience.com/automating-online-proctoring-using-ai-e429086743c8) identifies multiple persons before the camera and as well as the absence of a person before the camera. It is observed that the background affects the accuracy of the model. In [12], Haar Cascade algorithm combined with three weak classifiers based on skin hue-histogram matching, eyes detection and mouth detection. This is not suitable for the real time applications due to its high computational time. In [48], authors applied skin filter and entropy rate super pixels (ERSs) to obtain face candidates. Then, angle compensation, refinement and symmetry extension are applied to improve accuracy of face detection. This technique is robust to in-plane rotation even if out-plane rotation exists. Chong Li et al. [29] improved Yolov3 for the face detection by changing detection layer and choosing soft max loss function to maximize the difference of inter-class features.

Face alignment

Face alignment verification is not performed in this methodology which plays key role in proctoring. In [26], authors proposed PCA (Principal Component Analysis) to reduce the large amount of data storage. Self-space projection is used in face detection by defining a number of facial images with the own vectors of the covariance matrix. Authors in [54] designed a hierarchical attention mechanism instead of convolution neural networks. Authors proposed Gaussian kernels to search for proper positions and scales of local regions to extract informative features of the face. Then LSTM is used to model relations between the local parts and adjust their contributions to the detection tasks. The LSTM introduces additional computational cost. In [7], a three-level Cascaded Neural Network learned via residual learning was used for face alignment identification. These subnetworks are based on residual part heatmap regression. Wenyan wu et al. [53], uses boundary information of the face to predict landmarks coordinates. Face geometric structure is drawn using landmark coordinates which is used for verifying face alignment. The boundary aware face alignment algorithm does not perform well for very small faces as landmarks are not accurately reconstructed from boundary. In [45], authors proposed multiple shape prediction layers for detecting certain cluster of semantically relevant landmarks. As training should be performed for each cluster of landmarks, training cost increases. Authors in [61] used both non-linear layers and linear layers for face alignment. The non-linear layers with cosine activations encodes relationships between representations of images and shapes of facial landmarks. The linear layers are used to encode landmark correlations by low-rank learning. This works better for heavy occluded images. Authors of [6] estimated the facial landmarks by using the image intensities. Rotation Vector is calculated using Rodrigues rotation formula for verifying the face alignment. Authors [8] proposed discriminative Feature Learning method for face alignment based on the fully convolutional Network. Affine Transformation is used on hourglass network for aligning the face. In [30], authors analyze the texture of the facial images using multi-scale local binary patterns. The high-level features are robust to real time tasks than the textural features of image, deep features are extracted. Gustavo et.al [13], integrates deep texture features with LBP descriptor and gives as input to convolution neural network. The Caffe- framework is considered as a base framework.

Face Spoofing

In [28], Local Binary Patterns (LBP) layers were combined with convolution layers for face spoofing detection. The VGG-face network is used for face detection. For face spoofing, the VGG-face network needs to be fine-tuned. The accuracy of face spoofing detection depends upon the features extracted from the face. Authors in [46], designed a descriptor called spatial pyramid coding micro-texture feature to characterize the local appearance information and then uses SSD to conduct end-to-end face presentation detection. Then, fake faces are detected by designing template-face matched binocular depth feature. This method requires a special hardware device like binocular camera. In [56], authors used convolution neural networks named G-Net to rough estimate the head pose and localize seven primary landmarks. One more CNN LNet is used to learn local CNN features and predict pose. Overfitting is adjusted to only some extent.

Head Pose Estimation

In [5], the Regression algorithm applied to a web-shaped model was used for head-pose estimation. This method also predicts the value of the dependent variable for all three angular values. Pitch angular error is a limitation to this method. In [43], the authors used face identification, multiple face detection, and head-pose estimation for the identification of malpractices during examinations. In this work, the authors used yaw angle for head pose estimation and achieved 80% accuracy. Yaw angle was used to obtain the three-dimensional rotation angles of the face. To identify the malpractices, three-dimensional angles are not required which increases the complexity of the model. But this method identifies only some types of malpractices.

Mobile Detection

Authors in (https://towardsdatascience.com/automating-online-proctoring-using-ai-e429086743c8) used pre-trained Yolov3 for mobile phone detection. Yolo model is trained on Microsoft Common Objects in Context (COCO) dataset in which mobile phones are identified as class 67. Yolov3 is based on Darknet consisting of 53 layers. In [17], authors combined top-down and bottom-up connections to fuse features across scales. Authors also combined with various backbone models in RetinaNet Framework. Authors in [32] used weighted bi-directional feature pyramid network and Efficientnet backbones for object detections.

Vaishali and Singh (https://towardsdatascience.com/automating-online-proctoring-using-ai-e429086743c8) used the ResNet50 based Caffe model for online proctoring and implemented various modules such as face detection, head pose estimation, eye ball-tracking, mouth opening, and phone detection. In the performance evaluation, the authors mentioned that they have not received the expected accuracy due to issues with face alignment, and pre-processing. Table 1 presents a thorough review of the literature of online proctoring systems including modules such as face detection, object detection, and head-pose estimation.

Table 1.

Summary on existing models

Ref	Methodology	Dataset	Metrics
[20]	Multi-layer feed-forward deep neural network	Yale Faces	Accuracy-95.4%
[38]	Tree-Structured Model and Yaw angle.	Own Dataset of 39 videos each of 2 minutes duration on average	Accuracy-80%
[2]	Minimum Average Correlation Energy with SVM	OEP dataset (Prepared by own)	True Detection Rate-87% False Alarm Rate-2%
[14]	Viola-Jones Algorithm, PCA, ANN	BioID Face Database	Accuracy-94%
[49]	Caffe Model	CoCo, Kitti, Open Image	Accuracy-81%
[1]	6DoF pose in a Faster R-CNN–based framework	AFLW2000-3D, BIWI Wider Face	Easy-level − 90%; Medium-level- 89.1%; Hard-level- 83.9%
[21]	MLP for regression of pose angles.	BIWI and AFLW	Mean Absolute Error- 4.06
[43]	ResNet 50 with Cross-Entropy	AFLW2000, BIWI	Mean Average Error- 6.155
[58]	SSR-Net Based FSA-Net	300 W-LP, AFLW 2000, BIWI Dataset	Mean Absolute Error- 4.00
[16]	R-CNN is used for Feature Extraction and Histogram intersection kernel with SVM	VOC 2007, VOC 2010–12	Mean Average Precision- 62.4
[9]	Two Stream Convolution Neural Network	CASIA-FASD, REPLAY-ATTACK and OULU	Attack Presentation Classification Error Rate (APCER) and Bona Fide Presentation −3.6% Classification Error Rate − 2.2%
[34]	CNN and adaptive gradient methods	Prima Head pose dataset, AFLW and AFW datasets	Accuracy (Pitch)-73.9% Accuracy (Yaw)-66.6%
[52]	Google Net and Coarse to fine strategy	Biwi Kinect Head Pose Dataset, Pointing’04 dataset and Boston dataset	Accuracy (synthetic images)- 99.92% Accuracy (Real images) – 92.35%
[23]	GoogleNet with Multiple Regression Loss	ALFW2000 dataset	Mean absolute error of the Euler angles (EMAE) – 4.556 Mean absolute error of the quaternions (QMAE) -0.0248
[3]	POSEidon model along with Deterministic conditional GAN model	Biwi Kinect Head Pose and ICT-3DHP, and Pandora	Accuracy (Head)(Pandora Dataset) - 73.6%
[57]	Multi-Layer Perceptron with scatter max layer.	Biwi head pose dataset	MAE-1.26
[22]	Object Relation Module with Region Proposal Network	COCO dataset	mAP accuracy – 31.9
[33]	Local Binary Pattern Histogram Algorithm	Customised Dataset	Accuracy – 62%
[39]	Haar Cascade Method	Instagram Data	Accuracy −71.48%
[59]	Multi-Class Classification Network	Wider face and FDDB datasets	Face Detection Accuracy – 84.2%
[4]	CNN based cascade structure face detector	Masked Face dataset	Accuracy – 86.6%
[41]	Faster RCNN	JC2463 and AEC912 datasets	Precision −70.9% Recall −48.4% F-Measure −57.5%
[18]	Local Binary Patterns	UCCS datasets	Face detection rate- 91.2%

By considering the limitations of existing systems, authors proposed a real time system which requires to meet the objectives. Authors proposed Attentive system with less complexity and less computational time. The Attentive system identifies the face from the frame using three staged Attentive-Net. Features are extracted only from the aligned faces using the Face net algorithm which is based on inception-resnetv1 blocks. Further, the proposed system records if the examinee is seeing towards left, right, by estimating the head pose using SolvePnp Equation. The sequential layered Liveness Net Model is employed in this system to detect face spoofing.

Methodology

The proposed work is used to detect the face and used to check head alignment according to pose. Further, this system also detects the presence of the examinee and verifies the multiple faces and other accessories notification. The architecture of the proposed work is presented in Fig. 1. Figure 1 shows that the examinee’s video is continuously recorded through a webcam and then is divided into frames for further processing. As a part of the first module, all the faces within the frame are detected using the top, bottom, left, and right coordinates, and a customized border is drawn around the face for visualization. The three-stage cascaded neural network-based model Attentive-Net is used for face detection. Attentive-Net is formed by taking Multi-Task Cascaded Convolution Networks(MTCNN) [55] as base model along with naïve-inception module and face alignment verification. The frame with a minimum size of 80 × 80 is given as an input to the face detection algorithm. In our online proctoring system, face detection alone will not serve the purpose. We also need to detect the liveness and estimate the head pose of the face. To detect the liveness, first, the faces are checked for the alignment using Affine transformation matrix. For the aligned faces Inception-ResNet v1 architecture-based model, FaceNet is used to extract the features of the face. This feature extraction algorithm yields a feature vector of length 128 comprises landmark points such as nose tip, chin, left eye left corner, left mouth corner, right mouth corner, etc. From these landmarks, the spoofed face is identified and an alert is given to the examiner.

Fig. 1.

Architecture of the proposed attentive system

The same landmarks extracted by Attentive Net is used to find out rotation vector and translation vector of aligned face to estimate head pose either as left or right. The head-pose estimation is done for the live face image to avoid malpractice issues. In this process, if the examinee is taking help from any other devices or the neighborly people, the examiner is notified. Cell phones, laptops, and books in the frame are detected, and an alert is generated for the examiner. The following section describes the process of video acquisition followed by proctoring.

Video acquisition and frames extraction

In the proposed model, a webcam is used to continuously stream and capture the video of the examinee and convert them into frames. Furthermore, keyframes are extracted based on the motion detection or according to the changes in the visualization. Malpractices may be found even without major changes in motion and visualization. So, in our work, instead of extracting only keyframes, all the frames are processed. For all the frames, faces are detected by using the following process.

Face detection using Attentive-Net

All the frames detected are inputted to the Face Detection module. Face detection is used to detect the frontal faces from the frames captured through the webcam. Attentive-Net is used to detect the faces from the frame and resized into different scales and a scale pyramid is formed. Further, the image is smoothed with a sequence of smoothing filters, which has a radius twice the radius of before. The smoothing is carried with the deferrable smoothing filter f is mentioned in Eq. (1).

$$\widehat{s}=\sum _{m=-2}^2\sum _{n=-2}^{2}f\left(m,n\right).s_0\left(i-m\right)\left(j-n\right)$$ 1

Further, down sampling is applied to reduce the frame size by half of its size with smoothing.

$$s_{l+1}\left(i,j\right)=\sum _{m=-2}^2\sum _{n=-2}^{2}f\left(m,n\right).s_l\left(2i-m\right)\left(2j-n\right)$$ 2

Considering each image has only one face and multiple landmarks in real world, each image is represented as (x_n,y_n, z_n), n = 1,2,…,N. Here x_n represents the n-th image sample, y_n = c, c = 0,…,C-1 is Face label and_, z_n = [$z_{n,\dots ,}^1{z}_n^M$]^T is the facial landmarks for the n-th image sample. If the n-th image sample has the m-th facial landmark is set as $z_n^m$ = 1, otherwise $z_n^m$ = 0. Therefore a given dataset is denoted as (X,Y,Z) = {(x_n,y_n, z_n), n ∈{1,2,…,N}}. Three stages (not directly connected) are implemented on a scale pyramid, but the outputs of the earlier stage are fed as input to the next stage.

In stage 1, a full CNN called Naive-Proposal network (NP-net) is applied to extract facial windows and bounding box regression vectors. For each scaled frame, a 12 × 12 stage 1 kernel scans the image from the top left corner (0,0) to (12,12). In this connection, any face we find will be tracked. To achieve the real time performance, the same process is repeated with stride 2 reducing the number of operations to a quarter of the original. Within each of these kernels, 3 convolutions are run through with 3 × 3 kernels along with Naive Inception module. The naive inception module consists of NP-net contains 1 × 1 convolution, 3 × 3 convolution, 5 × 5 convolution along with max-pooling. For each convolution and max-pooling 16 × 16 filter is used. The final concatenated output results in 64 channels. The output of all these is concatenated and is given as input to the third convolution layer. After every convolution layer, the Parametric Rectified Linear Unit (PReLU) layer is added to multiply every negative pixel with alpha. To remove every other pixel and leave only the largest one, the max-pooling is added after the first PReLU layer. The network is split into two layers after the third convolution layer. The activations are passed on to both layers (convolution 4–1, convolution 4–2) but, the SoftMax function is applied to only one of the above convolution layers (convolution 4–1). The SoftMax adds up to the probability that there is a face or there is not a face. The convolution 4–2 outputs the coordinates of the bounding boxes. For each candidate box, we predict the offset between it and the nearest ground truth (i.e., bounding boxes left, top, height and width). As the learning objective is formulated as regression problem, Euclidean loss for each sample is implemented as below:

$$\text{Loss}={‖\widehat{p}\left(y\right)-p\left(y\right)‖}_2^2$$ 3

Where, $\widehat{p}\left(y\right)$ is the regression target obtained from network and p(y) is the ground truth coordinate. This results in overlapped bounding boxes with different confidences. Non-maxima suppression is implemented on all the bounding boxes of all kernels to remove bounding boxes with low confidence. For all the bounding boxes (b_oi) with different confidences (C_i), remove the bounding boxes with overlap (IoU ≥ NMS_t), where IoU: Intersection over Union, M: argmax of C, NMS_t: NMS threshold,

$$\mathit{IoU}\left(M,b_{\mathit{oi}}\right)\ge \mathit{NM}{S}_t,\text{where}{B}_o\leftarrow B_o–b_{\mathit{oi}};C\leftarrow C-C_i$$ 4

NMS threshold is considered as 50%. All bounding boxes are reshaped into a square in stage 1. The complete structure of the process can be explained with the following algorithm.

Algorithm 1.

NP-NET

NR-Net (Refine Network) is similar to NP-Net with more layers. In stage 2, NR-Net padding is applied to reshape out-of-bound boxes and also filter them. For the bounding boxes related to occluded or partial faces, padding is carried with zeros. NR-Net has 3 convolutions run through with 3 × 3 kernels along with Naive Inception module. The naive inception module consists of NP-net contains 1 × 1 convolution, 3 × 3 convolution, 5 × 5 convolution along with max-pooling. For each convolution and max-pooling 48 × 48 filter is used. The final concatenated output results in 192 channels. The output of all these is concatenated and is given as input to the third convolution layer. All the bounding boxes obtained from stage 1 are resized to 24 × 24 and their values are normalized between −1 and + 1. Similar to NP-Net, NR-Net also splits into two layers which returns classification and bounding boxes with confidence as shown in the following algorithm. Bounding boxes with low confidence are further deleted using the Non-maxima suppression (NMS) logarithm.

Algorithm 2.

NR-NET

In stage 3, Output Network (O-Net), is implemented for the out-of-bound boxes with padding as shown below.

Algorithm 3.

O-NET

The O-Net also splits into three layers which output the probability of face being in the box, coordinates of the bounding boxes, and the coordinates of the facial landmarks. In this stage, all the boxes are resized to 48 × 48 and then five facial landmarks like eye tips, nose tips, and lips tips are extracted. Finally, classification is applied to categorize the features as either face or no face. The binary cross entropy loss function is used for binary classification as per Eq. (5) given below, where t(x) is the probability produced by the network that indicates sample to be a face. The notation p(x) ∈{0,1} denotes ground-truth label.

$$L_b=-\left(\mathrm{p}\left(\mathrm{x}\right).logt\left(x\right)+\left(1-p\left(x\right)\right).log\left(1-t\left(x\right)\right)\right)$$ 5

The face landmarks are given as an output from the cascaded model. Different types of training images in learning process like face, nonface and partially aligned face are considered. In this case, Euclidean loss or Cross entropy loss are not alone used. The cross-entropy loss function is implemented for the sample of background region. This can be implemented directly with a sample type indicator as below:

$$Min{\sum }_{i=1}^N{\sum }_{j\in \left\{det,\mathrm{box},\text{landmark}\right\}}{\alpha }_{j,}{\beta }_i^j,L_i^j$$ 6

Where, N is the number of training samples and α_j, denotes on the task importance, L_i is the Euclidean Loss. We use α_det = 1, α_box = 0.5, α_landmark = 0.5 in NP-Net and NR-Net, while α_det = 1, α_box = 0.5, α_landmark = 1 in output network. ${\beta }_i^j$ ∈{0, 1} is the sample type indicator. In these cases, stochastic gradient descent is used to train the CNNs. Figure 2 shows the network architecture of the proposed cascaded model. The following section describes the extraction of landmarks of the faces detected from three staged networks of Attentive-Net.

Fig. 2.

Architecture of proposed Attentive-Net.

The first block of the architecture is the Naïve-proposal network which results in the classified faces and bounding boxes. The second block of the architecture is Naïve-Refine Network which performs further Non-Maxima Suppression and results in face classification and bounding boxes. The third block is Output Network which results in Face Classification, Bounding boxes and Landmarks Localization.

Landmarks localization

The extracted landmarks are localized to verify the alignment of the face image which is a submodule of Attentive-Net. The process continues only if the face detected is aligned otherwise alert is generated for the examiner by aborting the process. Generally, the accuracy of the facial alignment depends on the feature’s extraction. To check the face alignment, the landmarks obtained from the three-staged networks of Attentive-Net are considered as original points. The average of the face positions according to the face points is considered as the base point. Further, similarity transform is used to compute the difference between original and base points using the Frobenius norm as in Eq. (7).

$${‖B‖}_F=[{\Sigma }_{i,j}\mathrm{abs}{\left(b_{i,j}\right)}^{2]\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2$}\right.}$$ 7

The norm values of original and base matrices are used to calculate the covariance using Eq. (8):

$$\mathit{Cov}(B_{o,}{B}_{b)}=\Sigma \left(\left(B_{\mathit{oi}}–B_{\mathit{\text{omean}}}\right)\ast \left(B_{\mathit{bi}}-B_{\mathit{\text{bmean}}}\right)\right)/\mathit{ND}$$ 8

where B_o is Frobenius norm of the original matrix; B_{o mean} is the mean of Frobenius norm of Original matrix; B_{b mean} is mean of Frobenius norm of Base matrix, and ND is the number of data variables. Then, the singular value decomposition (SVD) of covariance such that UDV^T = SVD (Covariance) is computed, where U and V are the orthonormal left and right matrices of singular vectors, respectively. D is a diagonal matrix of decreasing positive singular values. Thereafter, the affine matrix is computed using u, d, and v values. Here, only the rotation about the base points is carried by using the affine transformation matrix using Eq. (9).

$$\left[\begin{array}{ccc}\mathit{\text{cosθ}}& sin\theta & 0\\ -sin\theta & \mathit{\text{cosθ}}& 0\\ 0& 0& 1\end{array}\right]$$ 9

The rotation matrix is calculated with pixel mapping is given in Eq. (10).

$$R^{\prime }=\left[\begin{array}{c}xcos\theta -y\mathit{\text{sinθ}}\\ xsin\theta +y\mathit{\text{cosθ}}\end{array}\right]$$ 10

The affine transformation is applied on the original points and the length of the aligned face is 160. All the aligned faces of a frame are marked, and feature extraction is done only if a single aligned face is detected in a frame. The algorithm for the above-described process is as shown below. The input given to the Face alignment algorithm is the landmarks of the faces extracted by Attentive-Net. L1 are the landmark positions. A1 are the Average Face Positions.

Algorithm 4.

Face Alignment

In (https://towardsdatascience.com/automating-online-proctoring-using-ai-e429086743c8), the process is implemented for all the faces detected in the frame which increases the complexity. The examinee’s face will be aligned during the examination and if not, an alert can be given to the proctor and further processing for non-aligned face can be terminated. FaceNet is a kind of deep neural network [44] applied to extract the features from a frame and it complies with inception_resnetv1 architecture as shown in Fig. 3. It is a 22-layered deep neural network. Face Net uses a 1 × 1 filter for dimensionality reduction. Along with these filters, 3 × 3, 5 × 5, and 7 × 7 filters are implemented parallelly to down sample the image by using max pooling. There is a chance of vanishing gradient problems during backpropagation. Therefore, two auxiliary outputs are tapped at middle layers and added to total loss by taking a weighted average. The facial points or fiducial points identified using FaceNet vary between 5 and 78. The minimum number of points to be identified is 5. In (https://towardsdatascience.com/automating-online-proctoring-using-ai-e429086743c8), 68 facial points or fiducial points were identified. FaceNet compresses a face into a vector of 128 Dimensions by calculating the distance between these facial points. The 128-dimensional embedded vector contains the numbers which represent the important features of the face. The specific feature of FaceNet is its Triplet Loss Function as well as SoftMax activation with cross-entropy loss function, as represented in Eq.(11).

$$\mathit{\text{Cost Function}}=\sum _i^N\mathit{\text{Triplet Loss Function}}+L2\mathit{\text{Regularizatio}}\mathrm{n}$$ 11

Fig. 3.

Inception-Resnet V1 architecture for FaceNet

The embedded vector acts as a hash code of the face. This vector of 128 dimensions which is the distances between 68 specific points (landmarks) is used for identifying the spoofed face as explained below.

Face spoofing and multiple face detection

Face spoofing and multiple face detection is a crucial phase in an online proctoring system. In this phase, an examiner can understand whether the examinee is spoofing or any other person helping the examinee. In (https://towardsdatascience.com/automating-online-proctoring-using-ai-e429086743c8), the spoofed face is not identified. All the faces identified in the frame are considered real and the remaining process is implemented. But there will be a great probability of spoofing the faces during an examination. Face spoofing is done by notifying the confidence level of the liveness of the face.

As shown in Fig. 4, the Face Detection module detects the face in the frame, if at least one face is aligned then it extracts the features from the aligned faces. The vector of 128 dimensions obtained from landmark localization which is unique for all faces is used for Liveness Model. The localized landmarks distance between the eye tips, distance between mouth tips, distance between eye tips and nose tip, distance between nose tip and mouth tip and the distance between mouth tips and chin from the Face Net are considered for the liveness detection algorithm. The procedure of extracting landmarks using Face net and detecting the liveness is described in the following algorithm.

Fig. 4.

Liveness detection. After detecting faces in the frame, face alignment is verified. Features are extracted for the aligned faces. The extracted features are used for liveness detection

Algorithm 5.

Liveness Detection with Face Net

As liveness detection is implemented to only aligned faces, false positives can be reduced. A sequential layered model is considered as base model on which liveness model is designed. The liveness model is a simple and shallow CNN with very few parameters to minimize the chances of overfitting small datasets. It exhibits VGGNet-esque qualities. Our CNN is very shallow with only a few learned filters. All the images loaded are resized into 32 × 32. All the pixel intensities are scaled to the range [0,1]. Two-layered sets of conv = > ReLU = > conv = > ReLU = > Pool are implemented along with batch normalization and dropout. Finally, fully connected and ReLU activated layers with a SoftMax are added. The complete summary of the model is shown in Fig. 5.

Fig. 5.

Liveness net model summary

In this model, the ReLU activation function is implemented, and the pool size is taken as 2 × 2 all-time for Max Pooling 2D. Later, different activation functions are used for testing. The SoftMax classifier is considered which returns the probabilities.

$$\sigma {\left(\stackrel{\rightharpoonup }{Z}\right)}_i=\frac{e^{z_i}}{{\sum }_{j=1}^k{e}^{z_j}}$$ 12

If the probability returned by the liveness model is less than 0.5, the detected face is considered as fake, i.e., spoofed face else, the face is identified as a real face. An alert will be given to the proctor if a spoofed face is identified. If more than one real face is detected, our system gives an alert to the proctor that multiple faces are detected. To reduce the computational complexity, the head pose is estimated only for the faces identified as real as described below.

Head-pose estimation

Examinees may take help from others who are in the room or also they can also take help from other resources like laptops, systems placed in the room. This kind of misbehavior can be identified by estimating the head pose orientation. The head pose is estimated only for the aligned faces. If the examinee is seeing continuously other directions like left and right, with a certain angle, an alert is sent to the proctor. In [2] (https://towardsdatascience.com/automating-online-proctoring-using-ai-e429086743c8), gaze tracking is implemented instead of head pose estimation. The eye-ball detection of the gaze-tracking does not give accurate results if the examinee is using spectacles. So, in our system, we are going with head-pose estimation using an affine rotation matrix. Step by step process involved in the head pose estimation is as follows:

1. Attentive net model is used to compute the landmarks such as Nose Tip, Chin, Left eye left corner, Right eye right corner, Left mouth corner, Right mouth corner.

2. Further, model points for 3D projection are also estimated. Camera Intrinsic parameters as below are also given as an input to the SolvePnp function. f_x and f_y, are focal lengths that are expressed in pixel units and a principal unit (c_x, c_y) is included in the matrix.

$$\left[\begin{array}{ccc}{f}_x& 0& c_x\\ 0& f_y& c_y\\ 0& 0& 1\end{array}\right]$$ 13

• 3.SolvePnp function is used to compute the Direct Linear Transform (DLT) solution followed by Levenberg-Marquardt optimization. DLT is used to solve a set of variables from a set of similarity relations like below:

$$x_i=A{y}_i\mathit{\text{where}},A=X{Y}^T{\left(Y{Y}^T\right)}^{-1}$$ 14

The Levenberg-Marquardt optimization algorithm starts with an initial guess g_0, where g is adjusted by only δ exclusively for downward steps as follows:

$$\left(J^{T}J+\mathit{\lambda I}\right)\mathrm{\delta }=J^{T}r$$ 15

where J is Jacobian Matrix, λ and is Damping parameter and r is a Residual vector.

• 4.A 3 × 3 Rotation Matrix (R`) and 3 × 1 translation vector (t) are calculated is as follows

$$P_c=\left[\begin{array}{cc}{R}^{`}& t\\ 0& 1\end{array}\right]P_w$$ 16

$$\left[\begin{array}{cc}{R}^{`}& t\\ 0& 1\end{array}\right]=\left[\begin{array}{cccc}{r}_{11}& r_{12}& r_{13}& t_x\\ r_{21}& r_{22}& r_{23}& t_y\\ r_{31}& r_{32}& r_{33}& t_z\\ 0& 0& 0& 1\end{array}\right]$$ 17

• 5.Plot the translation and rotation vector values in the view of the nose end tip.

From the above procedure, the head movement towards left and right can be identified. If the head is turned either to left or right, then an alert is given to the proctor. The above steps show that an alert will be given to the proctor either if face spoofing is identified for the aligned faces or if the head of real face is rotated towards left or right.

Experimental results and discussions

In this chapter, the authors discussed dataset statistics, evaluation procedures of the proposed method, discussion on experimental results, and comparative analysis with other methods. The proposed system is implemented on Keras with TensorFlow as the backend. In this work, each frame is captured video with a minimum size of the frame as 80*80 and then it will be given as an input to the system.

About dataset

The performance of the proposed model is evaluated on a dataset that comprises 200 videos with various combinations of malpractice behaviors. Attentive-Net is pre-trained on the CASIA-Web Face dataset which consists of 453,453 images to assess the liveliness of the examinees. Further, Liveliness is evaluated on a custom dataset with 200 images, where 100 images are fake and the remaining 100 images are real. Our proposed model is also evaluated on Computational Intelligence and Photography Lab (CIPL) dataset, which has 960 fake and 1081 real face samples. Figure 6 shows the sample fake and real images. The performance of the proposed method is evaluated by considering 75% samples for model building and the remaining 25% of samples are used to assess the model performance.

Fig. 6.

Sample real and fake images of CIPL and custom datasets

System evaluation

The input images extracted from frames are resized to 32*32 and are inputted to Attentive-Net. Firstly, acquired face image was duly aligned and then mean positions of face for five landmarks considered to determine the angle of rotation are x = [0.224152, 0.75610125, 0.490127, 0.254149, 0.726104] and y = [0.2119465, 0.2119465, 0.628106, 0.780233, 0.780233], respectively. Further, the affine matrix and the face position (left, right, center) details are computed. If at least one sample image is registered as aligned, then the FaceNet Algorithm is implemented to extract 128 dimensional features along with the computation of liveliness detection and estimating head pose. The performance of the proposed Attentive system is evaluated in both alignments: with and without face alignment. The experimentation shows that the feature extraction without face alignment has a high ratio of false positives. If the alignment process showing the examinee’s face is not captured, immediately proctor is sent an alert instead of feature extraction. If the examinee is writing the examination, through the camera we can find at least one face image, and a further aligned face can be identified. The Attentive system yielded a 14.67% of false-positive ratio (FPR) without face alignment which was reduced to 11.65% with face alignment. Similarly, False Negative Ratio (FNR) without face alignment is 15.45% and with alignment, it was reduced to 11.29%.

During model building learning rates are varied from 1e-4 to 5e-4 in the proposed work and it is observed that the model performance is stable at 1e-4 learning rate, batch size as 10 and number of epochs as 60. But decay in performance is observed at the learning rates of 2e-4, 3e-4, 4e-4, and 5e-4. Table 2 shows the performance of the proposed model with Adam as an optimizer with varied learning rates and batch sizes.

Table 2.

Accuracy analysis of various hyper parameters of CIPL dataset

Learning rate	Batch size	Accuracy	Precision	Recall
1e-10	8	0.63	0.65	0.74
1e-8	8	0.66	0.66	0.73
1e-8	10	0.71	0.74	0.82
1e-6	8	0.72	0.73	0.89
1e-6	10	0.75	0.78	0.9
1e-4	8	0.81	0.86	0.96
1e-4	10	0.86	0.9	0.97
1e-2	8	0.82	0.82	0.89
1e-2	10	0.87	0.88	0.89
1e-2	5	0.85	0.89	0.96
2e-4	8	0.73	0.76	0.86
3e-4	8	0.74	0.79	0.98
4e-4	8	0.76	0.81	0.83
5e-4	8	0.72	0.75	0.88

The performance of the proposed method is evaluated with various activation functions like Sigmoid, Hyperbolic Tangent, ReLu, and Leaky ReLu in the face detection module. Table 3 shows the performance of the proposed model in terms of Accuracy, Precision, Recall, and Loss; it also shows that the Sigmoid achieved lower accuracy when compared with SoftMax in the binary classification. The results show that the performance of ReLu and Leaky ReLu are almost the same. The experimentation shows that ReLu addresses well the gradient vanishing problem and is comparatively less computationally expensive. During the training process, it is observed that the training time corresponding to each face detection increases with increase in number of training samples. Conversely, the no. of training sample increases, the test time corresponding to each face detection decreases.

Table 3.

Evaluation of model based on activation functions on CIPL dataset

Activation function	Accuracy	Precision	Recall	Test loss
Sigmoid	0.71	0.75	0.82	0.09
SoftMax	0.80	0.86	0.95	0.05
Hyperbolic Tangent	0.74	0.80	0.94	0.09
ReLu	0.81	0.86	0.96	0.05
Leaky ReLu	0.79	0.85	0.96	0.08

Further, Figs. 7 and 8 shows the performance of the Attentive system on how it identifies the malpractice behavior of the examinee with the level of confidence. For each captured face image, face alignment is carried, and then liveliness is computed continuously for 30 frames.

Fig. 7.

Face detection with confidence

Fig. 8.

Liveliness detection

The SolvePnp equation is used to compute the rotation and translation vectors of each live face sample using the nose endpoint as illustrated in Fig. 9. To identify the left and right positions of the head image, the translation and rotation vectors were set as (40, 20) and (40, −20).

Fig. 9.

Head pose estimation

Our system also detected various objects like a mobile, book, laptop as shown in Fig. 10. An alert will be generated by our system if any of these objects are detected. Figure 11 shows the working of head-pose estimation. Our model checks for the aligned face and continues the further steps. Head pose is estimated only when at least one face is aligned. As the face is not aligned in the second part of Fig. 11, the Head pose is not estimated. In the remaining parts of the Figure, it is observed that the head pose towards the right, left and center can be detected by using the vector values displayed at the nose tip.

Fig. 10.

Mobile/book/laptop detection

Fig. 11.

(From Left to right) Head Right, not aligned face, Head Left, Head straight

Comparison of different proctoring approaches

The performance of the proposed method is compared with the Vardhaman AI Proctoring system (https://towardsdatascience.com/automating-online-proctoring-using-ai-e429086743c8), Automated Online Proctoring system [2] as in Table 4. It is evident that the precision value of the proposed method is 0.81 and it is superior to the other two methods as shown in Fig. 12. The Vardhaman AI_Proctoring is limited in the recognition of live faces efficiently in the dim light when compared to our proposed approach. These existing systems do not verify the face alignment during the verification of face spoofing. Considering our Attentive system as a Computer Vision Problem, our system is compared with existing system [2] in different scenarios, Attentive system performs well in face detection over Viola-Jones. Also Attentive-Net estimates head-pose more accurately using Direct Linear Transform over Lucas-Kanade method used in [2]. Attentive system uses YOLOv3 model to detect various gadgets and books which can be used for malpractice.

Table 4.

F1 score for existing systems and proposed system on the custom dataset (100 videos)

Method	Accuracy	Precision	Recall	F1 score
Vardhaman AI_Proctoring	0.66	0.65	0.86	0.65
Automated Online Proctoring	0.64	0.60	0.85	0.64
Attentive System (Proposed)	0.87	0.81	0.96	0.83

Fig. 12.

Precision-recall curve over custom dataset (100 videos) for proposed attentive system and existing systems

The proposed research is also evaluated on a dataset with video samples having 15% of mischievous behavior when compared with cheating activities and the remaining section have regular activities. The mischievous actions include seeking help from others, usage of mobile phone, substituting the examinee, and moving away from the camera, etc. The performance of the proposed model is evaluated in two different scenarios like segment-based and instance-based metrics [2]. Both these metrics are determined using True Detection Rate (TDR) and False Alarm Rate (FAR). The mischievous behavior of the examinee is assessed using the following formulae:

$$\text{Instance}-\text{based}\mathrm{TDR}=\frac{\mathrm{\Sigma i}\#\text{detected cheating instances}}{\mathrm{\Sigma i}\#\text{cheating instances}}$$

$$\text{Segment based}\mathrm{TDR}=\frac{\mathrm{\Sigma i}\#\text{detected cheating segments}}{\mathrm{\Sigma i}\#\text{round truth cheating segments}}$$

$$\text{Instance}-\text{based}\mathrm{FAR}=\frac{\mathrm{\Sigma i}\#\text{false cheat instances}}{\mathrm{\Sigma i}\#\text{of cheat}-\text{free minutes}}$$

$$\text{Segment based}\mathrm{FAR}=\frac{\mathrm{\Sigma i}\#\text{false cheat segments}}{\mathrm{\Sigma i}\#\text{of cheat}-\text{free segments}}$$

By using the same process, two existing methodologies are compared with our system as shown in Fig. 13. Observably, when compared to the Automated Online Proctoring system [2] and Vardaman AI Proctoring, our system performs better both in the segment-based and instance-based metrics. As Vardaman AI Proctoring does not check for the face alignment, FAR are high. These observations show that our system gives FAR less compared to these systems and the True detection rate is high compared to other systems.

Fig. 13.

a Instance-based comparison b Segmentation-based comparison of FAR with TDR.

The performance of the proposed method is compared with Vardhaman AI Proctoring with the combination of various optimizers and activation functions. Table 5 shows further results. It is observed that the loss curve fits better with the Adam Optimizer when compared with other loss functions such as cosine, hinge, and mean squared logarithmic error. The proposed model maintains 0.86 accuracies for the cosine proximity and Kullback Leibler Divergence loss functions. Also, the loss function affects the learning rate. The characteristics of various optimizers are as follows; Adam needs less memory, Ada max is based on infinity norm, Adagrad depends on parameter specific learning rate and Adadelta is a stochastic gradient descent method. The existing Vardhaman AI proctoring system uses Adam delta optimizer, with the batch size 10 and learning rate 1e-4. The Vardhaman AI proctoring system is a ResNet50 based model which failed to extract embeddings for several faces from images with high Gaussian blur. It is also not able to vectorize poor quality images which can be done by the face net.

Table 5.

Performance comparison of the proposed method with Vardhaman AI in tune with various hyperparameters on the CIPL dataset

Model	Loss Function	Opt = Adam, LR = 0.0001			Opt = Adam delta, LR = 0.0001			Opt = Adamax, LR = 0.0001			Opt = SGD, LR = 0.0001			Opt = Adagrad, LR = 0.0001
		A	P	R	A	P	R	A	P	R	A	P	R	A	P	R
Vardhaman_AI	Binary Cross Entropy	0.66	0.65	0.86	0.62	0.65	0.73	0.63	0.63	0.72	0.63	0.64	0.75	0.68	0.65	0.87
	Poisson	0.75	0.8	0.87	0.69	0.71	0.73	0.7	0.72	0.85	0.6	0.63	0.84	0.61	0.62	0.64.
	Cosine Proximity	0.66	0.68	0.69	0.62	0.63	0.62	0.63	0.64	0.77	0.71	0.73	0.82	0.6	0.62	0.62
	Kullback Leibler Divergence	0.66	0.66	0.7	0.62	0.63	0.69	0.61	0.62	0.72	0.69	0.6	0.7	0.6	0.62	0.77
	Hinge	0.63	0.68	0.86	0.63	0.66	0.72	0.61	0.62	0.61	0.58	0.59	0.68	0.61	0.63	0.74
	Mean squared logarithmic error	0.62	0.64	0.79	0.6	0.62	0.82	0.61	0.64	0.79	0.6	0.61	0.7	0.6	0.62	0.79
Proposed Method	Binary Cross Entropy	0.81	0.86	0.96	0.8	0.82	0.83	0.83	0.83	0.92	0.83	0.84	0.85	0.82	0.85	0.89
	Poisson	0.85	0.9	0.97	0.79	0.81	0.83	0.8	0.82	0.85	0.8	0.83	0.94	0.81	0.82	0.84.
	Cosine Proximity	0.86	0.88	0.89	0.82	0.83	0.82	0.83	0.84	0.87	0.81	0.83	0.84	0.8	0.82	0.82
	Kullback Leibler Divergence	0.86	0.86	0.9	0.82	0.83	0.89	0.81	0.82	0.82	0.79	0.8	0.9	0.8	0.82	0.87
	Hinge	0.83	0.84	0.86	0.83	0.86	0.92	0.81	0.82	0.81	0.78	0.79	0.98	0.81	0.83	0.84
	Mean squared logarithmic error	0.82	0.84	0.89	0.8	0.82	0.92	0.81	0.84	0.89	0.8	0.81	0.9	0.8	0.82	0.89

The average processing time of different modules like Face Detection, Head Pose Estimation, Phone Detection, Face Spoofing and Multiple person detection are compared and the results are presented in the Table 6. Apparently, introducing the face alignment module into our Attentive system saves the time of Head Pose Estimation significantly.

Table 6.

Average processing time (ms)

Model	Face Detection module	Face Alignment	Head Pose Estimation	Phone Detection	Face Spoofing	Multiple person detection
Vardhaman AI_Proctoring	110	NA	125	238	NA	112
Automated Online Proctoring	120	NA	125	238	NA	122
Attentive System	98	110	109	236	120	100

From the above discussions, it is observed that our Attentive system superior to existing methods in face detection, pose estimation and phone detection which can be embedded to any platform/software for online proctoring with additional services or features such as User Verification and Active Window Detection to avoid some more malpractice.

Discussion on challenges

The face detection module identifies an individual only if the complete face is bounded in the frame. In the face detection module, as the partial or occluded faces are not properly identified, an alert cannot be sent to the proctor. In some of the cases, those partial faces could be of people other than the examinee who may be extending their help to the examinee but are not identified. Besides, the examinee may spoof his/her face. The liveness detection model identifies the face spoofing and also alerts the proctor about the mischievous behavior of the examinee. We are training the model with images of different poses and expressions. If the faces with neutral expressions are categorized into spoofed images and faces with some expressions are categorized into live images, then the examinee who is with neutral expression while writing examination is also identified as spoofed image. The examinee’s face is in the same position continuously for a long duration is identified as a spoofed image as there is no facial movement or expression. The accuracy in these terms can be improved by categorizing the datasets with more combinations. Along with liveliness detection, face verification also needs to be done. If the examinee’s face is partially covered with a mask or hand then the pose is not estimated properly, as all facial features are not extracted accurately. The marking of the pose angles will also be disturbed due to missing facial features.

The liveness detection and estimation of the head pose may not be sufficient as the examinee may seek help from other persons or devices located in the room. The examinee may not rotate his/her head but moves eyes to seek help. These types of malpractice behaviors can be detected by using eye-gaze tracking. Without changes in the position of the head and eyes, the examinee can also speak to others to seek help that is not identified. Our Attentive system is limited to the recognition of some mischievous behaviors of the examinees. The proposed online proctoring system works efficiently in the place of a human proctor. Our proposed proctoring system detects the faces from live video, identifies the face spoofing, detects cell phones, laptops, books and estimates the head pose. Our system helps proctors to identify the malpractices carried by the examinees. The experimental results prove that the proposed system better performs when compared to the existing Proctoring system. The performance of the proposed system can be improvised in identifying if the examinee is speaking to someone else during an examination.

Conclusion

The main objective of this paper is to develop a well-rounded automation system that is capable of helping the proctor to monitor the students attending an online examination. Out of the several proposed features of the system, our paper has developed the ability to do multiple-person detection, face spoofing, and head pose estimation. While these form important milestones, the ultimate objective is to develop the automation system. Our system reduced the hardware and labor cost as the process is implemented only for aligned faces. Our model is trained in such a way to detect faces from frames with poor quality. The experimental results demonstrated that our automation system performs better with a precision value of 0.81 and recall of 0.91. Various cheating behaviors are also identified with 0.86 accuracy, where accuracy for existing systems is around 0.64. The model also performs better with various Hyperparameters. At the moment, the scope of the project is restricted to video Processing. It is hoped that the system can also be implemented using the examinee identity verifications, mouth opening detection mechanisms, and other audio capturing tools.

Declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Ethical approval

This article does not contain any studies with human participants performed by any of the authors.