Abstract
Analyzing a vehicle’s abnormal behavior in surveillance videos is a challenging field, mainly due to the wide variety of anomaly cases and the complexity of surveillance videos. In this study, a novel intelligent vehicle behavior analysis framework based on a digital twin is proposed. First, detecting vehicles based on deep learning is implemented, and Kalman filtering and feature matching are used to track vehicles. Subsequently, the tracked vehicle is mapped to a digital-twin virtual scene developed in the Unity game engine, and each vehicle’s behavior is tested according to the customized detection conditions set up in the scene. The stored behavior data can be used to reconstruct the scene again in Unity for a secondary analysis. The experimental results using real videos from traffic cameras illustrate that the detection rate of the proposed framework is close to that of the state-of-the-art abnormal event detection systems. In addition, the implementation and analysis process show the usability, generalization, and effectiveness of the proposed framework.
Key words: digital twin, deep learning, vehicle detection, abnormal behavior
Footnotes
These authors contributed equally to this work.
References
- [1].Jiang F, Wu Y, Katsaggelos A K. Abnormal event detection from surveillance video by dynamic hierarchical clustering; San Antonio, TX: IEEE; 2007. pp. V145–V148. [Google Scholar]
- [2].Wang J Q, Li H P. Intelligent analysis of vehicle illegal behavior based on deep learning algorithm. Journal of Shanghai Ship and Shipping Research Institute. 2019;42(4):49–54. [Google Scholar]
- [3].Fuller A, Fan Z, Day C, et al. Digital twin: Enabling technologies, challenges and open research. IEEE Access. 2020;8:108952–108971. doi: 10.1109/ACCESS.2020.2998358. [DOI] [Google Scholar]
- [4].Kumar S A P, Madhumathi R, Chelliah P R, et al. A novel digital twin-centric approach for driver intention prediction and traffic congestion avoidance. Journal of Reliable Intelligent Environments. 2018;4(4):199–209. doi: 10.1007/s40860-018-0069-y. [DOI] [Google Scholar]
- [5].Ham Y, Kim J. Participatory sensing and digital twin City: Updating virtual City models for enhanced risk-informed decision-making. Journal of Management in Engineering. 2020;36(3):04020005. doi: 10.1061/(ASCE)ME.1943-5479.0000748. [DOI] [Google Scholar]
- [6].JOCHER G, NISHIMURA K, MINEEVA T, et al. YOLOv5 (2020) [EB/OL]. (2020-07-10) [2021-01-30]. https://github.com/ultralytics/YOLOv5.
- [7].Wang C Y, Mark Liao H Y, Wu Y H, et al. CSPNet: A new backbone that can enhance learning capability of CNN; Seattle, WA: IEEE; 2020. pp. 1571–1580. [Google Scholar]
- [8].Lin T Y, Dollár P, Girshick R, et al. Feature pyramid networks for object detection; Honolulu, HI: IEEE; 2017. pp. 936–944. [Google Scholar]
- [9].Wang W H, Xie E Z, Song X G, et al. Efficient and accurate arbitrary-shaped text detection with pixel aggregation network; Seoul: IEEE; 2019. pp. 8439–8448. [Google Scholar]
- [10].Lin T Y, Maire M, Belongie S, et al. Microsoft COCO: Common objects in context; Cham: Springer; 2014. pp. 740–755. [Google Scholar]
- [11].Wen L Y, Du D W, Cai Z W, et al. UA-DETRAC: A new benchmark and protocol for multi-object detection and tracking. Computer Vision and Image Understanding. 2020;193:102907. doi: 10.1016/j.cviu.2020.102907. [DOI] [Google Scholar]
- [12].PUNN N S, SONBHADRA S K, AGARWAL S, et al. Monitoring COVID-19 social distancing with person detection and tracking via fine-tuned YOLO v3 and Deepsort techniques [EB/OL]. [2021-01-30]. https://arxiv.org/pdf/2005.01385.pdf.
- [13].Mikhail E M, Bethel J S, Mcglone J C. Introduction to modern photogrammetry. New York: John Wiley; 2001. [Google Scholar]
- [14].Hussein A, García F, Olaverri-Monreal C. ROS and unity based framework for intelligent vehicles control and simulation; Madrid: IEEE; 2018. pp. 1–6. [Google Scholar]
- [15].Zhao C H, Li Y W, Shi X, et al. Recognition of abnormal vehicle behaviors using combined features; Reston, VA: ASCE; 2019. pp. 2180–2186. [Google Scholar]
- [16].Chen Z J, Wu C Z, Huang Z, et al. Dangerous driving behavior detection using video-extracted vehicle trajectory histograms. Journal of Intelligent Transportation Systems. 2017;21(5):409–421. doi: 10.1080/15472450.2017.1305271. [DOI] [Google Scholar]
- [17].Doshi K, Yilmaz Y. Fast unsupervised anomaly detection in traffic videos; Seattle, WA: IEEE; 2020. pp. 2658–2664. [Google Scholar]
- [18].Garcia-Aunon P, Roldán J J, Barrientos A. Monitoring traffic in future cities with aerial swarms: Developing and optimizing a behavior-based surveillance algorithm. Cognitive Systems Research. 2019;54:273–286. doi: 10.1016/j.cogsys.2018.10.031. [DOI] [Google Scholar]
- [19].Wahyono J K H. Cumulative dual foreground differences for illegally parked vehicles detection. IEEE Transactions on Industrial Informatics. 2017;13(5):2464–2473. doi: 10.1109/TII.2017.2665584. [DOI] [Google Scholar]
- [20].Xie X M, Wang C Y, Chen S, et al. Real-time illegal parking detection system based on deep learning; New York: ACM; 2017. pp. 23–27. [Google Scholar]
- [21].Cai Y F, Wang H, Chen X B, et al. Trajectory-based anomalous behaviour detection for intelligent traffic surveillance. IET Intelligent Transport Systems. 2015;9(8):810–816. doi: 10.1049/iet-its.2014.0238. [DOI] [Google Scholar]
- [22].Wang C, Musaev A, Sheinidashtegol P, et al. Towards detection of abnormal vehicle behavior using traffic cameras; Cham: Springer; 2019. pp. 125–136. [Google Scholar]
- [23].Wang G A, Yuan X Y, Zheng A T, et al. Anomaly candidate identification and starting time estimation of vehicles from traffic videos; Long Beach, CA: IEEE; 2019. pp. 382–390. [Google Scholar]