Abstract
The evolution of artificial intelligence has enabled the creation of hyper‐realistic synthetic images and videos, significantly complicating the authentication of media evidence. Many algorithmic approaches to media authentication have proven ineffective in digital forensics, particularly due to the difficulty of expressing results from black‐box algorithms. Compounding this challenge, practitioners face long examination times and an ever‐increasing volume of media evidence in modern investigations. To address these challenges, the authors developed an artificial intelligence expert system incorporating several logic‐based authentication tests related to media file structure, attribute similarity analysis, and internal file attributes. These tests were organized into distinct authentication pathways to produce concise, natural language conclusions comparable to opinions rendered by human examiners. The automated nature of an expert system may allow for demonstrably accurate results at scale without extensive human resources. This preliminary study compared the accuracy of the artificial intelligence expert system's opinions with those of trained human examiners from diverse digital forensic backgrounds. Both the expert system and the examiners were provided the same dataset, consisting of original, transmitted, edited, and synthetic videos. Both groups received file structure data, attribute similarity analyses, proprietary structural data, and metadata values. Using a 14‐question survey, responses were collected and evaluated for accuracy, and error rates were calculated. Across 20 respondents, the human mean score was 69% as compared to the expert system score of 91%. These results can be used to evaluate the appropriateness of deploying artificial intelligence expert systems for forensic examinations, alongside human interpretations.
Keywords: artificial intelligence, expert system, interpretation of results, media authentication
Highlights.
The study evaluates a novel expert system's accuracy as compared to a human examiner.
An expert system can be proven proficient in the interpretation of results compared to a human.
Human proficiency compared against AI output may prove effective in validating AI systems.
1. INTRODUCTION
With the advancements in generative artificial intelligence and the threat of synthetic media and deepfakes, the need for reliable and widespread media authentication at scale in digital forensic examinations and investigations has never been greater. The authors had previously designed a non‐content‐based approach to media authentication and provenance identification, proven effective in not only identifying synthetic media, but also proving the authenticity of media in a demonstrably reliable manner for use in legal proceedings [1, 2, 3, 4]. Using this approach, an automated software system, Magnet Verify (previously Medex), was built for skilled forensic examiners to analyze the authenticity and provenance of video, image, and audio files. In this software system, an unknown media file is parsed, and a unique signature is generated based on the makeup of its internal structures as required by specific format specifications. That signature is then compared against a robust reference library of known encoding to identify matches [5]. Machine learning classifiers also leverage that structural signature, coupled with specific metadata values, to calculate similarity and to return results about source device brand and model as well as last generation. In addition to the structural and classifier outputs, data about any hexadecimal manipulation, metadata values, and non‐decoded metadata are also returned for examiner analysis. In total, over a hundred data points about the unknown file may be returned for the human examiner to form an opinion about authenticity and/or provenance.
Upon deploying Magnet Verify to new users, it became apparent that leveraging these complex data points for exhaustive analysis of files was challenging to the user. This can be attributed to the novel approach of file structural analysis and the need to provide in‐depth training about media file encoding to examiners who had no previous experience in the concept. Even with the automation reducing hours of manual work to minutes, there was still significant human effort required to interpret results and generate an opinion about authenticity/provenance. Based on these early findings, the authors set out to develop an expert system to automate interpretation of results in a reliable, repeatable, and explainable way.
1.1. Expert system for interpretation of results
An expert system is a deterministic computer algorithm that replicates a human's output based on a set of pre‐programmed decisions. It is one of the earliest forms of artificial intelligence and has been used successfully for decades in medicine, space exploration, and automated attendant telephone systems [6]. Expert systems have been employed in various aspects of forensics, including DNA analysis, network intrusion, and digital forensics, although their prevalence has declined in recent years due to the explosion of large language models and generative AI [7, 8, 9]. Despite this decline, expert systems have been proven to provide succinct, reliable, repeatable, and explainable results [9]. For these reasons, the authors decided to develop an expert system based on their interpretation of the data points output by Magnet Verify.
The logic for the expert system is based on over 100 potential individual Boolean conditions of different outputs. A combination of Boolean conditions is then used to build specific ‘authentication pathways’ where the output of the pathway is dependent on the outcome of specific tests. That output is then summarized as a natural language result in a similar manner by which a human may report an opinion [10, 11]. Example outputs from the expert system, termed “Automated Analysis” within Magnet Verify, along with an example authentication pathway can be found in Figures 1, 2, 3, 4, 5.
FIGURE 1.
Example automated analysis output of a camera original file.
FIGURE 2.
Example automated analysis output of a transmitted file.
FIGURE 3.
Example automated analysis output of a generative AI (deepfake) file.
FIGURE 4.
Example inconclusive automated analysis output.
FIGURE 5.
Example automated analysis pathway.
After employing the expert system in the Magnet Verify software, the authors sought to understand how the accuracy and reliability of expert system results compare against human performance.
2. MATERIALS AND METHODS
An anonymous survey of 14 questions relating to interpreting the authenticity/provenance of individual video files using a non‐content‐based analysis was created. The purpose of the survey was to compare the results from a wide variety of human examiners to the questions against the output from the Automated Analysis expert system. To elicit responses to authentication outputs, the principal investigator requested 14 files to be created by a colleague who had no knowledge of the study or purpose for creating the files. The specific requested files that were generated are:
3 Camera original files.
1 AI‐generated file.
2 Edited files.
3 Social media files.
1 FFmpeg transcoded file.
1 File transmitted through a mobile web browser to replicate submission to a public evidence submission portal [4].
2 Files that would specifically not match any signatures within the Verify Reference Library.
It should be noted that all files were created anew for this study. None were taken from the existing Verify Reference Library. A pdf output from Magnet Verify with all data points available was then generated for each file. The Automated Analysis output and any Magnet Verify branding were removed from the pdf prior to distribution to study participants. An example of the pdf shared with participants is available in the supplemental information.
The 14‐question survey was then distributed to over 2000 digital and multimedia forensic practitioners via LinkedIn, targeted emails to Magnet Verify users, and various industry listservs, including the IACIS listserv, kloving listserv, and fva‐list listserv. Being that media authentication is a specialized skill within digital forensics, the survey advertisement specifically asked for participation by digital forensic practitioners even if media authentication was not their primary role. A total of 25 responses were collected, with 4 submissions completing only the demographic information section and 1 submission not providing consent to participate.
In an effort to collect data about how different factors may affect human interpretation of media authentication results, the following anonymous demographic questions were asked of participants:
- How many years of experience do you have in digital forensics?
-
◦Experience is defined as active casework or employment where your duties encompass digital forensics (computer, cell phone, multimedia, IoT). It does not include time spent as a student.
-
◦
- How many years of experience do you have specific to multimedia forensics?
-
◦Experience is defined as active casework or employment where your duties specifically encompass multimedia analysis (image, video, audio forensics). It does not include time spent as a student.
-
◦
What is your highest level of education?
- How many hours of digital forensics training have you had?
-
◦Digital forensics training is defined as classes outside of formal college/university education such as computer, cell phone, multimedia, IoT forensics.
-
◦
- How many hours of training specific to multimedia forensics have you had?
-
◦Multimedia forensics training is defined as classes outside of formal college/university education specific to image, video, and/or audio forensics.
-
◦
- How many hours of training specific to media authentication have you had?
-
◦Media authentication training is defined as classes outside of formal college/university education specific to image, video, and/or audio forensics.
-
◦
FIGURE 6.
Participant years of digital forensics experience.
FIGURE 7.
Participant years of multimedia forensics experience.
FIGURE 8.
Participant amount of digital forensics training.
FIGURE 9.
Participant amount of multimedia forensics training.
FIGURE 10.
Participant amount of media authentication training.
After completing demographic information and consenting to participation, study participants were provided with the following instruction for all 14 sample outputs:
Please use the data in the following outputs to generate an opinion about the file's authenticity/provenance. You are encouraged to express a level of support with your findings consistent with forensic science reporting such as Strong Support or Support for/against an opinion, and that opinion; or a reason why no opinion can be reached.
Individual results were then evaluated and assigned a score based on the accuracy of the result as compared to the ground‐truth data of the sample output. Scores were assigned using the following rubric below, with a total possible 70 points to be awarded. The same rubric was also used to score the expert system output for the sample files. Sample responses from human participants and expert system responses to the same question are provided in Table 1.
5 points = Analysis correct with all potential authenticity/provenance information provided.
4 points = Analysis correct, but additional insight about authenticity/provenance was possible.
2 points = Analysis reported as inconclusive where an opinion to authenticity/provenance and additional insight was possible.
0 points = Analysis of authenticity/provenance was incorrect.
TABLE 1.
Sample human and automated analysis outputs.
| Human output | Expert system output | Result |
|---|---|---|
| This file is an original video created from the front or rear camera of an iPhone 16 | Automated analysis finds strong support that the submitted file is consistent with camera original encoding on an Apple iPhone 16 device | Both outputs correct (5 points) |
| File_002.AVI is likely NOT a camera original. There is evidence that it was last encoded with FFmpeg | Automated analysis finds strong support that the submitted file is not camera original. There is support the file was last encoded by FFmpeg | Both outputs correct (5 points) |
| Strong probability of authenticity based on the metadata | Automated analysis finds strong support that the submitted file is consistent with an Apple iPhone 13 Pro Max file that has been transmitted or edited by the Mobile Browser | Human incorrect (0 points)/Expert system correct (5 points) |
| The analysis suggests that the file is a camera‐original file | Automated analysis finds support that the submitted file was originally recorded on a Samsung device and was last encoded by X | Human incorrect (0 points)/Expert system correct (5 points) |
| This is not an original file. It has been edited by multiple applications and passed through multiple device types | Automated analysis finds strong support that the submitted file is not camera original. There is support the file was last encoded by Runway | Human correct, additional insight possible (4 points)/Expert system correct (5 points) |
| The analysis suggests that the file is a non camera‐original as it contains traces of 2nd generational (or more) editing | Automated analysis finds strong support that the submitted file is not camera original. There is support the file was last encoded by Roop | Human correct, additional insight possible (4 points)/Expert system correct (5 points) |
| Not original video. Use of Core Media to change it. Audio length discrepancies | Automated analysis is inconclusive as to the authenticity and/or provenance of this file. Further manual examination is required | Human correct (5 points)/Expert system inconclusive (2 points) |
| Not original video. Use of U‐law to edit audio | Automated analysis is inconclusive as to the authenticity and/or provenance of this file. Further manual examination is required | Human correct (5 points)/Expert system inconclusive (2 points) |
3. RESULTS
3.1. Human participants
An analysis of all 280 questions asked across all participants yielded a maximum score of 66, minimum score of 22, with a mean score of 47.95 across all participants. Participants responded with the correct analysis (5 points) 102 times (36% of responses) and an incorrect analysis (0 points) 40 times (14% of responses).
Figures 11, 12, 13, 14, 15 provide the mean score by level of training, experience level, and digital forensic certification. It should be noted that 4 respondents had the Magnet Certified Authenticity Examiner (MCAE) certification – that certification was removed from overlapping respondents and listed as ‘any certification + MCAE’ to more accurately represent the data.
FIGURE 11.
Participant mean scores by amount of digital forensics training.
FIGURE 12.
Participant mean scores by amount of multimedia forensics training.
FIGURE 13.
Participant mean scores by media authentication training.
FIGURE 14.
Participant mean scores by certification.
FIGURE 15.
Participant mean scores by years of experience.
An analysis of the human participant scores found that training in media authentication had the strongest positive relationship with score, with those who received more than 40 h of training scoring 8.05 points higher than the average (Spearman's ρ = 0.47, p = 0.03). Overall digital forensics training also improved participant scores, with respondents having over 80 h of training scoring 1.16 points higher than average (Spearman's ρ = 0.43, p = 0.03). Multimedia training hours were found to have little statistical significance with scores, with participants having 160+ hours of multimedia training scoring both the maximum (66 points) and minimum scores (22 points) and one participant with no multimedia training scoring 62 points (Spearman's ρ = 0.43, p = 0.12).
Experience level was found to have no correlation to score, regardless of subdiscipline, with scores remaining statistically similar regardless of years of experience with digital forensics or multimedia forensics (Spearman's ρ = 0.01, p = 0.97).
Like media authentication training, media authentication certification had a high correlation with increased score. Magnet Certified Authentication Examiner (MCAE) [12] respondent scores were 6.25 points higher than the average and included 4 of the top 5 scores (Spearman's ρ = 0.47, p = 0.03). The Law Enforcement and Emergency Services Video Association (LEVA) Certified Forensic Video Analyst (CFVA) certification [13] was found to have the largest overall negative impact on score, with CFVA respondents scoring 15.62 points less than the average score (Spearman's ρ = −0.46, p = 0.04). Other certifications held by respondents were The International Association of Computer Investigative Specialists. Certified Forensic Computer Examiner (CFCE) [14] and the Amped Software, Amped FIVE Certified Examiner (AFCE) [15] – those certifications were found to have no substantive impact on score.
3.2. Expert system
An analysis of all 14 outputs from Magnet Verify Automated Analysis yielded a total score of 64, 91.42% of the 70 total possible points. The system responded with the correct analysis (5 points) 12 times, an inconclusive response (2 points) 2 times, and no incorrect analysis responses. This resulted in recall of 85.7% and precision of 100%, most importantly not returning an incorrect result. The expert system scored 3rd overall, with an increase of 16.05 points over the human average. In evaluating expert system score as compared to training, the expert system scored 17.2 points higher than those with 80+ hours of multimedia training, 14.45 points higher than those with 80+ hours digital forensics training, and 8 points higher than those with 16+ hours authentication training (Table 2).
TABLE 2.
Human respondent vs. expert system results.
| Human category | Human score (mean) | Δ expert system score (64 points) |
|---|---|---|
| All participants | 47.95 points | +16.05 points (+33.47%) |
| 80+ hours multimedia forensics training | 46.8 points | +17.2 points (+36.75%) |
| 80+ hours digital forensics training | 49.55 points | +14.45 points (+29.16%) |
| 16+ hours media authentication training | 56 points | +8 points (+14.29%) |
4. DISCUSSION
The small sample size of respondents in this preliminary study limits the statistical power of analysis, yet it is still of importance as the first evaluation of human performance as well as that of an expert system in interpreting complex data to provide reliable opinions about media authenticity/provenance. Analysis of human outputs found that media authentication is a unique skill that requires a specific knowledge base for proficiency, as evidenced by increased scores by participants with increased training and certification. Based on these results, it was found that specialized training on media authentication is key to building proficiency. While other digital forensic training may be helpful, it is not nearly as impactful as skill‐specific authentication training. Furthermore, skills built on multimedia training and certification that concentrate on other aspects of signal analysis like clarification, content comparison, format conversion, and metadata value analysis [16] may not effectively translate to complex authentication examinations, which could adversely impact proficiency.
Upon evaluating the human scores compared to the expert system, the expert system performed 33% better than the average of all participants and 14% better than those with substantive media authentication training (the largest indicator of increased score by training). Despite the small sample size, these results are a strong indicator that the expert system is more proficient than a human in providing a natural language “opinion” response about unknown file authenticity/provenance. These results build a path forward to confidently leverage these automated expert system outputs for providing a reliable opinion about authenticity/provenance in investigations and legal proceedings by providing all‐important context about error rates in automated systems vs. human counterparts.
While the expert system is explainable and can show the logic used to form an opinion, only the output of the system was used to evaluate and ultimately demonstrate validity in this study. This is impactful to the forensic science community as it addresses the output of an automated system in the same manner the community currently addresses human outputs in proficiency testing. This approach of comparing human vs. machine outputs can also be applied to outputs of black box artificial intelligence algorithms to evaluate validity, or lack thereof, in those outputs even when the underlying decision logic is unknown.
Based on the findings of this study, future studies expanding to other forms of artificial intelligence would be warranted to evaluate the reliability of outputs as compared to current human examiner output, which may or may not perform better than a human. This may also be expanded to court admissibility considerations based upon the evaluation of a human vs. AI system. Other avenues of research may include error analysis as to what type of questions challenge the human and AI system the most, which may help establish guidance toward minimum proficiency for human and AI outputs analyzing digital evidence.
CONFLICT OF INTEREST STATEMENT
The authors of this manuscript received research funding from Magnet Forensics, which is developing products related to the research described in this publication.
Supporting information
File S1.
Epstein B, Lyons B, Fischer D, Zlogar A. Evaluating the use of a novel expert system for interpretation of media authentication results. J Forensic Sci. 2026;71:2364–2372. 10.1111/1556-4029.70377
Presented at the 78th Annual Scientific Conference of the American Academy of Forensic Sciences, February 9–13, 2026, in New Orleans, LA.
[Correction added on 01 September 2026, after the first online publication: The legends for Figures 14 and 15 have been revised in this version.]
DATA AVAILABILITY STATEMENT
The data that support the findings of this study are available from the corresponding author upon reasonable request.
REFERENCES
- 1. Le BM, Kim J, Woo SS, Moore K, Abuadbba A, Tariq S. SoK: systematization and benchmarking of deepfake detectors in a unified framework. Proceedings of the tenth IEEE European symposium on security and privacy. Piscataway, NJ: IEEE; 2025. p. 883–902. [Google Scholar]
- 2. Grimm PW, Grossmann MR, Cormack GV. Artificial intelligence as evidence. J Technol Intellect Prop. 2021;19(1):9–104. [Google Scholar]
- 3. Epstein B. Source and generational analysis of transmitted video files to an apple iPhone [thesis]. Denver, CO: University of Colorado; 2020. [Google Scholar]
- 4. Epstein B, Lyons B. Using file structure to authenticate video files uploaded to cloud storage via iOS mobile browser. Proceedings of the 76th annual scientific conference of the American Academy of forensic sciences. Colorado Springs, CO: American Academy of Forensic Sciences; 2024. [Google Scholar]
- 5. Fischer DJ, Lyons BC, Bruehs W. Source identifying forensics system, device, and method for multimedia files. US Patent 11,809,482 2023.
- 6. Tan H. A brief history and technical review of the expert system research. IOP Conf Ser: Mater Sci Eng. 2017;242(1):012111. 10.1088/1757-899X/242/1/012111 [DOI] [Google Scholar]
- 7. Cowell RG. Finex: a probabilistic expert system for forensic identification. Forensic Sci Int. 2003;13(2–3):196–206. 10.1016/s0379-0738(03)00164-6 [DOI] [PubMed] [Google Scholar]
- 8. Kim J‐S, Kim M, Bong‐Nam N. A fuzzy expert system for network forensics. Proceedings of the international conference on computational science and its applications. Berlin, Germany: Springer; 2004. p. 175–177. [Google Scholar]
- 9. Karabiyik U. Building an intelligent assistant for digital forensics [dissertation]. Tallahassee, FL: Florida State University; 2015. [Google Scholar]
- 10. Organization of Scientific Area Committees (OSAC) for Forensic Science . OSAC 2022‐S‐0001 – Standard guide for image comparison opinions. 2026. https://www.nist.gov/system/files/documents/2023/06/13/OSAC%202022‐S‐0001%20Standard%20Guide%20for%20Image%20Comparison%20Opinions_20230604_clean%20copy.pdf
- 11. European Network of Forensic Science Institutes . ENSFI guideline for evaluative reporting in forensic science. 2026. https://enfsi.eu/wp‐content/uploads/2016/09/m1_guideline.pdf
- 12. Magnet Forensics . Magnet certified authenticity examiner. 2026. https://training.magnetforensics.com/w/courses/98‐magnet‐certified‐authenticity‐examiner
- 13. Law Enforcement and Emergency Services Video Association . Certification program. 2026. https://www.leva.org/certification/
- 14. The International Association of Computer Investigative Specialists . Certified forensic computer examiner (CFCE). 2026. https://www.iacis.com/certification/cfce/
- 15. Amped Software . Amped certification. 2026. https://ampedsoftware.com/afce‐certification
- 16. Law Enforcement and Emergency Services Video Association . Providing Opportunities for Professional Development. 2026. https://www.leva.org/training/.
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
File S1.
Data Availability Statement
The data that support the findings of this study are available from the corresponding author upon reasonable request.