Abstract
NIST, the US Government’s technology standards agency, is examining how well facial recognition systems can detect people wearing masks – after finding major flaws in the systems developed before the Covid-19 pandemic struck.
NIST examined 89 major commercial facial recognition algorithms, and found they showed error rates of between 5% and 50% in matching photos of the same person with and without a mask.
The agency tested the algorithms on a set of about 6 million photos used in previous studies under its Face Recognition Vendor Test (FRVT) programme. To perform the tests, it digitally applied mask shapes to the original photos. Because real-world masks differ, NIST also developed nine mask variants, with different shapes, nose coverage and colours.
The algorithm accuracy with masked faces declined substantially across the board. With unmasked images, the best algorithms failed to authenticate a person about 0.3% of the time. But masked images raised even these top algorithms’ failure rate to about 5%, while many otherwise competent algorithms failed between 20% to 50% of the time.
The study also explored three levels of nose coverage – low, medium and high – and found that accuracy degraded with greater nose coverage. The shape and colour of a mask also matters. Algorithm error rates were generally lower with round masks. Black masks also degraded algorithm performance more than surgical blue ones, though time and resource constraints meant the team could not test the effect of colour completely.
NIST is now examining the accuracy of algorithms that were intentionally developed with masked faces in mind. But while none of the products tested to date were designed to handle masks, NIST computer scientist Mei Ngan said it is possible to “draw a few broad conclusions from the results”.
NIST tested facial ID apps against different mask shapes, sizes and colours.
Photo: B Hayes/NIST.
She explained: “With respect to accuracy with face masks, we expect the technology to continue to improve. But the data taken so far underscores one idea common to previous FRVT tests: individual algorithms perform differently. Users should get to know the algorithm they are using thoroughly and test its performance in their own work environment.”
The report, ‘Ongoing Face Recognition Vendor Test (FRVT) Part 6A: Face recognition accuracy with face masks using pre-Covid-19 algorithms’, details each algorithm’s performance. The researchers have also posted additional information online. Their study was conducted in collaboration with the US Department of Homeland Security’s Science and Technology Directorate, Office of Biometric Identity Management, and Customs and Border Protection.