In a new round of testing of facial recognition algorithms submitted by different vendors, the top products demonstrated highly accurate matching rates in simulating boarding of airline passengers, says the government agency that conducted the evaluations.
In one-to-many testing, the seven best performing algorithms successfully identified passengers 99.5 percent of the time if the databased contained a single image of each individual and the top performing algorithm had a nearly 99.9 percent accuracy rate, says the National Institute of Standards and Technology (NIST) in Face Recognition Vendor Test (RFVT) Part 7: Identification for Paperless Travel and Immigration (NISTIR 8381).
If a database contains more than one facial image of an individual, more algorithms show high accuracy rates, says NIST. In the tests, when six prior images of a person were included in the gallery, NIST says at least 18 developers’ algorithms identified more than 99.5 percent of travelers accurately.
The testing simulated the passenger boarding experience that U.S. Customs and Border Protection uses for its biometric exit program on international flights leaving the U.S. For biometric exit, which is sponsored by participating airlines and airports, a traveler pauses briefly at the departure gate to have a photo taken of their face. The image is immediately compared against a gallery of images managed by CBP for flights and if a match is confirmed, the passenger enters the jetway to board the flight.
Biometric exit is mandatory for foreign nationals leaving the U.S., at least where the technology is deployed. U.S. citizens may opt out of using the face recognition program but most don’t. CBP uses the program to verify the departure of foreign nationals from the U.S. in accordance with their visa terms and airlines use it to identify travelers in lieu of presenting a boarding pass to permit passengers to get on the plane.
A CBP spokesperson tells HSR the agency is matching at 98 to 99 percent for biometric exit. The spokesperson also said that the agency may have multiple images of a traveler in the gallery, noting that in a flight with 300 passengers, CBP may have 1,000 images. Those images can include passport and visa photos, photos from the Global Entry trusted traveler program, or photos from prior encounters with the Department of Homeland Security.
NIST points out that congressional legislation passed in 2017 directs that 97 percent of travelers exits from the U.S. be verified. The agency says “That requirement can be met with almost all of the algorithms tested here.”
“We ran simulations to characterize a system that is doing two jobs: identifying passengers at the gate and recording their exit for immigration,” says Patrick Grother, one of the report’s authors and a face recognition expert at NIST. “We found that accuracy varies across algorithms, but that modern algorithms generally perform better. If airlines use more accurate ones, passengers can board many flights with no errors.”
A NIST evaluation of face recognition algorithms published in 2019 stirred controversy because many algorithms produced different rates of accuracy based on sec, age, and race or country of birth. However, the better performing algorithms were generally more accurate regardless of demographics and produced few errors. Much of the media reporting at the time didn’t distinguish between the algorithms.
In FRVT Part 7, the NIST found little differences in demographics when it came to gender and national origin.
“Algorithms performed with high accuracy across all these variations,” NIST says. “False negatives, though slightly more common for women, were rare in all cases.”
One thing NIST didn’t evaluate was camera quality.
NIST evaluated 29 algorithms designed for one-to-many matching that were voluntarily submitted for the testing.
In the 2019 report, algorithms submitted by NEC Corp. were the most accurate. For that series of tests, images were searched against large populations of mugshots. CBP in its Traveler Verification System for biometric exit uses the NEC-3 algorithm supplied by NEC.
NIST in its latest report says the NEC algorithms “remain in to top five on that benchmark today.” However, NIST says the older NEC algorithms were surpassed in the latest evaluation.
The agency says the NEC-3 algorithm correctly identified 98.7 percent of individuals enrolled with a single image and 99 percent enrolled with images from multiple prior encounters. It adds that the most accurate algorithm in the recent evaluation, Visionlabs-10, correctly identified 99.9 percent of those enrolled with a single image and 100 percent with multiple images, “corresponding to about a factor of 10 fewer errors than NEC-3. Note that NEC-3 is now more than two years old and we may assume NEC has since improved its capability.”
The top 10 overall algorithms in FRVT Part 7, in terms of matching against a single image in a gallery and multiple images in a gallery, were Vision Labs-10, IDEMIA-08, Vision Labs-09, Cloudwalk -HR-000, Deepglint-09, Canon-CIB-000, Xforwardai-01, Paravision-07, Trueface-000, and Neurotechnology-0.