PhoneAbility

Testing the Performance and "Accessibility" of Biometrics

Tony Mansfield
National Physical Laboratory


Slide 2
Outline

How do we measure accessibility of biometric solutions?

• i.e. measure the extent of known problems and determine if there are other accessibility issues.

Are existing standards sufficient?

What are the priorities for testing to improve accessibility?

Slide 3
Potential Accessibility Problems

Foreseeable

• Biometric characteristic
  • missing / damaged
  • difficult to present due to disability
• User unable to
  • access the system
  • find out how to use system
• User dislikes using the system
  • e.g removing glasses, contact lenses, burka, etc,
  • concerns about safety

Unforeseeable

• E.g.
  • biometric not sufficiently distinctive
  • problems not correlated with known disabilities
    

Slide 4
Problems of Being Inclusive Increase with Scale

E.g. ID cards
Which is the bigger problem?

• Identification against national-size database
  • Identify 1 in 60 million
• Including of all sectors of society
  • Unskilled in using technology & biometrics
  • Disabilities
  • Antagonistic
  • Must be non-discriminatory
  • Must maintain high throughput
    • throughput times dominate biometric costs

Slide 5
The Role of Evaluation

To guide and support research

• Give feedback on what works and what doesn't
• Help identify important dimensions of variability

To assess readiness for deployment

• Help in selecting systems to be deployed
• Provide data for modelling system performance

To monitor performance in the field

• Tune system performance
• Identify technical challenges as feedback to R&D

Slide 6
Aspects of Performance

Ideal biometric characteristic is:

Slide 7
Aspects of Performance Related to Accessibility

Ideal biometric characteristic is:

Slide 8
Types of Testing

Slide 9
FAR vs FRR

Slide 10
The Menagerie (Doddington)

Sheep

• Sheep are well behaved, predictable, reliably recognized, not easily confused with others. (They are the ideal users for biometric systems.)

Goats

• Goats are unpredictable, not reliably recognized (though not necessarily easily confused with others).
  • Goats can have accessibility problems, but are not necessarily disabled.

Lambs and Wolves

• Lambs and wolves are easily confused with others. (Lambs are the victims, wolves are the criminals.)

Slide 11
Performance Dependent on Many Factors

Implementation

• sensors and algorithms
• user interface

Application

• time allowed
• level of supervision
• elapsed time from enrolment to verification, …

User

• types of users
• motivation
• familiarity with the system
• training

Environment

• Indoor / outdoor
• lighting
• cleanliness, …

Performance under one set of conditions is not a good prediction of performance under different conditions.

Slide 12
Example BioP2

BIOP2

• Trial of face, fingerprint and iris recognition at Frankfurt
• Verification performance dependent on frequency of system use:
  
  

  FRR at FAR = 0.1%	Face	Finger	Iris
  Using system < 10 times	10%	7%	23%
  Using system > 120 times	2%	1%	2%

Slide 13
Standard (FCD) 19795 Biometric Performance Testing & Reporting

Empirical testing

• error rates reported are those observed
• i.e., can't claim FAR = 1 in 109 without 1 billion comparisons

Guidance to

• improve estimate of field performance
• reduce bias due to inappropriate data collection or analysis

Requires full information about test conditions to be reported

Part 1 - Principles & Framework at Final Committee Draft Expected to become Draft International Standard in July

Slide 14
Standard (FCD) 19795 Biometric Performance Testing & Reporting

Guidance applicable to testing Accessibility

Recommended

• Test users should be typical for the application, not under-representing known problems
• Reporting of end-to-end error rates, inclusive of enrolment failures, acquisition failures, and matching failures

Suggested where appropriate

• Report different error rates for different demographic groups
• Histogram showing variation of error rates among test subjects

Slide 15
Multi-modal biometrics

Using multiple biometrics may help improve accessibility

• Users are less likely to have problems with all modalities

However

• Most research on multi-modality is focussed towards improving accuracy and not accessibility
• There are few genuine (I.e. correlated) multi-modal databases, especially with data regarding accessibility problems
• Optimisation of Multi-modal biometric systems requires a calibration test of the component biometric systems.

Slide 16
Conclusions

Standard performance tests and metrics can measure accessibility of biometric systems …

• IF test subjects are representative of disabilities and other problem cases.

Very few tests to date with a focus on the disabled sector

• UKPS Pilot is a notable exception

Research to make biometrics more inclusive may be more important than to improve accuracy further on "ideal" users

• Possible role for multi-modal biometrics
• Need for data to evaluate accessibility at the research level …
• … and not just during, or after, roll-out

Slide 17
References

ISO/IEC 19795 - Biometric Performance Testing & Reporting
ISO/IEC 19792 - Framework for Security Evaluation of Biometric Systems
May be obtained through British Standards Institute

Best Practices in Testing & Reporting Biometric Device Performance
www.cesg.gov.uk/site/ast/biometrics/media/BestPractice.pdf

Contents page

Previous page

Next page