DATA SETS AND EVALUATION METRICS

For any work, it is very important to have pre-set data set(s) and an appropriate evaluation metric(s) to validate the work and the results. Further, this allows to establish fixed guidelines for comparison with other methods and to establish the standard as well as best methods.

Data sets. CASIA

Here, we briefly discuss the various data sets [108] used by the researchers in their work.

CASIA has been compiled by the Chinese Academy of Science - Institute of Automation (CASIA); it is the first freely available iris database for research purposes [8]. To date, it has four versions, CASIA-Iris VI, V2, V3, and V4, wherein each of the data sets has their own subsets [1]. The first version, i.e. CASIA-Iris VI, comprises 756 iris images (320 X 280) that were acquired from 108 subjects using a home-made iris camera. The second version comprises two equal subsets, each comprising 1,200 iris images (640 X 480) acquired through OKI IRISPASS-h device and CASIA-IrisCamV2. As compared to its predecessors, the third version introduced important noise factors and comprised nearly 22,034 images of iris of 700 subjects divided unequally amongst three sets. The Interval set has 2,639 images (320 x 280), the Lamp subset has 16,212 images (640 x 480), and the Twins subset has 3,183 images (640 x 480) collected from 100 pair of twins. The latest version, i.e. CASIA- Iris V4, which is an extended version of CASIA-Iris V3, consisting of the addition of three new subsets. The first subset CASIA-Iris-Distance comprises 2,576 images (2,352 x 1,728), while the second subset CASIA-Iris-Thousand comprises 20,000 images (640 x 480), and the last subset CASIA-Iris-Syn comprises 10,000 (640 x 480) generated images from CASIA-Iris VI. The versions were released in the order in 2002, 2004, 2010, and 2010.

12.4.1.1 UBIris v1

The UBIris data sets [75,79] were compiled by the Soft Computing and Image Analysis Group (SOCIA), University of Beira Interior, Portugal. VI comprises nearly 1,877 images from 241 subjects, whereas V2 comprises 11,102 images from 259 subjects. VI was captured by Nikon E5700 camera in two parts; in the first part, the noise elements were controlled by having the image acquire set-up in a unilluminated room, while in the second part, the images were under normal light which simulated the images captured with minimal active participation and introduced several noise factors such as contrast, focus, and reflections. V2 was acquired using a Canon EOS 5D camera in unconstrained environments, such as on the visible-wavelength

Some images and their corresponding groud-truth segmentation masks from the IITD data set [58]

FIGURE 12.5 Some images and their corresponding groud-truth segmentation masks from the IITD data set [58].

and on-the-go, which simulated more realistic noise factors as compared to VI. VI was released in 2004, and V2 was released in 2010. While VI is available in multiple resolutions such as 800 x 600 pixels and 200 x 150 pixels, V2 is available in 400 x 300.

12.4.1.2 NICE-I[1] [2]and NICE-II3

Both these data sets [76,78], part of the Noisy Iris Challenge Evaluation, have been distributed by the same group as UBIris. Both the data sets are subsets of UBIris v2. NICE-I was held in 2008, and NICE-II was held in 2010.

12.4.1.3 ND-lris-0405

This data set [16] includes more than 64,979 iris images acquired from nearly 356 subjects taken from 2004 to 2005. Its subset [73] is associated with the iris challenge evaluation, which was organised by the National Institute of Standards and Technology, USA, in 2005. It comprises 2,953 iris images of resolution 480 X 640 acquired from 132 subjects under NIR illumination using an LG EOU 2,200 acquisition system.

12.4.1.4 IITD[3]

It was compiled by the Biometrics Research Laboratory of Indian Institute of Technology Delhi, India, and contains 2,240 images of resolution 320 x 240 of nearly 224 subjects acquired using a fully digital CMOS, JPC1000, JIRIS camera. The subjects comprises students and staff at IIT-D itself having age between 14 and 55 years, out of which 48 subjects were female and 176 were male. The data set [58] was published in 2007 (Figure 12.5).

12.4.1.5 CSIP[4]

CSIP (Cross-Sensor Iris and Periocular data set) [92] was compiled by the SOCIA group and was acquired using four different mobile devices, i.e. W200 (THL), Xperia Arc S (Sony Ericsson), U8510 (Huawei), and iPhone 4 (Apple). The images were taken by choosing both the front and rear cameras with flash, which led to 10 combinations and their corresponding set-ups. Also, the lighting condition was varied between natural, artificial, and mixed. Owing to all the factors, several noises were incorporated in the data set. The data set comprises 2004 iris images of multiple resolutions from 50 subjects and was released in 2014.

12.4.1.6 MICHE-I and MICHE-II[5] [6] [7]

Both MICHE-I and MICHE-II data sets [30] have been created specifically for mobile biometric applications. Part of the Mobile Iris Challenge Evaluation is compiled by the Biometric and Image Processing Lab, University of Salerno, Italy, and it [30] comprises images captured solely from mobile phones in non-restrained environments without the use of any sophisticated equipment to model real-world image acquisition, thereby incorporating various noise factors into the data set. Images were captured using three mobile devices, namely, Samsung Galaxy S4 (2,322 x 4,128), Samsung Galaxy Tab2 (640 x 480), and iPhone5 (1,536 x 2,048), where each captured 1,297, 632. and 1,262 images, respectively. MICHE-II SPECIFICATIONS MICHE-I was released in 2015, and MICHE-II was released in 2016.

12.4.1.7 SBVPP0

It is distributed by the Faculty of Computer and Information Science, University of Ljubljana, and comprises 1,858 high-resolution (3,000 x 1,700) eye images acquired from 55 subjects. Each subject contributed 32 images, which comprises the person looking at four different gaze-directions, i.e. straight, up, left, and right. As the name suggests, corresponding to each image, there is a separate binary mask for the sclera, pupil, iris, and periocular region. It is a fairly new data set [85,86,102] released in 2018.

12.4.1.8 IRISSEG-CC"

The data set [7,41] has been compiled by the Halmstad University, wherein the ground truths for the subset of or the whole data set of three other iris data sets are generated. The first is the BioSec Multimodal Biometric Database Baseline [34], which was acquired through an LG IrisAccess EOU3000 close-up infrared iris camera for 3,200 images (640 x 480) from 200 subjects. The IRISSEG-CC comprises ground truth for 75 of them. Next is the CASIA Iris v3 Interval Database [1] which comprises 2,655 iris images (320 x 280) from 249 subjects acquired using a close-up infrared iris camera. The whole ground truth for this one was compiled into the IRISSEG-CC. The last is the MobBIO database [95], which contains 800 iris images (240 x 200) from 100 subjects through an Asus Eee Pad Transformer TE300T Tablet. There were two distinct illumination conditions while varying the orientation of the eye with considerable occlusion. For this, too, the ground truth for all the images was compiled (Table 12.4).

TABLE 12.4

Architecture of the Implemented UNet

Bock Name

Layer Name

No. of Filters

Strides

Output Shape

-

Input

-

-

256 x 256 x 1

Encoder

Convl_l

16

(1.1)

256 x 256 x 16

Convl_2

16

(1.1)

256 x 256 x 16

Pooll

16

(2,2)

128 x 128 x 16

Conv2_l

64

(1.1)

128 x 128 x 64

Conv2_2

64

(1.1)

128 x 128 x 64

Pool2

64

(2,2)

64 x 64 x 64

Conv3_l

128

(1,1)

64 x 64 x 128

Conv3_2

128

(1,1)

64 x 64 x 128

Pool3

128

(2,2)

32 x 32 x 128

Conv4_l

256

(1,1)

32 x 32 x 256

Conv4_2

256

(1,1)

32 x 32 x 256

Pool4

256

(2,2)

16 x 16x256

BottleNeck

Convl_l

512

(1,1)

16 x 16 x 512

Convl_2

512

(1,1)

16 x 16 x 512

Decoder

Upl

256

(2,2)

32 x 32 x 256

Concat 1

-

-

32x32x512

Convl_l

256

(1,1)

32 x 32 x 256

Convl_2

256

(1,1)

32 x 32 x 256

Up2

128

(2,2)

64 x 64 x 128

Concat2

-

-

64 x 64 x 256

Conv2_l

128

(1,1)

64 x 64 x 128

Conv2_2

128

(1,1)

64 x 64 x 128

Up3

64

(2,2)

128 x 128 x 64

Concat3

-

-

128 x 128 x 128

Conv3_l

64

(1.1)

128 x 128 x 64

Conv3_2

64

(1.1)

128 x 128 x 64

Up4

16

(2,2)

256 x 256 x 16

Concat4

-

-

256 x 256 x 32

Conv4_l

16

(1.1)

256 x 256 x 16

Conv4_2

16

(1.1)

256 x 256 x 16

Output

1

(1.1)

256 x 256 x 1

UNet [83]

FIGURE 12.6 UNet [83].

12.4.1.9 IRISSEG-EP

Compiled along with IRISSEG-CC [7,41], the data set [41] was compiled by the Multimedia Signal Processing and Security Lab, University of Salzburg. It comprises ground truths of other iris data sets, namely, UBIRIS v2 [79], IIT D [58], Notredame 0405 Iris Image data set [16], and CASIA Iris v4 Interval [1]. The ground truth for the Notredame data set consists of 837 images (640 x 480) whose original images were acquired using LG 2200 close-up near-infrared camera in indoor lighting but with noises such as occlusion, off-angle, and blur. The ground truth for the Casia data set consists of 2,639 images (320 X 280) whose original images were taken using CASIA close-up near-infrared camera (Figure 12.6).

Some examples of the artefacts present in the non-ideal iris images

FIGURE 12.7 Some examples of the artefacts present in the non-ideal iris images.

12.4.1.10 MMU1 and MMU2’3

Courtesy of Multimedia University, MMU1 [2] comprises about 450 images of nearly 45 subjects captured using LG IrisAccess 2200. MMU2 comprises 995 images from 100 subjects acquired using the Panasonic BM-ET100US camera. Images in the data set are of poor resolution and were taken in NIR lighting.

12.4.1.11 OpenEDS13

The data set [35] was compiled by Facebook Research comprising 356,649 eye images of resolution 400 x 640 collected from 152 subjects, wherein only 12,759 images have pixel-level annotations of the pupil, the iris, the sclera, and the background. The images were acquired under controlled illumination through a head- mounted display.

12.4.1.12 iBUG

This data set was compiled by the Intelligent Behavior Understanding Group for their work [65]. They compiled their own non-ideal iris data set through manual annotation of nearly 4,461 face images picked individually from IMDB [87], HELEN [60], UTDallas Face database [66]", 300 VW [96], CVL [72], 300 W [90]. and Columbia Gaze database [98] to finally obtain 8,882 iris images (Figure 12.7).

Here, we discussed not only the old and explored data sets but also about some new data sets that are yet to be extensively experimented upon.

Performance Metrics

In this section, we will discuss the metrics that are used for quantifying the results of various works.

12.4.2.1 Jaccard Index (Jl)

It signifies the overlap, i.e. intersection over the combined area, i.e. a union of the segmentation maps for each class. It is calculated over each class and averaged as given by the formula:

Here, N is 2, i.e. binary classes. C„ is the common pixels, i.e. all the pixels having both the ground truth and predicted label as i. Here P, and GT, are the number of pixels where predicted label is i and the other whose ground truth label is i, respectively. The final value is reported after averaging over all the images.

12.4.2.2 Mean Segmentation Error

Also termed as El, it is the overall pixel-wise classification error (PCL) calculated as the exclusive-OR (XOR) (®) between the given segmentation map (Mgt) and the predicted segmentation map (Mr). The equation is given as follows:

Thereafter, PCL is calculated for all the testing images and averaged to report the overall mean segmentation error, i.e. £,.

12.4.2.3 Nice2 Error

Nice2 error or E2 is another measure to evaluate the disparity of the two regions, i.e. non-iris and iris pixels. £), the error for ith image is computed by taking mean of the False-Positive Rate (FPR) along with False-Negative Rate (FNR), which itself is computed at pixel level. Formulas for all stated are given by

Thereafter, E is computed for every test image and averaged to report the final £2. £, and E2 are bounded between [0, 1], and as they are errors, the closer their values are to “0” the better while the opposite holds for values closer to “1”. However, the opposite is true for Intersection Over Union (IOU).

Next, we briefly state some metrics standard in the case of classification; in our case, binary classification. For that, we first define some terms:

  • • True-Positive (Tp): total foreground-pixels classified correctly as iris pixels.
  • • False-Positive (Fp): total pixels incorrectly classified as foreground-pixels.
  • • True-Negative (Tj: total background-pixels classified correctly as non-iris pixels.
  • • False-Negative (F„): total pixels incorrectly classified as background-pixels. With the knowledge of the above terminology, the following metrics are defined:
    • 1. Accuracy: Fraction of all pixels classified correctly irrespective of the class upon all the pixels in a data set.

2. Precision: Fraction of all positive-class pixels predicted correctly upon all the pixels predicted as positive.

3. Recall: It is the fraction of all the positive-class pixels classified correctly upon all the positive-class pixels.

4. F-score: It optimises both the recall and precision as it is the harmonic mean of both.

  • [1] Link-http://nicel.di.ubi.pt/
  • [2] Link-http://nice2.di.ubi.pt/
  • [3] Link-http://www4.comp.polyu.edu.hk/ csajaykr/IITD/Databasel ris.htm
  • [4] Link-http://csip.di.ubi.pt/
  • [5] Link-http://biplab.unisa.it/MICHE/MICHE-II/
  • [6] Link-http://sclera.fri.uni-lj.si/database.html
  • [7] Link-http://islab.hh.se/mediawiki/IrisS egmentationGroundtruth
 
Source
< Prev   CONTENTS   Source   Next >