Fusing Face-Verification Algorithms and Humans

FusingFace-VerificationAlgorithmsandHumans

AliceJ.O’Toole,HervéAbdi,FangJiang,andP.JonathonPhillips,SeniorMember,IEEE

Abstract—Ithasbeendemonstratedrecentlythatstate-of-the-artface-recognitionalgorithmscansurpasshumanaccuracyatmatchingfacesoverchangesinillumination.Therankingofalgorithmsandhumansbyaccuracy,however,doesnotprovideinformationaboutwhetheralgorithmsandhumansperformthetaskcomparablyorwhetheralgorithmsandhumanscanbefusedtoimproveperformance.Inthispaper,wefusedhumansandalgorithmsusingpartialleastsquareregression(PLSR).Inthefirstexperiment,weappliedPLSRtoface-pairsimilarityscoresgeneratedbysevenalgorithmsparticipatingintheFaceRecogni-tionGrandChallenge.ThePLSRproducedanoptimalweightingofthesimilarityscores,whichwetestedforgeneralitywithajack-knifeprocedure.Fusingthealgorithms’similarityscoresusingtheoptimalweightsproducedatwofoldreductionoferrorrateoverthemostaccuratealgorithm.Next,human-subject-generatedsimilarityscoreswereaddedtothePLSRanalysis.Fusinghumansandalgorithmsincreasedtheperformancetonear-perfectclassi-ficationaccuracy.Theseresultsarediscussedintermsofmaxi-mizingface-verificationaccuracywithhybridsystemsconsistingofmultiplealgorithmsandhumans.

IndexTerms—Faceandgesturerecognition,humaninformationprocessing,performanceevaluationofalgorithmsandsystems.

Fig.1.Samplepairoffaceimagesfroma“match”trial.Participantsre-spondedbyratingthelikelihoodthatthepictureswereofthesamepersonusingafive-pointscalerangingfrom“1)suretheyarethesameperson”to“5)suretheyarenotthesamepeople.”

I.INTRODUCTION

HEFIELDofautomaticface-recognitionalgorithmshasexpandedinthepastdecadefromconsistingofsimplealgorithmsthatoperateonhighlycontrolledimagesoffacestomoresophisticatedalgorithmsaimedatoperatinginthenaturalconditionsthatcharacterizemostsecurityapplications.Oneparticularlydifficultchallengeinadvancingalgorithmsfromcontrolledtonaturalenvironmentshasbeentheproblemofoperatingoversubstantialchangesinillumination.Thecomputationaldifficultiesposedbytheilluminationproblemhavebeenwelldocumentedintheautomaticface-recognition(cf.[1]–[3])andhuman-perceptionliterature[4]–[6].

Inmorepracticalterms,theperformanceofface-recognitionalgorithmsincontrolledanduncontrolledilluminationenviron-mentswasassessedrecentlyintheFaceRecognitionGrandChallenge(FRGC),aU.S.Government-sponsoredtestofface-recognitionalgorithmsaimedatfosteringalgorithmdevelop-ment[7],[8].TheFRGC(2004–2006)includedacademic,

ManuscriptreceivedMay25,2006.TheworkofA.O’TooleandH.AbdiwassupportedbyacontractfromtheTechnicalSupportWorkingGroup.TheworkofP.J.PhillipswassupportedinpartbytheNationalInstituteofJustice.ThispaperwasrecommendedbyGuestEditorK.Bowyer.

A.J.O’Toole,H.Abdi,andF.JiangarewiththeSchoolofBehavioralandBrainSciences(GR4.1),TheUniversityofTexasatDallas,Richard-son,TX75083-0688USA(e-mail:otoole@utdallas.edu;herve@utdallas.edu;fxj018100@utdallas.edu).

P.J.PhillipsiswiththeNationalInstituteofStandardsandTechnology,Gaithersburg,MD20899USA(e-mail:jonathon@nist.gov).

Colorversionsofoneormoreofthefiguresinthispaperareavailableonlineathttp://ieeexplore.ieee.org.

DigitalObjectIdentifier10.1109/TSMCB.2007.907034

T

industrial,andresearchlaboratorycompetitors.Competitorsparticipatedintheprogrambyvolunteeringtohavetheiralgo-rithmstestedononeormoreofsixface-matchingexperimentsvaryingindifficulty.Thesetofexperimentsincludedbothacontrolled-illuminationface-matchingexperimentandamoredifficultexperimentwherealgorithmsmatchedfaceidentityinimagestakenunderdifferentilluminationconditions.BecausetheFRGCtestedmultiplealgorithmssimultaneouslyusingastandardizedevaluationprotocolandacommonimageset,itprovidesausefultime-lockedlookattheperformanceofstate-of-the-artface-recognitionalgorithms.

ThedifficultyoftheilluminationproblemcanbeseenclearlybycomparingtheperformanceofthealgorithmsinthecontrolledanduncontrolledilluminationexperimentsoftheFRGC.Inbothcases,thetaskofthealgorithmswastodecideforeachofalargenumberoffacepairs(>128million),whethertheimageswereofthesamepersonorofdifferentpeople.Inthecontrolled-illuminationexperiment,theillumi-nationconditionswerethesameforbothimagesinthepair.Intheuncontrolled-illuminationexperiment,oneimagewastakenundercontrolled-illuminationconditions,andtheotherwastakenunderuncontrolledillumination(seeFig.1forasampleimagepair).

Twentyalgorithmscompetedinthecontrolled-illuminationexperimentandachievedanaverageverificationrateof0.91at0.001false-acceptrate.Bycontrast,intheuncontrolled-illuminationexperiment,onlysevenalgorithmsparticipated,achievinganaverageverificationrateof0.41at0.001false-acceptrate.Thedifferenceinparticipantnumbersandav-erageperformanceintheseexperimentsisevidencethattheilluminationproblemcontinuestochallengeface-recognitionalgorithms.

Aratherdifferentperspectiveontherelativelypoorper-formanceofalgorithmsintheuncontrolled-illuminationex-perimentcomesfromcomparingthealgorithmstohumans

1083-4419/$25.00©2007IEEE

1150IEEETRANSACTIONSONSYSTEMS,MAN,ANDCYBERNETICS—PARTB:CYBERNETICS,VOL.37,NO.5,OCTOBER2007

performingacomparabletask.Inarecentstudy[9],human-face-matchingperformancewascomparedtotheperformanceofthesevenalgorithmsparticipatingintheuncontrolled-illuminationmatchingexperimentoftheFRGC.Wedescribethispreviousstudyinsomedetails,here,becauseitprovidesthefusiondatausedinthispaper.A.SourceofFusionData

AlgorithmsintheFRGCuncontrolled-illuminationex-periment(experiment4inFRGCnomenclature)matchedfaceidentitiesinallpossiblepairsof16028targetim-agesand8014probeimages,withtargetimagestakenun-dercontrolled-illuminationconditionsandprobeimagestakenunderuncontrolled-illuminationconditions(seeFig.1forasamplepair).Theoutputforeachalgorithmwasamatrixofsimilarityscoresforallpossiblepairsoffaces.Foreachalgorithm,areceiveroperatingcharacteristic(ROC)curvewasgeneratedfromthesimilarityscorematrix.TheperformanceofthesevenalgorithmswascomparedusingtheseROCcurves(cf.[9]forcompleteresults).

TheprimarydifficultyincomparingtheperformanceofhumanstoalgorithmsintheFRGCistheimplausiblylargenumberofface-paircomparisonsrequiredforanexhaustivecomparison.Therefore,tocomparetheperformanceofhumanstoalgorithms,facepairsweresampledfromthematrixbyselectingasetoftheeasiestandmostdifficultpairs[9].Inthispaper,weconcentrateonthemostdifficultimagepairs.Inbothcases,however,thesamplingwasdonewiththehelpofacon-trolalgorithmbasedonaprincipalcomponentanalysis(PCA)ofthealignedandscaledfaceimages.Usingthisalgorithm,easymatchpairsweredefinedbasedonsimilarityscoresthatweresubstantiallygreaterthanthemeanforthedistributionofmatchedfacepairs,i.e.,highlysimilarimagesofthesameperson.Difficultmatchpairswerethosewithsimilarityscoressubstantiallylowerthanthematchmean,i.e.,highlydissimilarimagesofthesameperson.Easyanddifficultnonmatchpairsweredefinedinversely.

Humansubjectsmatchedtheidentityof240samplefacepairsbyratingtheircertaintythatthepairswereofthesameperson.Humanresponsesrangedonafive-pointscalefrom“certainthetwoimagesareofthesameperson”to“certainthattwoimagesarenotofsameperson.”TheratingdataallowedforthegenerationofaROCcurveforhumanperformancethatwascomparabletotheROCcurvesderivablefromtheperformanceofthealgorithms.

Thehuman–machinecomparisonwasconductedbyextract-ingthealgorithms’similarityscoresforthesamefacepairstestedinthehuman-face-matchingexperiment.ThesewereplottedonROCcurvesalongwithhumanmatch-accuracydata[9].Theresultsdemonstratedclearlythatthreealgorithms[10]–[12]surpassedhumanperformanceonthedifficultfacepairs.Ofthese,thealgorithmfromTheNewJerseyInstituteofTech-nology[10]andthealgorithmfromCarnegieMellonUniversity[11]havebeenpublished.Detailsonthethirdalgorithm,fromtheViisageCorporation,1areonlypartiallyavailable[12].

1See

Acknowledgment.

Inadditiontothefindingthatthreealgorithmswerecom-petitivewithhumansonthedifficultpairsoffaces,allbutonealgorithmsurpassedhumanperformanceontheeasyfacepairs.Combined,thesefindingssuggestthat,althoughthealgorithmperformanceontheuncontrolled-illuminationexperimentintheFRGCmaybepoorinabsoluteterms,itisnonethelesscompetitivewiththehumanperformance.Thiscomparisonisofinterestduetothefactthathumansarecurrentlyperformingthistaskinmostappliedsituations.Thispreviousstudyformsthebaseofthispaper.B.RationaleforFusion

Althoughthequantitativerankingofhumanperformancerelativetoasetofalgorithmsprovidesausefulbenchmark,thisrankingdoesnotofferanyinsightintowhetheralgorithmsrecognizefacesinwaysthataresimilartohumans.TheFRGCshowedthatalgorithmsperformedpoorlyonfacerecognitioninuncontrolled-illuminationenvironments.Ourpreviousworkshowedthesameresultforhumans.Ifalgorithmsandhumanstakediverseapproachestotheproblemoffacematching,itispossiblethatanappropriatefusionofalgorithmsandhumanscanyieldbetterperformancethanasinglealgorithmorthefusionofmultiplealgorithms.Indeed,previousworkhasshownthatfusingthemultipleface-recognitionalgorithmsimprovesperformanceoverasinglealgorithm(cf.[13]–[15]).However,nopreviousstudieshavefusedhumanandalgorithmperformance.

Inthemajorityofapplicationsforfacerecognition,ahu-manoperatorispresentandinvolvedinthedecisionprocess.Thus,itmaybeofgeneralvaluetooptimizesystemper-formancebyexplicitlyincorporatinghuman-face-recognitioncapabilitiesintothedecisionprocess.Towardthisend,wepresentamethodologyforfusingalgorithmandhumanperformance.

Inthispaper,weaskedtwoquestions.First,canperfor-mancebeimprovedbyfusingalgorithmsfromtheFRGCuncontrolled-illuminationexperiment?Second,doesfusinghu-mansandalgorithmsimproveperformanceabovethelevelachievedbythealgorithmfusion?Theavailabilityofmultiplealgorithmestimatesoffacesimilarity,inconjunctionwithanalogoushumanestimatesofsimilarity,offersthepossibilityofexploringthesequestionsinamoresystematicwaythangenerallypossible.Here,weinvestigatedthepossibilityoffusingface-similarityestimatesfromalgorithmsandhumanstoimproveface-matchingperformance.

Fusionwasperformedbypartialleastsquareregression(PLSR),astatisticaltechniquethatgeneralizesandcombinesfeaturesfromthePCAandmultipleregression[16],[17].Thetechniqueisusedtopredictasetofdependentvariablesfromasetofindependentvariables(predictors).ThechoiceofPLSis,inpart,arbitrary,becauseotherpatternclassificationorneuralnetworktechniqueswillgivecomparableresults.WeusedthePLSbecauseithastheadvantageofprovidingeasilyinterpretableweightsforindividualpredictors(seeasfollows).AlthoughthePLSislesswellknowninpattern-recognitionliterature,itiswidelyusedinchemometrics,sensoryevalua-tion,andforneuroimagingdataanalysis(cf.[16],[18],[19],

O’TOOLEetal.:FUSINGFACE-VERIFICATIONALGORITHMSANDHUMANSand[21]).WegivecompletealgorithmdetailsforthePLSRalgorithmintheAppendix.

Inthispaper,algorithmandhumanestimatesoffacesimilar-itywerethepredictors,andthematchstatusofindividualfacepairs(i.e.,samepersonordifferentpeople)wasthedependentvariable.ThePLSRgivesasetoforthogonalfactors,sometimescalledlatentvectors{t1,...,tl},fromthecovariancematrixofpredictorsanddependentvariables.Thesecanbeusedtopredictthedependentvariable(s),byappropriatelyweightingthepredictors.ThissetofweightsiscalledBplsinthePLSRliterature[16].Tofusealgorithms,theweightsprescribedinthelatentvector(s)areusedtocombinethesimilarityscoresfromeachofthesevenalgorithmstoproduceanestimateofthematchstatusforthefacepairs.Whenfusinghumansandal-gorithms,thereareeightpredictors:sevenfromthealgorithmsandonefromtheaveragedhumandata.

Thepredictivepowerofthesefactorsisgenerallyassessedwithcross-validationtechniquessuchasabootstraporjack-knifeprocedure.Allfactors,oronlyasubsetofthem,canbeusedtocomputethepredictionofthedependentvariable(s),whichareobtainedasaweightedcombinationoftheoriginalpredictorsgivenbyBpls.Thelargerthenumberoffactorskept,thebetterthepredictionofthe“learningset.”Ingeneral,however,asmallernumberoffactorsisoptimalforrobustprediction(i.e.,fortest-setpredictions).

Inthefirstexperiment,weappliedthePLStothesimilarityscoresgeneratedbysevenalgorithmsthatparticipatedintheFRGCuncontrolled-illuminationexperiment.Wetestedthegeneralityoftheoptimalweightsfoundintheanalysisforpredictingface-matchstatususingajackknifeprocedure.Inthesecondexperiment,weaddedhuman-generatedsimilarityscorestothealgorithms’scoresandmeasuredthecontributionhumanestimatesmaketothefusion.

II.PROCEDURE

A.Stimuli

FacestimuliwerechosenfromalargedatabasedevelopedfortheFRGCstudy[7],[8].Theuncontrolled-illuminationprobefaceshadaresolutionof2272×1704pixels.Thecontrolled-illuminationtargetfaceshadaresolutionof1704×2272pixels.Forthepresentanalyses,weusedthesamesetofdifficultfacepairssampledforthepreviousquantitativecomparisonbetweenhumansandalgorithms[9].Theseweresampledfromthe128448392pairsavailable,whichincluded407352(0.32%)matchpairs(i.e.,imagepairsofthesameperson)and128041040(99.68%)nonmatchpairs(i.e.,imagepairsofdifferentpeople).Toeliminatethepossibilitythathumanscouldbaseidentitycomparisonsonthesurfacefacialcharacteristicsassociatedwithraceorage,allimagesinthestudywereoffacesofCaucasianmalesandfemalesintheirtwenties.Allpairswerematchedbysex.Althoughthesede-mographicchoiceshaveconsequencesforthecomparisonofhumansversusalgorithmsinabsoluteterms,thesechoiceswerebestsuitedwiththegoalofthepreviousstudy[9].

Inthispaper,only“difficultfacepairs”wereincluded.ThesewerechosenusingacontrolalgorithmbasedonthePCAofthealignedandscaledimages.Specifically,difficultmatch

1151

facepairs(n=60)weresampledrandomlyfrommatchpairsthathadsimilarityscoreslessthantwostandarddeviationsbelowthematchmean.Difficultnonmatchfacepairs(n=60)weresampledrandomlyfromnonmatchpairsthathadsimilarityscoresgreaterthantwostandarddeviationsabovethenonmatchmean.

ThevalidityofthePCAasaprescreeningalgorithmforhumansandalgorithmswassupportedinthepreviousstudy[9].ThePCAalgorithmreliablypredicted“easy”and“difficult”setsoffacepairsforhumansinthreeexperiments[9].AllsevenalgorithmswerelikewisemoreaccurateonthePCA-screenedeasyfacepairsthanonthePCA-screeneddifficultfaces[9].ThePCA,therefore,canserveasausefulsamplingtool,eventhoughitisnotconsidered“state-of-the-art.”WedidnotusethealgorithmsavailablefromtheFRGC,whichperformmoreaccuratelythanPCA,becauseofthepotentialtobiasthesuccessofparticularalgorithmsinthealgorithm–humanevaluation[9].

B.Human-SubjectJudgmentsofFaceSimilarity

Thehuman-subjectdataforthisexperimentwerecollectedinanexperimentinwhichsubjectsviewedtheimagepairsandratedthelikelihoodthattheimageswereofthesamepersonorofdifferentpeople[9].Forcompleteness,wesketchoutthemethodsusedinthatstudy.Thereare49subjects(25malesand24females)thatviewedthe120pairsoffacesfor2seachandrespondedbyratingeachpaironthefollowingscale:1)surethatthepicturesareofthesameperson;2)thinkthatthepicturesareofthesameperson;3)donotknow;4)thinkthatthepicturesarenotofthesameperson;and5)surethatthepicturesarenotofthesameperson.Ofthe120pairs,halfwerematchpairsandhalfwerenonmatchpairs.Equalnumbersofmaleandfemalepairswereincludedinthematchandnonmatchconditions.Thesubjectswereinstructedtoexaminethefaceimagesandtodeterminewhethertheimageswereofthesamepersonorofdifferentpeople.Subjectswerenotinformedabouttheproportionofmatchversusnonmatchtrialsnorweretheygivenpracticetrials.Theimagepairswerepresentedfor2s,buttherewasnotimelimitforenteringaresponse.

Foreachpairoffaces,theaverageratingwascomputedacrossthe49subjects.ThisaverageservedasthehumansimilarityscoreforthatpairoffacesinthePLSR.C.Algorithms’JudgmentsofFaceSimilarity

Thesimilarityscoresofthe120difficultfacepairspresentedtoparticipantsinthehumanexperimentwereextractedfromeachalgorithm’s16028×8014similaritymatrix.ThesescoresservedasthealgorithmdataforthePLSR.

III.RESULTS

A.Experiment1—AlgorithmFusionbyPLSR

Thesimilarityscoresforthesevenalgorithmsforthe120difficultfacepairs(60matchand60nonmatch)werecombinedinacolumnwisematrix.Thedependentvariablewasa120-elementvectorcontainingthematchstatus(+1formatch

1152IEEETRANSACTIONSONSYSTEMS,MAN,ANDCYBERNETICS—PARTB:CYBERNETICS,VOL.37,NO.5,OCTOBER2007

TABLEI

WEIGHTSFORALGORITHMFUSIONDIFFICULTFACEPAIRS−1fornonmatch)foreachfacepair.PLSRwasappliedsimultaneouslytothecombinedsimilarityscoreandmatch-statusdatamatrices.

WevariedthenumberofPLSRfactorsretainedfromonetofiveandfoundathree-factorsolutiontobeoptimal.Retainingthreefactorsindicatesthatthefirstthreelatentvectors,whichareorderedaccordingtotheproportionofvarianceexplainedinthecovariancematrix,arecombinedlinearlytospecifytheweightsforcombiningthesimilarityscores.

Arobustperformanceestimatewasdeterminedwithajack-knifesimulation.Westartedwiththe120facepairsavailableandsystematicallydeletedeachfacepairinturn,recomputingthePLSRwiththeremaining119pairsoffaces.Wetestedthematch-statuspredictionsforthePLSRsolutionsderivedfrom119pairsoffacesonthe“left-out”facepair.Thisyielded120generalizedmatch-predictiontests.Theerrorratewereportisthefractionofleft-outfacepairsincorrectlyclassifiedaccord-ingtomatchstatus.

Errorratesforclassificationwithonethroughfivefactorswere0.067,0.075,0.059,0.067,and0.083,respectively.Theseerrorratesarealllowerthantheminimumerrorrateachievedbyanysinglealgorithmoperatingalone(cf.TableIforer-rorratesforeachindividualalgorithm).Specifically,thedataindicatethatfusion,followingtheoptimalweightingderivedwiththePLSR,cutstheerrorrateofthebestperformingalgorithm(NJIT[10]witha0.12errorrate)byafactoroftwo.

Forpurposesofinterpretation,theweightsforcombiningsimilarityscoresappearinTableI.Theseweightsareusedtocombinethesimilarityscoresfromthesevenalgorithmstoachieveamaximalseparationbetweenthematchandnonmatchface-pairdistributions.Algorithmswithweightsthathavelargeabsolutevaluesarethemostusefulinimprovingtheperfor-mancewithfusion.

Usingthisasaninterpretationguide,itisclearthatmostoftheimprovementinaccuracycomesfromcombiningjusttwoalgorithms,NJIT[10]andViisage[12],whoseweightshavethelargestabsolutevalues.Thismightbeduetothesealgorithmshavingmaximallydiversestrategiesincomputingthefacesimilarity.ThisinterpretationseemslikelygiventhattheCMUalgorithm[11]performedsomewhatbetterthanthealgorithmofViisage[12].Thus,morebenefitcanbederivedfromcombininglesserperformingalgorithmsthatoperateindifferentfashionsthanbycombininghigherperformingsimilaralgorithms.

TABLEII

WEIGHTSFORHUMAN–ALGORITHMFUSION

Canfusinghumansandalgorithmsaddtotheaccuracyofthematchestimatesandfurtherimproveclassificationoverthatobtainedwiththefusedalgorithms?Inthisexperiment,weaddedhumansimilarityestimatestothePLSRmodel.Theanalysisproceededasbeforebutwithacolumnvectorcontainingtheaveragedhumansimilaritydataappendedtothepredictormatrix.2

Again,wevaried,fromonetofive,thenumberofPLSRfactorsweretained.Inthiscase,wefoundatwo-factorsolutiontobemostrobust,usingthejackknifeproceduredescribedpreviously.TheweightsforcombininghumanandalgorithmsimilarityestimatesareshowninTableII.Performancewithonefactorthroughfivefactorsyieldedclassificationerrorratesof0.042,0.008,0.033,0.033,and0.042,respectively.

Theseresultsillustratethatitispossibletoobtainnearlyperfectclassification,whenhumansareaddedintothepredic-tormatrix.Thissuggeststhathumanstrategiesforassigningsimilaritiestofacesaddusefullytothoseemployedbythebestalgorithms.Inparticular,thisresultshowsthathumansimilarityratingsprovidespecificinformationabouttheface-paircompar-isonsthatarenotavailablefromanyofthealgorithms.

Itisworthnotingfrompreviouswork[9]thattheaccuracyofhumanswasfoundtobebelowthatofNJIT[10],CMU[11],andViisage[12]butabovetheaccuracyofalgorithmsA,B,C,andD.Inthatstudy,similarityratingsfromindividualsubjectswerecollapsedacrossthe120facepairstocreateanROCcurveforeachsubject.TheseindividualROCcurveswerethenaveragedtogiveanoverallestimateofhumanaccuracy.Here,weaveragedthesimilarityratingsfor120facepairs,collapsingacrosstheindividualsubjects.Interestingly,althoughperhapsnotsurprisingly,wefoundthatbyaveragingacrossthe49humansubjects’estimatesoffacesimilarityforeachfacepairindividually,humanerrorratewas0.12,comparabletoNJIT,whichisthebestalgorithm.Thissuggeststhatindividualsub-jects,likealgorithms,mayemploydiversestrategiesforjudgingthesimilarityofthefacepairs.Byconsequence,combiningthesimilarityestimatesofindividualsubjectsbyfusioncouldlikewisebenefitaccuracy.

IV.DISCUSSION

Fusinghumansandalgorithmsisareasonablegoalforface-recognitionresearchersandcorporationswithhopesof

2The

directionofthesimilarityscoresforthehumanswasinvertedas

comparedtothealgorithms,soforinterpretationpurposes,attentionshouldbepaidonlytotheabsolutevaluesofthePLSRweights.

O’TOOLEetal.:FUSINGFACE-VERIFICATIONALGORITHMSANDHUMANSapplyingtheirsystemstorealapplications.Knowinghowac-curatelyalgorithmsandhumansarebythemselvesisastartintryingtoestimatehowwellcombinationsofalgorithmsandhumanswillwork.However,quantitativemeasuresofaccuracyforindividualalgorithmsandhumansarenotsufficientinguid-ingthedevelopmentofhybridsystems.Thispaperillustratesthatthemostusefulfusionsofalgorithmsandhumansarelikelytocomefromcombiningface-recognitionsystems(algorithmsorhumans)withdiverseface-recognitionstrategies.

Inthispaper,wedemonstratedthatfusingalgorithmsandhumanssubstantiallyimprovedperformanceonadifficultface-matchingtask.TheuseofPLSRtofusethealgorithmsandhumansalsoyieldedapreciseindicationofhowtocombinetheindividualcomponentsofthefusionoptimally.Thisweightvectorservessimultaneouslyasarecipeforfusingsystemsandasanindicatorofthesimilarityofalgorithmandhumanstrategiesforfaceverification.

Giventhatneitheralgorithmsnorhumansperformfacerecognitionwellinuncontrolledenvironmentsandthatama-jorityofapplicationshaveahumanoperatorintheloop,area-sonablegoalofresearchersshouldbetodesignface-recognitionstrategiesthatoptimallycombinealgorithmsandhumans.Fu-sionofalgorithmsandhumanstocreategoodhybridscan,therefore,beausefulandpracticalapproachtoimprovingface-matchingperformanceinimportantapplications.

APPENDIX

InthisAppendix,wegiveabriefdescriptionofthePLSR.Amorecompletepresentationcanbefoundinpreviousworks[16],[20].MATLABprogramscanbedownloadedfromwww.utdallas.edu/~herve.ThePLSRgeneralizesandcombine,featuresfromPCAandmultipleregression.Itsgoalistooptimallypredictasetofdependentvariablesfromasetofpredictors.Specifically,PLSRsearchesforasetofcomponents(calledlatentvectors)thatperformsasimultaneousdecompo-sitionofXandYwiththeconstraintthatthesecomponentsexplainasmuchaspossibleofthecovariancebetweenXandY.ThisstepisfollowedbyaregressionstepwherethedecompositionofXisusedtopredictY.A.Notation

TheIobservationsdescribedbyK-dependentvariablesarestoredinanI×KmatrixdenotedbyY,andtheI×JmatrixofpredictorsisdenotedX.Withoutlossofgenerality,bothXandYareassumedtobecenteredandnormalized.Thecommonsetof(orthogonal)latentvectorsisstoredintheI×LmatrixT(i.e.,TTT=I).PLRSdecomposesXas

X=TP

T

wherePisaJ×LmatrixcalledtheX-loadingmatrix.ThematrixYisestimatedas

Y=TBCT

1153

whereBisadiagonalmatrixwiththe“regressionweights”asdiagonalelements,andCisthe“weightmatrix”ofthedependentvariables.

B.ComputationsofLatentVectors,Loadings,andWeightsAlatentvectorisobtainedbyfindingtwosetsofweightswandcinordertocreate(respectively)alinearcombinationofthecolumnsofXandYsuchthattheircovarianceismaximum.Specifically,thegoalistoobtainafirstpairofvectors

t=Xw

u=Yc

(1)

undertheconstraintthat

wTw=1tTt=1tTubemaximal.

(2)

Whenthefirstlatentvectorhasbeenfound,itissubtractedfrombothXandY,andtheprocedureisiterateduntilXbecomesanullmatrix(seethealgorithmsectionformore).C.Algorithm

ThedifferentcomponentsofPLSRcanbefoundbyaseriesofsingular-valuedecompositions,eachfollowedbyadeflation.Specifically,thefirstweightvectorswandcare,respectively,thefirstrightandleftsingularvectorsofthematrixXTY.Vectorstanduarethenderivedusing(1).Withthesevectors,thevalueofbiscomputedasb=tTuandthenusedtopredictYfromtasY=btcT.ThefactorloadingsforXarecomputedasp=Xt.Now,subtract(i.e.,partialout)theeffectoftfrombothXandYasfollows:X=X−tpTandY=Y−btcT.Thevectorst,u,w,c,andparethenstoredinthecorrespondingmatrices,andthescalarbisstoredasadiagonalelementofB.IfXisanullmatrix,thenthewholesetoflatentvectorshasbeenfound;otherwisetheprocedureisrepeated.

D.PredictionoftheDependentVariables

Thedependentvariablesarepredictedusingthemultivariateregressionformuladefinedas

Y=TBCT=XBPLS

(3)

with

BPLS=PT+BCT

(4)wherePT+istheMoore–PenrosepseudoinverseofPT.

ACKNOWLEDGMENT

ThisworkwasperformedfortheDepartmentofJusticeinaccordancewithSection303oftheBorderSecurityAct,codifiedas8U.S.C.1732.Specifichardwareandsoftwareproductsidentifiedinthispaperwereusedinordertoperformtheevaluationsdescribedinthispaper.InnocasedoessuchidentificationimplyrecommendationorendorsementbytheNationalInstituteofStandardsandTechnology(NIST)nordoesitimplythattheproductsandequipmentidentifiedare

1154IEEETRANSACTIONSONSYSTEMS,MAN,ANDCYBERNETICS—PARTB:CYBERNETICS,VOL.37,NO.5,OCTOBER2007

necessarilythebestavailableforthepurpose.TheprimarygoaloftheFRGCistoencourageandfacilitatethedevelopmentofface-recognitionalgorithms.Toprovidetheface-recognitionre-searchcommunitywithanunbiasedassessmentofstate-of-the-artalgorithms,researchgroupsvoluntarilysubmitsimilarityscoresfromprototypedexperimentstotheNISTforanalysis.TheresultsoftheanalysisbyNISTareanonymous,unlessotherwiseagreedtobytheparticipatingalgorithmdevelopers.Allparticipatinggroupsweregiventhechoiceofremaininganonymousorbeingidentifiedinthispaper.Performancere-sultsarefromJanuary2005forallalgorithmsexceptXieetal.,2005,whereresultsarefromAugust2005.

REFERENCES

[1]R.Gross,S.Baker,I.Matthews,andT.Kanade,“Facerecognitionacross

poseandillumination,”inHandbookofFaceRecognition,S.Z.LiandA.K.Jain,Eds.NewYork:Springer-Verlag,2005,pp.193–216.

[2]P.J.Phillips,H.Moon,P.Rizvi,andP.Rauss,“TheFERETevaluation

methodologyforface-recognitionalgorithms,”IEEETrans.PatternAnal.Mach.Intell.,vol.22,no.10,pp.1090–1104,Oct.2000.

[3]P.J.Phillips,P.Grother,R.Micheals,D.Blackburn,E.Tabassi,and

J.M.Bonein“FRVT2002EvaluationReport,”Tech.Rep.NISTIR6965,2003.[Online].Available:http://www.frvt.org

[4]W.J.Braje,“Illuminationencodinginfacerecognition:Effectofposition

shift,”J.Vis.,vol.3,no.2,pp.161–170,2003.

[5]W.J.Braje,D.Kersten,M.J.Tarr,andN.F.Troje,“Illuminationeffects

infacerecognition,”Psychobiology,vol.26,no.4,pp.371–380,1999.[6]W.J.Braje,G.E.Legge,andD.Kersten,“Invariantrecognitionofnatural

objectsinthepresenceofshadows,”Perception,vol.29,no.4,pp.383–398,2000.

[7]P.J.Phillips,P.J.Flynn,T.Scruggs,K.W.Bowyer,J.Chang,K.Hoffman,

J.Marques,J.Min,andW.Worek,“Overviewofthefacerecognitiongrandchallenge,”inProc.IEEEComput.Vis.PatternRecog.,2005,vol.1,pp.947–954.

[8]P.J.Phillips,P.J.Flynn,T.Scruggs,K.W.Bowyer,andW.Worek,

“Preliminaryfacerecognitiongrandchallengeresults,”inProc.7thInt.Conf.Autom.FaceGestureRecog.,2006,pp.15–24.

[9]A.O’Toole,P.J.Phillips,F.Jiang,J.Ayyad,N.Pénard,andH.Abdi,

FaceRecognitionAlgorithmsSurpassHumans,2005.Tech.Rep.NISTIR.[Online].Available:http://face.nist.gov

[10]C.Liu,“Capitalizeondimensionalityincreasingtechniquesfromimprov-ingfacerecognitionGrandChallengeperformance,”IEEETrans.PatternAnal.Mach.Intell.,vol.28,no.5,pp.725–737,2006.

[11]C.M.Xie,M.Savvides,andV.Kumar,“Kernelcorrelationfilterbased

redundantclass-dependencefeatureanalysis(KCFA)onFRGC2.0data,”inProc.IEEEInt.WorkshopAnal.ModelingFacesGestures,2005,pp.32–43.

[12]M.Husken,B.Brauckmann,S.Gehlen,andC.vonderMalsburg,“Strate-giesandbenefitsoffusionof2Dand3Dfacerecognition,”inProc.IEEEComput.Soc.Conf.CVPR,2005,vol.3,p.174.

[13]P.Grotherin“FaceRecognitionVendorTest2002SupplementalReport,”

Tech.Rep.NISTIR7083,2004.[Online].Available:http://www.frvt.org[14]O.Melnik,Y.Vardi,andC.-H.Zhang,“Mixedgroupranks:Preference

andconfidenceinclassifiercombination,”IEEETrans.PatternAnal.Mach.Intell.,vol.26,no.8,pp.973–981,Aug.2004.

[15]J.Czyz,J.Kittler,andL.Vanderdorpe,“Combiningfaceverification

experts,”inProc.16thInt.Conf.PatternRecog.II,2002,pp.28–31.[16]H.Abdi,“Partialleastsquaresregression(PLS-regression),”inEncyclo-pediaforResearchMethodsfortheSocialSciences,M.LewisBeck,A.Bryman,andT.Futing,Eds.ThousandOaks,CA:Sage,2003,pp.792–795.

[17]T.Naes,T.Isaksson,T.Fearn,andT.Davis,MultivariateCalibrationand

Classification.Chichester,U.K.:NIRPublications,2004.

[18]H.MartensandM.Martens,MultivariateAnalysisofQuality.London,

U.K.:Wiley,2001.

[19]A.R.McIntoshandN.Lobaugh,“Partialleastsquaresanalysisof

neuroimagingdata:Applicationsandadvances,”Neuroimage,vol.23,pp.250–263,2004.

[20]H.Abdi,“Partialleastsquaresregression,”inEncyclopediaofMeasure-mentandStatistics,N.J.Salkind,Ed.ThousandOaks,CA:Sage,2007,pp.740–744.

[21]H.Abdi,“Multivariateanalysis,”inEncyclopediaforResearchMethods

fortheSocialSciences,M.LewisBeck,A.Bryman,andT.Futing,Eds.ThousandOaks,CA:Sage,2003,pp.699–702.

AliceJ.O’ToolereceivedtheB.A.degreeinpsy-chologyfromTheCatholicUniversityofAmerica,Washington,DC,in1983andtheM.S.andPh.D.degreesinexperimentalpsychologyfromBrownUniversity,Providence,RI,in1985and1988,respectively.

ShespentthefollowingyearandahalfasaPost-doctoralFellowwiththeUniversitédeBourgogne,Dijon,France,whichwassupportedbytheFrenchEmbassytotheU.S.,andwiththeEcoleNationaleSuperieuredesTélécommunications,Paris,France.

Since1989,shehasbeenaProfessorwiththeSchoolofBehavioralandBrainSciences,TheUniversityofTexasatDallas,Richardson.In1994,shewasawardedaFellowshipfromtheAlexandervonHumboldtFoundationforasabbaticalyearattheMaxPlanckInstituteforBiologicalCybernetics,Tübingen,Germany.Herresearchinterestsincludehumanperception,memory,andcognition,withanemphasisoncomputationalmodelingofhigh-levelvision.Currentprojectsincludethestudyofhumanmemoryforfaces,thecom-parisonofhumanandalgorithmperformanceonface-recognitiontasks,andthecomputationalmodelingofdatafromfunctionalneuroimagingexperiments.

HervéAbdiwasborninBelfort,France.HereceivedtheM.S.degreeinpsychologyfromtheUniver-sityofFranche-Comté,Besancon,France,in1975,theM.S.(D.E.A.)degreeineconomicsfromtheUniversityofClermond-Ferrand,Clermond-Ferrand,France,in1976,theM.S.(D.E.A.)degreeinneurol-ogyfromtheUniversityLouisPasteur,StrasbourgCedex,France,in1977,andthePh.D.degreeinmathematicalpsychologyfromtheUniversityofAix-en-Provence,Aix-en-Provence,France,in1980.HewasanAssistantProfessorwiththeUniversity

ofFranche-Comtéin1979,anAssociateProfessorwiththeUniversityofBourgogne,Dijon,France,in1983,andaFullProfessorwiththeUniversityofBourgognein1988.HeiscurrentlyaFullProfessorwiththeSchoolofBehavioralandBrainSciences,TheUniversityofTexasatDallas,Richardson,andanAdjunctProfessorofradiologywiththeUniversityofTexasSouthwest-ernMedicalCenteratDallas.HewastwiceaFulbrightScholar.HehasalsobeenaVisitingProfessorinBrownUniversity,Providence,RI,andwiththeUniversityofDijon,Dijon,France,ChuoUniversity,Tokyo,Japan,andtheUniversityofGeneva,Geneva,Switzerland.Hisrecentworkisconcernedwithfaceandpersonperception,odorperception,andwithcomputationalmodelingoftheseprocesses.Heisalsodevelopingstatisticaltechniquesinanalyzingthestructureoflargedatasets(e.g.,inbrainimagingandsensoryevaluation)withpartialleastsquareregression,STATIS,DISTATIS,discriminantcorre-spondenceanalysis,multiple-factoranalysis,andadditivetreerepresentations.

O’TOOLEetal.:FUSINGFACE-VERIFICATIONALGORITHMSANDHUMANS1155

FangJiangwasborninChina.ShereceivedtheM.S.degreeinappliedcognitionandneurosciencefromTheUniversityofTexasatDallas,Richardson,in2004andthePh.D.degreefromtheSchoolofBehavioralandBrainSciences.

ShehasworkedonthemodelingofMRIdataandonthecomparisonbetweenhumanandmachinefacerecognition.Herrecentworkisprobingthenatureofhigh-levelfacerepresentationusingadaptation.Shehaspublishedpapersintheareasofhumanpercep-tion,computationalcomparisonsbetweenhumans

andface-recognitionalgorithms,andcognitiveneuroscience.

P.JonathonPhillips(SM’06)receivedthePh.D.de-greeinoperationsresearchfromRutgersUniversity,Piscataway,NJ.

HeisaLeadingTechnologistinthefieldsofcom-putervision,biometrics,facerecognition,andhumanidentification.From2000to2004,hewasassignedtotheDefenseAdvancedResearchProjectsAgencyasProgramManagerfortheHumanIdentificationataDistanceProgram.HewaswiththeU.S.ArmyResearchLaboratory.HeiscurrentlywiththeNa-tionalInstituteofStandardsandTechnology(NIST),

Gaithersburg,MD,whereheistheProgramManagerfortheFaceRecognitionGrandChallengeandIrisChallengeEvaluationandtheTestDirectorfortheFaceRecognitionVendorTest(FRVT)2006.HewastheTestDirectorfortheFRVT2002.Hiscurrentresearchinterestsincludecomputervision,facerecognition,biometrics,andcomputationalpsychophysics.Hisworkhasbeenreportedinprintmediaofrecord,includingTheNewYorkTimesandtheEconomist.

Dr.PhillipsisanAssociateEditorfortheIEEETRANSACTIONSONPATTERNANALYSISANDMACHINEINTELLIGENCE.HewastherecipientoftheDepartmentofCommerceGoldMedalforhisworkonFRVT2002.