ofelectricpowersystems
LouisWehenkel,MevludinGlavic,PierreGeurts,DamienErnst
DepartmentofElectricalEngineeringandComputerScienceUniversityofLi`ege-Sart-TilmanB28-B-4000Li`ege
{L.Wehenkel,P.Geurts,dernst}@ulg.ac.be,glavic@montefiore.ulg.ac.be
Abstract-Thepaperconsidersthepossibleusesofauto-maticlearningforimprovingpowersystemperformancebysoftwaremethodologies.Automaticlearningperseisfirstre-viewedandrecentdevelopementsofthefieldarehighlighted.Thentheauthors’viewsofitsmainactualorpotentialap-plicationsrelatedtopowersystemoperationandcontrolaredescribed,andineachapplicationpresentstatusandneedsforfurtherdevelopmentsarediscussed.
Keywords-Automaticlearning,sensing,monitoring,control,electricpowersystems
1INTRODUCTION
InthefieldofpowersystemsautomaticlearningwasfirstproposedbyTomDyLiaccointhelatesixties,inthespecificcontextofon-linesecurityassessment[1].Sincethen,automaticlearninghasbeenappliedbytheacademiccommunitytomanyotherpowersystemproblems,in-cludingloadforecasting,equipmentmonitoring,expan-sionplanning,andautomaticcontrol.Whileelectricloadforecastinghasbecomeastandardapplicationofauto-maticlearning,inthefieldofdecisionmakinginoperationandcontrolofpowersystems,real-lifeapplicationshavebeenscarce,inspiteofaverysignificantandsuccessfulresearcheffortsincethemideighties.
Havingbeeninvolvedinresearchinpowersystemmonitoringandcontrolontheonehand,andautomaticlearninganddataminingontheotherhand,andmuchwiththeapplicationofthelattertotheformer,ourmainobjectiveinthispaperistoprovideourviewonpromisingapplicationsofautomaticlearninginthecontextofad-vancedsensing,monitoringandcontrolofelectricpowersystems,andtosuggestareasforfurtherdevelopment,aswellasguidelinestotakebetteradvantageoftheavailablemethodsinpratice.
Tofixideas,westartthepaperwithaquickreviewofwhatautomaticlearninganddataminingareallabout,introducingthemainlearningproblems,protocolsandter-minologyandreviewingthemainresultsofresearchinthefieldwhileprovidingsomepointerstotherelevantlitera-ture.Thereaderalreadyfamiliarwithautomaticlearning,beitatanintuitivelevel,canskipthissection.
Thebodyofthepaperiscomposedofseveralinde-pendentsectionsreviewingdifferenttypesofapplicationsthatwedeemrelevantforthefuture,althoughtheyarecurrentlyatverydifferentlevelsofmaturation.Eachoneofthesesectionshasitsowndiscussionandconclusions.Currently,thisisaworkingpaperwithoutmuchreferences
toexistingworkinthefield.Atalaterstage,weintendtocompletethesurveybyamoresystematicreviewofthelitteratureinthefield.
2AUTOMATICLEARNINGPERSE
Generallyspeaking,automaticlearningaimsatex-ploitingdatagatheredfromobservations(orsimulations)ofasystem(oranenvironment),inordertobuildmod-elsexplainingthebehaviorofthesystemand/ordecisionrulestointeractinanappropriatewaywithit.
Inwhatfollows,wefirstdescribethethreemainauto-maticlearningproblems,thenwereviewdifferentproto-cols,andweprovideashortdiscussionoftherelationofautomaticlearningtootherfields.2.1Typesofautomaticlearningproblems
Tointroducethethreemaintypesofautomaticlearn-ingproblems(supervised,reinforcement,unsupervised),wewillusetheprobabilistic/statisticalformalizationandterminology.Werefertheinterestedreadertomoregen-eraltextbooksforfurtherinformationaboutautomaticlearningtheory,itsrelationtootherdisciplines,andtheprecisedescriptionofthealgorithmsthatweonlymention[2,3,4,5,6].
2.1.1Supervisedlearningproblem
Givenasample{(xi,yi)}Ni=1ofinput-outputpairs,asupervisedlearningalgorithmaimsatautomaticallybuild-ingamodelyˆ(x)tocomputeapproximationsofoutputsasafunctionofinputs.Belongtothiscategorymethodslikedecisiontrees,neuralnetworks,linearregressionetc.
Thestandardprobabilisticformalizationofsupervisedlearningconsidersx∈Xandy∈Yastworandomvari-ablesdrawnfromsomeprobabilitydistributionPfinedoverX×Y,alossfunctiondefinedoverX,Yde-Y×Y,andahypothesisspaceH⊂YXofinput-outputfunc-tions,andmeasurestheinaccuracy(oraverageloss)ofamodelf∈Hby
L(f)=X×Y
(y,f(x))dPX,Y.Denotingby(X×Y)∗theset∞
finitesizesamples,a(deterministic)Nsupervised=1(X×Y)NofalllearningalgorithmAcanthusformallybestatedasamapping
A:(X×Y)∗→H
from(X×Y)∗intothehypothesisspaceH.Foranysam-plels∈(X×Y)∗wewillhencedenotebyA(ls)the1
modelreturnedbythealgorithmA.Assumingthatsam-pleslsN={(xi,yi)}Ni=1aredrawnaccordingtosomesamplingdistributionP(X,Y)N,thesamplingprocessandalgorithminduceaprobabilitydistributionoverthehy-pothesisspaceandhenceaprobabilitydistributionoverinaccuraciesL(A(lsN)).Letusdenoteby
NLA=L(A(lsN))dP(X,Y)N
(X,Y)N
Fromatheoreticalpointofview,reinforcementlearn-ingcanbeformalizedwithinthestochasticdynamicpro-grammingframework.Inparticular,supposingthatthe
systemobeystoatimeinvariantdynamics
xt+1=f(xt,dt,wt),
wherewtisamemorylessandtimeinvariantrandompro-cessandobtainsaboundedtimeinvariantrewardsignal
rt=r(xt,dt,wt),
overaninfinitehorizon(h→∞),onecanshowthatthetwofollowingequationsdefineanoptimaldecisionstrat-egy
Q(x,d)=E{r(x,d,w)+γmaxQ(f(x,d,w),d)},
d
theexpectedaveragelossofAforfixedsamplesizeN,by
L∗H=infL(f)
f∈H
thelowestreachableaveragelossinH,andby
L∗=infL∗H
H⊂YX
d∗(x)=argmaxQ(x,d).
d
thelowestpossibleaverageloss.
Besidesdefininggeneralconditions(onX,Y,PX,Y,P(X,Y)N,,H,Aetc.)underwhichtheaboveintroducedquantitiesindeedexist,theobjectiveofstatisticallearning
N
theoryisessentiallytostudywhetherorinwhatsenseLA
1
andL(A(lsN))convergetoL∗H[7].
Ontheotherhand,thedesignofsupervisedlearningalgorithmsessentiallyaimsatconstructingsequencesofhypothesisspacesHnandlearningalgorithmsAnwith
∗
goodconvergencepropertiesandsuchthatL∗Hn→L.Inparticular,muchoftheresearchinsupervisedlearninghasfocusedonthedesignofalgorithmswhichscalewellintermsofcomputationalrequirementswiththesamplesizeandwiththedimensionalityoftheinputandoutputspacesXandY,andwhichuse“large”hypothesisspacesabletomodelcomplexnon-linearinput-outputrelations.Fromthisresearchtwobroadclassesofalgorithmshaveemergedduringthelastfifteenyears,basedrespectivelyonkernels[8,9]andonensemblesoftrees[10,11].2.1.2ReinforcementlearningproblemGivenasampleoftrajectoriesofasystem
iiiiiiiN
{(xi0,d0,r0,x1,...xhi−1,dhi−1,rhi−1,xhi)}i=1,
reinforcementlearningaimsatderivinganapproximation
ˆ∗(x,t)maximizingsys-ofanoptimaldecisionstrategyd
temperformanceintermsofacumulatedperformancein-dexoveracertainhorizonh,definedby
R=
h−1t=0
γtrt,
whereγ∈(0,1]isadiscountfactor.Inthisframework,
xtdenotesthestateofadynamicsystemattimet,dtisthecontroldecisionappliedattimet,andrtisaninstan-taneousrewardsignal[12,13].
1Notice
Reinforcementlearningcanthusbetackledbydevelopingalgorithmstosolvetheseequations(ortheirtime-variantandfinitehorizoncounterparts)approximatelywhenthesoleinformationavailableaboutthesystemdynamicsandrewardfunctionareprovidedbyasampleofsystemtra-jectories.Thetheoreticalquestionsthathavebeenstudiedinthiscontextconcernthestatementofconditionsonthesamplingprocessandonthelearningalgorithmensuringconvergencetoanoptimalpolicyinasymptoticconditions(i.e.,whenN→∞).
Recentworkinthefieldhasallowedtotakefulladvan-tagefromstate-of-theartsupervisedlearningalgorithmsbydefiningappropriateframeworkstoplugthesealgo-rithmsinthereinforcementlearningprotocol.Inpartic-ular,modelbasedreinforcementlearningmethodsusethesampletobuildapproximationsofthesystemdynamicsandrewardfunctionanddynamicprogrammingmethodstoderivefromthemanapproximationoftheoptimalde-cisionstrategy.Ontheotherhand,theQ-learningframe-workusessupervisedlearninginordertoconstructfromthesampleanapproximationoftheQ-functionandderivefromitthedecisionpolicy.WhilethefirstgenerationofQ-learningmethodsusedparametricapproximationtech-niquestogetherwithon-linegradientdescent[14],there-centlyproposedfittedQiterationmethodallowstofullyexploitanyparametricornon-parametricbatchmodesu-pervisedlearningalgorithminthiscontext[15].
Noticethatevenwhenthesystemdynamicsandre-wardfunctionsareknown(orcanbesimulated),there-inforcementlearningframeworkmaystillbeusedasanalternativetodirectoptimization(e.g.dynamicprogram-mingormodelpredictivecontrol),byextractingdecisionpoliciesfromsamplesgeneratedautomaticallybyMonte-Carlosimulation.Inthiscontext,theadvantagesofrein-forcementlearningareitscapabilitytoexploitefficientlylargesamplesandcopewithhigh-dimensionalnon-linearandstochasticproblems.
thatwhileoriginally,statisticallearningtheorywasdevelopedinthelateseventiesandeightiesundertheclassicalassumptionofi.i.d.sam-N,morerecentworkaimsatweakeningtheassumptionstoplingaccordingtothedistributionPX,Y,i.e.undertheassumptionthatP(X,Y)N=PX,Y
caseswherethesamplesarenotindependentlydistributedanymore[3].
2
2.1.3Unsupervisedlearningproblems
Givenasampleofobservations{yi}Ni=1obtainedfromacertainsamplingdistributionPunsupervisedlearningisYoveraspaceY,theob-jectiveofessentiallytodetermineanapproximationofthesamplingdistribution.Inthemostinterestingcase,YisaproductspaceYbyndiscreteorcontinuousrandom1×···×Yvariables,nde-finedandthemainobjectiveofunsupervisedlearningistoidentifytherelationsamongtheselatter(independancerelations,colinearityrelations)aswellastheparametersoftheirdis-tributions.
Earlierworkinthisfieldconcernedclustering,princi-palcomponentanalysisandhiddenMarkovmodels.Morerecentresearchtopics,stillveryactivetoday,concernin-dependentcomponentanalysisaswellastheveryrichfieldofgraphicalprobabilisticmodels,suchasBayesianbeliefnetworks.
Independentcomponentanalysisaimsatexplainingtheobservedvariablesyiasy
linearcombinations
i=βi,jxj,wherethexjareindependentsourcevariables.
Bayesiannetworksmodelthejointdistributionoftherandomvariablesasaproductofconditionaldistributions
P(y1,...,ynn)=
P(yi|Pa(yi)),
i=1
wherePa(yi)denotesforeachvariableasubsetofso-calledparentvariables[16,17].Theparent-childrela-tionisencodedintheformofadirectedacyclicgraph,whichexplicitlyidentifiesconditionalindependencerela-tionshipsamongsubsetsofvariables.Unsupervisedlearn-ingofBayesiannetworksaimsatidentifyingfromasam-pleofobservationsthestructureoftheparent-childrela-tionshipandforeachvariabletheparametersdefiningtheconditionalprobabilitydistributionP(ymoresophisticatedversionofthisproblem,i|Pa(ycurrentlyi))[18].Asub-jectofactiveresearch,consistsofintroducingso-calledhiddenvariablesintothemodelanddefiningtheprobabil-itymodelovertheobservedvariablesasamarginalizationofthefollowingform[19]
P(y
1,...,yn)=
P(y1,...,yn,x1,...xm),x
wherethesumextendsoverallconfigurationsofthem
hiddenvariablesxi.Noticethataparticularcaseofthistypeofmodelistheso-calledhiddenMarkovmodelwherethejointdistributionP(x,y)=P(yobservedandhiddenvariablesfactorizes1,...,yasn,follows
x1,...xn)ofnP(x,y)=P(x
1)P(y1|x1)P(xi|xi−1)P(yi|xi).
i=2
Inthisparticularcase,theidentificationofthestructureofthemodelreducestothedeterminationofthenumberofstates(i.e.thenumberofpossiblevaluesofthevariablesxi)andefficientlearningalgorithmsforthishavealreadybeendevelopedseveraldecadesago[20].
2.2Reviewofdifferentlearningprotocols
Intheabovedescriptionofthedifferentautomatic
learningproblems,wehaveassumedthatthelearningal-gorithmusesawholebatchofsamplestobuilditsmodel.Inthissubsectionwereviewadaptationsofthesealgo-rithmsneededtocopewithpracticalconditionswhenitisnotpossible(ornotdesirable)toassumethatallthesam-plesareavailable(orshouldbecollected)beforehand.2.2.1Batchmodevson-linemodelearning
Inmanypracticalapplicationssamplesareprovidedonebyoneanditisusefultoconsiderso-calledon-linelearningalgorithmswhichessentiallygenerateasequenceofmodelsinthefollowingway
mi=A(mi−1,zi)
wherem0isaninitialmodel,andzistandsforinput-outputpairs(xitransitions(xi,yi)insupervisedlearning,forsystem
t,dit,rit,xi
t+1)(orlongertrajectories)inre-inforcementlearning,andforobservationvectorsyiinun-supervisedlearning.
Atypicalexampleofthissituationconcernsalearn-ingagentinteractingwithasystemandcollectingcontin-uouslyinformationaboutthesystembehaviorsubjecttothedecisionstakenbytheagent.Ideally,suchanagentshouldbeabletoadaptitsdecisionpolicyateachtimestepinconstanttime,andwithboundedmemoryrequirements,assoonasanewobservationbecomesavailable.Further-more,ifthesystemisnotstationary,theagentshouldalsobeabletoforgetobsoleteinformationcollectedinremotepastsoastoadaptitslearningonthemostrecentlyac-quiredobservations.
Typically,thecomputationalconstraintsofon-linelearningimplytheuseofsimpleparametricmodelsbythelearningagent.However,theinvestigationofappro-priatetradeoffsbetweenthesecomputationalrequirementsandtheflexibilityoftheusedhypothesisspacesdeservesfurtherresearch,soasdoestheformalizationofadaptivelearningstrategies.
2.2.2Passivevsactivelearning
Intheabovedescriptionwehavealsoassumedthatthelearningalgorithmcannotinfluencethesamplingprocessandispurelypassive.However,inmanypractical(e.g.on-line)situationsitispossibleandinterestingtoinflu-encethesamplingprocesssoastospeeduplearningandreducetimeandcostimplied.
Activelearningisaquiterichresearchfieldaimingatthedesignofalgorithmswhichareabletointeractwiththesamplingmechanisminordertoinfluencetheinformationgatheringprocessandtherebyspeeduplearning[21].Thisareaisstronglyrelatedtooptimalexperimentdesign[22]anddualcontrolmethods[23].3
2.3Discussion
Asitmaybeclearfromthepreviousoverview,auto-maticlearningtacklesessentiallyclassicalmodelingprob-lemsofstatistics.However,whileclassicalstatisticshasmuchmorefocusedontheanalyticalstudyofparameteridentification,assumingthatthefunctionalformsofdistri-butionsarealreadyknown,automaticlearninghasmuchmorefocusedonthedesignofdatadrivenalgorithms,whicharegenerallynotexploitinganystrongparametricassumptionsandhencecaninprinciplecopewithalargerclassofmodelingproblems[24].
Inautomaticlearningmanyalgorithmshavebeenorig-inallydesignedinaheuristicwayandwereinitiallystud-iedonlyempirically,byapplyingthemtosyntheticorreal-lifedatasetsandcomparingtheirresultswiththoseofothermethods.Thedevelopmentsincomputerhardware,theavailabilityoflargedatabasesandthegoodempiri-calperformancesofthesealgorithmsmadethembecomemoreandmorepopularinpractice.Duringthelasttwentyyears,statisticiansandtheoreticalcomputerscientistsbe-camemorestronglyinterestedinthisfieldandtheydrovesignificanttheoreticalresearchallowingtobetterunder-standthebehaviorofthesealgorithms,andevenimprovetheirdesignthankstothisnewinsight[3,4,7,10].
Inpractice,manydifferenttypesofmethodsexistto-daywhichareabletocopewithmillionsofsamplesand/ormillionsofdimensions.
Furtherworkisfocusingondevelopingtailoredal-gorithmswellsuitedtohandlespecificclassesofpracti-calproblems,liketime-seriesforecasting,imageandtextclassificationforinstance,wheretheinput(and/ortheout-puts)havespecificproperties[25,26].
2.4AnoteonautomaticlearningvsdataminingDataminingaimsatextractingrelevantandusefulin-formationfromlargebodiesofdata[27,28].Assuch,itisoneofthemainapplicationfieldsofallautomaticlearningalgorithms.Dataminingfocusestypicallyonapplicationswhereafieldexpertusesvariousalgorithmstogetherwithhisdomainknowledgetoextractinformationfromverylargebodiesofdata.Inadditiontotheoreticalaccuracyofautomaticlearningmethodsitisthusalsoconcernedwithinterpretability,scalabilityandvalidationofresultsthroughinteractionwiththefieldexpert.
Inthelastyears,datamininghasbeenoneofthemaindriversforresearchinautomaticlearning.
3SECURITYASSESSMENTSTUDIESTheapplicationofautomaticlearningtopowersys-temsecurityassessmentaimsatextractingdecisionrulesallowingtoidentifythemainweakpointsofthesystem,toquicklyassessitssecurity,andifnecessarytochooseappropriateactionsinordertoreducetheriskofinsecu-rity.Inthiscontext,thedatasetsaregenerallynotobtainedfromreal-lifemeasurements,rathertheyaregeneratedau-tomaticallybyMonte-Carlosimulationsusingexistingse-curityassessmenttools[29,30].
3.1Methodology
Themethodolgyconsistsessentiallyofthreesteps:1.Databasegeneration.
Thegoalofthisstepistoscreenarepresentativesetofoperatingscenariosinordertogatherdetailedin-formationaboutthecapabilityofthestudiedsystemtofacedisturbances.Eachsecurityscenarioisspecifiedbythreecomponents:anoperatingpointspecification;adisturbance(orcontingency)specification;adescrip-tionofthestaticanddynamicmodelingassumption.Foragivensecuritystudy,thedatabasegenerationcon-sistsoftwosuccessivesteps.Thefirststepaimsatspecifyingtherangeofconditionsthatwillbescreened(intheformofasetofindependentparametersandtheprobabilitydistributionsthatwillbeusedforsamplingthem)andthetypeofinformationthatwillbeextractedfromeachscenario(intheformofasetofattributesdescribingthesystempre-faultconditionsanditspost-faultdynamicbehavior).Thesecondstepconsistsofsamplingagivennumberofscenariosandcarryingoutthetime-domainsimulationsandextractingtheselectedvariablesandstoringthemintothedatabase.Thislat-terpurelyautomaticstepcantakeadvantageofagrid-computinginfrastructuretospeedupthedatabasegen-erationprocess.
Typically,theindependentparametersthatarescreenedarecomposedoftwotypesofparameters:primarypa-rametersofwhichthestudyaimsatevaluatingtheef-fectonthesecurityofthesystem(e.g.loadlevel,gener-ationdispatch,topologicalconditionsetc.);secondaryparameterswhichreflectuncertaintieswithrespecttowhichtheoutcomesofthestudyshouldberobust(e.g.externalsystemconditions,detailedloaddistribution,uncertaindynamicmodelsofloadandprotectionsys-tems).
Asconcernstherangeofattributesextractedfromeachsimulation,theydependalsoontheparticulartargetofthestudy.Forexample,inapreventivesecurityassess-mentstudy,wherethegoalistodefinesafeoperatinglimitsexpressedintermsofparametersthataremean-ingfultotheoperator,theattributeswillcoverontheonehandpre-faultvariablessuchaspowerflowsandin-jections,topologicalconditionsandvoltages,andontheotherhandasecuritymarginmeasuringhowac-ceptablethepost-faultbehavioris.Ontheotherhand,inthecontextofemergencycontrol,wherethegoalistodeterminetriggeringrulesforemergencycontrolexpressedintermsofpost-faultmeasurements,theat-tributeswillalsoprovidedetailedinformationaboutthepost-faultbehaviorofthesystem.2.Applicationofautomaticlearning.
Thequalityoftheinformationthatcanbeextractedfromadatabasestronglydependsonthenumberandrepresentativityofthescenariositcontains.Thus,thefirststepofdataanalysisconsistsinvalidatingthedatabaseinformationbyanalysingthedistributionsof
4
attributesandnumberofscenariosofdifferenttypesfoundinthedatabase.Atthisstage,thekindoftoolsthatareusefularemostlyunsupervisedmethodsandgraphicalvisualizationtoolssuchashistogramsandscatterplots.Ifthedatabaseisnotsufficientlyrepre-sentativethisanalysisshouldleadtorecommendationsallowingtomodifythedatabasespecification.Thesecondstepofanalysisconsistsofusingsupervisedlearningmethodsinordertoextractfromthedatabasedecisionrulesallowingtodeterminethesecuritylevelofthesystemasafunctionofpre-faultorpost-faultat-tributes.Tothisend,asubsampleofscenariosischosenandamongtheattributesstoredinthedatabaseasub-setisdefinedascandidateattributes(inputvariables)andanoutputvariableisselectedamongthecomputedsecurityindicators(margins,classesasappropriate).Oncethesearedefined,differentsupervisedlearningal-gorithmsmaybeappliedtothecorrespondingdataset,andtheiraccuracyisestimatedbycross-validationonanindependenttestsample.Eventually,thisanalysisallowstoidentifyamongthecandidateattributesthosethatreallyinfluencethesecuritylevelofthesystemandtobuilddecisionrulesusingthemasinputs.Itallowsalsotoassessthelearnedrulesbycomparingthemwiththoserulespreviouslyusedinoperation.
Noticethatatthisstage,theanalysisgenerallystartswithdecisionorregressiontreeinduction,sincetheselatterareabletoquicklyidentifythemostsalientandinformativeattributesamongalargenumberofcan-didateones,thusallowingonetoreducethedimen-sionalityoftheproblemandprovidemoreeasilyinter-pretableinformation.However,sincetreeinductionisoftensuboptimalfromtheaccuracypointofview,itisalsoveryusefultoapplymoresophisticatedtechniquessuchasneuralnetworks,kernelbasedmethodsoren-semblemethods,soastohaveamorepreciseideaoftheresidualerrorduetotheinfluenceofexternalsys-temconditions,detailedloaddistributionanddynamicmodelswhichcannotbetakenintoaccountinthede-cisionrulessincetheyarenotavailableinthecontextwheretherulesaregoingtobeused.
3.Validation,exploitationandmaintenanceofextractedinformation.Atthisstepthegoalistodecidewhetherthedecisionrulesextractedduringthestudyshouldindeedbeex-ploitedinoperation(orusedtochangethesettingsofemergencycontroldevices).Beyondtheeffectonsecu-rity,itisalsonecessarytoevaluatethepotentialeffectofthenewrulesintermsofinducedcosts,anditisoftenrequiredtotranslatetherulesintoadirectlyexploitableformfordecisionmaking.
Finally,maintenanceoftheextracteddecisionrulesisnecessarywhenthesystemconditionschangesignifi-cantlywithrespecttotherangeofconditionsscreenedduringthepreviousstudy.Dependingonthefocus(orbroadness)ofthestudy,maintenancemaybenecessaryatmoreorlessfrequentintervals.Noticehoweverthat
whileaninitialstudyisgenerallyrathertimeconsum-ing,themaintenanceofthedecisionrulesistypicallymuchmoreincrementalandfasttocarryout.3.2Status
ThisapproachwasfirstproposedbyTomDyLiaccointhelatesixties,inordertodevelopfastenoughon-linemethodsforpreventivedynamicsecurityassessment.Re-searchinthisfieldwascarriedoutmainlyduringtheeight-iesandearlynineties,leadingtoamatureandwiderang-ingmethodologypresentlyusedbyseveral(althoughnotmany)systemoperatorsforplanningandoperationalplan-ningstudies.
Inparticular,ajointprojectbetweenRTE(Frenchsys-temoperator)andNationalGrid(Englishsystemoperator)calledASSESS,hasledtothedevelopmentofasoftwareplatformcombiningscenariospecification,samplingandsimulation,withdataminingandreportingtoolsspecifi-callytargetingthesekindofstudies.ThistoolispresentlyusedbothforoperationplanningandsystemexpansionplanningbyseveralEuropeanTSOs.
Theabovedescribedmethodologyisasoundandhighlyscalableapproachforcarryingoutsecurityassess-mentstudiesincomplexanduncertainenvironments,suchaselectricpowersystems.3.3Furtherwork
ManyTSOshavedeveloppedinthepastMonte-Carlosimulationtoolsusedforexpansionplanningunderuncer-tainties.Whilethesetoolsaretypicallyonlyextractingsyntheticinformation(suchasestimatesofexpectationsandvariancesofcostsandreliabilityindices),theycouldbeupgradedbycombiningthemwithdataminingtoolsinordertoextractmorerefinedinformationaboutconditionsleadingtohighcostsorlowreliabliity,andtherebyhelpengineerstotakebetteradvantageoftheirsystem.
Thedescribedapproachcouldallowtoassesstheef-fectofuncertaintiesduetolimitedamountofinformationavailablefordecisionmakingonthesecurity/economytradeoff.Itthuswouldprovideasystematicmeanstoas-sessoff-linehowtodecomposesecurityassessmentandcontroloveralargeinterconnectionintowellsuitedsub-problems,andtoidentifywhichinformationtoexchangeamongthecorrespondingdecisionmakingentitiessoastoensurereliablecontrol.Moregenerally,itcouldpro-videasystematicapproachtoassesstherobustnessofthepowersystemdynamicbehaviorinunusualconditionsandhowthisrobustnessisaffectedbyvariousparametersun-dercontrolofthedesigner.
Today,themethodologyisusedmainlyinoff-lineplanningkindofstudies,typicallyseveralweeksormonthsaheadoftime.However,itcouldaswellbeusedinday-aheadstudiesorevenon-linetosupportoperatordecisionmaking.Nevertheless,theeffectiveuseofthemethodologyrequiresashiftofparadigmwithrespecttotraditionaldeterministictools,andthisneedssignificanteducationeffortsamongpowersystemengineers.
Also,insecurityassessmentstudiestheapplicationofbatch-modereinforcementlearningcouldbeofvaluein
5
ordertodesigndecisionpoliciesfromMonte-Carlosimu-lations,whenthereexistsnoalternativewaytodeterminetheoptimaldecisions.
Finally,thesystematicuseofthismethodologyispos-sibleonlyifsignificantinvestmentismadeintermsofsoftwaretoolsandcomputationalinfrastructure.4AUTOMATICCONTROLSYSTEMDESIGNSincesecurityassessmentstudiesessentiallyaimatprovidingdecisionaidstohumanoperatorsitsresultsmustbeinterpretableandcompatiblewiththeinformationavailableinacontrolcenter.Giventhelargeamountofavailableinputsandofpossibledecisions,themaingoalofsecurityassessmentstudiesistoreducecomplexitybyidentifyingasubsetofrelevantinputsanddecisions.
Ontheotherhand,thedesignofanautomaticcon-troldevicehastypicallydifferentrequirements,relatedtolimited(oftenlocal)dataacquisitionandstrongtimecon-straints.Thusinatypicalautomaticcontrolsystemde-signapplication,thenumberofavailablemeasurementsismuchsmaller,thecontrolsignalisalreadydefined,andtheproblemtobetackledisalreadywellcircumscribed.4.1Methodology
Reinforcementlearningapplicationtothedesignofthecontrolpolicyofanautomaticcontroldeviceisessen-tiallycomposedofthreesteps.
1.Formulationoftheoptimalcontrolproblem.
Theproblemformulationessentiallyaimsatdefiningapseudo-stateandrewardsignalthatcanbecomputedfromavailablemeasurements.Typically,thesemea-surementsdonotprovidedirectobservabilityofthefullsystemstate,anditisbettertouseaspseudo-stateanin-formationvectorcomputedfrompresentandpastmea-surementsandpastcontrolsignals.
Therewardsignalontheotherhandshouldreflectthecontrolobjectiveandpenalyzeundesiredsituations(e.g.violationofstabilityorsafetyconstraints).2.Off-linegatheringofdataandinitiallearning.Generally,whenanewdeviceisputinoperation,thefirststageofdesigningthecontrolpolicyshouldbebasedonsimulatedscenarios.Ifanexistingcontrollerisalreadyworkingonthesystem,andtheobjectiveistoredesignitscontrolpolicy,pastmeasurementsrelatedtothiscontrollercouldalsobeusedatthisstage.Ineithercase,batch-modereinforcementlearningcanthenbeappliedtosamplesoftrajectoriesinoff-linemode,untiltheperformanceofthecontrollerissufficientlygood.Justlikeinthesecurityassessmentstudiesthisoff-linetuningneedsagoodexperimentdesignandacarefulvalidationoftheresultingcontroller,andsystematiccomparisonsofalternativedesigns.3.On-linelearningandcontrol.
Oncethecontrolagentispluggedinthesystem,itusesitspolicyinordertocontrolthesysteminclosed-loopfashion.Ifthesystemconditionschange,thecon-trollerbecomessuboptimalandeventuallyneedstoberetuned.Thiscaneitherbedoneoff-lineoron-linede-pendingontheamountofcomputingpowerthatcanbemadeavailabletothecontrolagent.Inbothcasesthelearningagentcanexploitrealmeasurementscollectedfromthesystemmeasurementsduringthetimethecon-trolagenthasbeeninoperation.4.2Status
Uptonow,workonreinforcementlearningapplica-tiontopowersystemautomaticcontrolhasbeencarriedoutexclusivelyintheacademiccontext,basedonsimula-tionswithsmallsizedsystemsinwelldefinedconditions.Theapplicationsconsideredconcernedthedampingcon-trolbyTCSCdevicesandunder-frequencyload-sheddingagents[31].
Themainprogressinthelastyearscamefromthere-searchinreinforcementlearningitself,withthedesignofnewalgorithmsabletoextractmoreefficientlyinforma-tionfromsystemtrajectories.Inpowersystems,somere-centstudiesaimedatassessingtheadvantageofreinforce-mentlearningbasedcontrolwithrespecttomodelpredic-tivecontrolandothermoreclassicaldeterministiccontrolmethods[32,33].4.3Furtherwork
Furthersignificantamountofworkisrequiredinor-dertohighlighttheintrinsicadvantagesofreinforcementlearningmethods,whichstemfromtheircapabilitytohan-dlestochasticconditionsandtoadaptautomaticallytheircontrolpolicytochangingsystemconditions,andtocon-vincethepowersystemsengineeringcommunityoftheusefulnessofthisapproachtohelpdesigningforincreasedrobustnessandoptimalitythenumerousoldandnewauto-maticcontroldevicesthatinteractthroughthepowersys-tem.
Withinthiscontextitisimportanttonoticethefactthatthelearningcontrollerswilloperateinahighlydis-tributedmulti-agentcontext,andthatthetheoryofmulti-agentreinforcementlearningispresentlyonlystartingtobedeveloped.
5APPLICATIONTOFORECASTING5.1Methodology
Forecastingessentiallyaimsatpredictingthevalueofsomequantityatsomefutureinstant,givenpresentandpastmeasurementsofthisquantityandsomeexogenousvariableswhichmayaffectthebehavior.
Fromtheviewpointofautomaticlearning,forecastingisthusbasicallyasupervisedlearningproblem,andsu-pervisedlearningmethodsmaybeviewedasalternativesolutionstobecomparedorcombinedwithclassicaltime-seriesforecastingtechniques.5.2Status
Systemloadforecastinghasbeenoneofthemostsuc-cessfulapplicationsofsupervisedandunsupervisedlearn-
6
ingtoelectricpowersystems.Morerecentlymarketpriceforecastingandwindforecastinghavebeeninvestigatedalongthesamelines.5.3Furtherpossibilities
Withinthecontextofpowersystemmonitoringandcontrol,shorttermandlocalloadandwheatherforecast-ingtoolscouldbeveryusefulinordertoenhancedecisionmaking.
Itwouldthereforebeworthtoanalysethepotentialusefulnessofautomaticlearninginthiscontext,whereitwouldnotbepracticaltousealotofhumanexpertisetodesignaforecastingmodelforeachindividualloadorge-ographicalarea.
6EXTERNALEQUIVALENTMODELS6.1Suggestion
Dynamicsecurityassessmentaswellassystemdesignstudiescarriedoutbyasystemoperator,relyonthequalityofthemodelsthatareusedtorepresentsubsystemswhicharenotdirectlyunderhiscontrolandwhoseinternalstateisnotmonitoredbyhim,suchasdistributionsubsystemsandinterconnectedneighboringtransmissionsystems.Theoperatorsoftheexternalsystemscancollectin-formationaboutinterfacevariablesandtheycouldalsoen-richthesemeasurementsbyprovidingmeasurementscor-respondingtoarichersetofsimulatedconditions.Us-ingsuchdatasets,itwouldinprinciplebepossibletocon-struct,bysupervisedlearning,syntheticinput-outputmod-elsrelatingthedynamicsofinputsignalstothoseofout-puts.Thesamecouldbedonetoimprovemodelsusedtorepresentlargeindustrialplantsinsystemstudies.6.2Furtherwork
Toourknowledgenotmuch,ifany,workhasbeencar-riedoutinthedirectionofdesigningexternalequivalentmodelsbyautomaticlearning.Nevertheless,webelievethattheneedforincreasedqualityequivalentsisstronglyfeltandthattheiravailabilitywouldbeaninterestingalter-nativetotheuseofcentralized(andmoreandmorecom-plextooperateandmaintain)wideareadataacqusitionsystems.
Furtherresearchworkshouldallowtoassessthepos-sibilityofaccuratelyrepresentingthedynamicsofalargetransmissionsystemseenfromoutsideusingautomati-callylearnedblack-boxmodels.
7DESIGNINGSOFTSENSORS
7.1Principle
Asoftsensorisanalgorithmcomputinganestimateofsomeinternalvariableofasystemwhichcannotbedirectlymeasurednorcomputedfromavailablemeasure-mentsandmodelsbecauseoflimiteddataorcomputingressources.
Asoftsensorcanbedesignedfromdetailednumericalsimulationsofasystem,byrecordingthesimulatedinter-nalandexternalvariablesandapplyingsupervisedlearn-inginordertocomputeanapproximationofthecondi-tionalexpectationoftheinternalvariablegiventheexter-nalmeasurements.Inothercircumstances,theycanbede-signedusingrealsystemmeasurementsobtainedoff-line.
Softsensorscanbeusefulinreal-timemonitoringandcontrolapplicationswhenfullflegdedmodelbasedstateestimationisnotfeasibleeitherforcomputationalreasonsorbecausenogoodmodelsexist.7.2Exampleapplication
Theideaofsoftsensorshasbeenappliedtothede-signofarotorangleandspeedestimatorfromsynchro-nizedphasormeasurements,usingneuralnetworksinsu-pervisedlearningmode[34].
Withinthiscontext,itseemsplausiblethatonecande-signbyautomaticlearningasoft-sensorusingonlyonlo-calmeasurementsinordertopredictwhenapowerplantisintheprocessofloosingsynchronism.Suchadevicecouldthenbeusedinordertodetermineclosedlooplocalcontroldevicesabletostabilizethepowerplant.
Similarapplicationscouldbeimaginedforvoltagecollapsepredictionandcontrolaswellasfortheidenti-ficationanddampingofslowinter-areamodes.
8APPLICATIONTOMONITORING8.1Suggestion
Monitoringapplicationsaremultitudinousinpowersystemsoperationandcontrol.Intrinsically,monitoringaimsatcombininginformationfromlowlevelreal-timemeasurementsinordertocomputeahighlevelindicatorofsystemhealth,relatedtotheproximityofthecurrentstateofthesystemtostabilitylimits,tothedirectioninwhichthecurrenttrendisdrivingthesystem,orsimplytoidentifywhetherthesystemhasenteredanabnormalcondition.
Thesemonitoringproblemsmaydirectlybeformu-latedasautomaticlearningproblems,supervisedorun-supervisedones.Wethusbelievethatautomaticlearningmethodscouldbeusefulinordertosynthesizeautomat-icallysystemmonitoringalgorithmsfrommeasurementsorfromsimulations.
9CONCLUSION
Inthefirstpartofthispaperwehavereviewedstate-of-the-artautomaticlearningproblems,protocolsandalgo-rithmswiththeobjectiveofhighlightingtheirapplicationpotentialsinthecontextofadvancedsensing,monitoringandcontrolofelectricpowersystems.
Inthesecondpartofthepaperwehavetriedtoexplainhowautomaticlearningcanbeappliedtovariousbroadclassesofpracticalproblems,relatedtosecurityassess-ment,automaticcontrol,forecasting,equivalencing,softsensing,andmonitoring.
Webelievethatthepotentialofapplicationofauto-maticlearningtopowersystemsishuge,andgiventhegrowingdifficultiestomanagecomplexitywithinthiscon-
7
text,wehopethatthispapercancontributetofosterfur-therresearchandinparticularmoreseriousandwide-spreadattemptsforreal-lifeapplications.
ACKNOWLEDGMENTS
DamienErnstandPierreGeurtsacknowledgethesupportoftheBelgianFNRS(FondsNationaldelaRechercheScientifique)wheretheyarepost-doctoralresearchers.
REFERENCES
[1]T.E.DyLiacco,“Controlofpowersystemsviathemulti-levelconcept,”Ph.D.dissertation,CaseWesternReserveUniversity,SystemsResearchCenter,1968.[2]S.RusselandP.Norvig,ArtificialIntelligence:aModern
Approach.PrenticeHall,1994.[3]M.Vidyasagar,ATheoryofLearningandGeneralization:
withApplicationstoNeuralNetworksandControlSystems.Springer,1997.[4]V.Vapnik,StatisticalLearningTheory.Wiley,NewYork,
1998.[5]T.Hastie,R.Tibshirani,andJ.Friedman,TheElementsof
StatisticalLearning:DataMining,InferenceandPredic-tion.Springer,2001.[6]R.DudaandP.Hart,PatternClassification,2nded.John
Wiley&Sons,Inc.,2001.[7]T.Poggio,R.Rifkin,S.Mukherjee,andP.Niyogi,“Gen-eralconditionsforpredictivityinlearningtheory,”Nature,vol.428,pp.419–422,2004.[8]B.Scholkopf,C.Burges,andA.Smola,AdvancesinKer-nelMethods:SupportVectorLearning.MITPress,Cam-bridge,MA,1999.[9]C.CristianiniandJ.Shawe-Taylor,AnIntroductiontoSup-portVectorMachines.MITPress,Cambridge,MA,2000.[10]L.Breiman,“Randomforests,”Machinelearning,vol.45,
pp.5–32,2001.[11]P.Geurts,D.Ernst,andL.Wehenkel,“Extremelyrandom-izedtrees,”MachineLearning,pp.1–39,2006,(toap-pear).[12]D.BertsekasandJ.Tsitsiklis,Neuro-DynamicProgram-ming.Belmont,MA:AthenaScientific,1996.[13]R.SuttonandA.Barto,ReinforcementLearning.AnIntro-duction.MITPress,1998.[14]C.Watkins,“LearningfromDelayedRewards,”Ph.D.dis-sertation,CambridgeUniversity,England,1989.[15]D.Ernst,P.Geurts,andL.Wehenkel,“Tree-basedbatch
modereinforcementlearning,”JournalofMachineLearn-ingResearch,vol.6,pp.503–556,April2005.[16]J.Pearl,ProbabilisticReasoninginIntelligentSystems.
SanMateo:MorganKaufmann,1988.[17]R.Cowell,A.P.Dawid,S.L.Lauritzen,andD.J.Spiegel-halter,ProbabilisticNetworksandExpertSystems.NewYork:Springer,1999.[18]V.AuvrayandL.Wehenkel,“Ontheconstructionofthe
inclusionboundaryneighbourhoodformarkovequivalenceclassesofbayesiannetworkstructures,”inProceedingsofUncertaintyinArtificialIntelligence,2002,pp.26–35.
8
[19]V.Auvray,P.Geurts,andL.Wehenkel,“Asemi-algebraic
descriptionofdiscretenaivebayesmodelswithtwohiddenclasses,”inProceedingsofthe9thInternationalSympo-siumonArtificialIntelligenceandMathematics,jan2006,(toappear).[20]L.Rabiner,“Atutorialonhiddenmarkovmodelsandse-lectedapplicationsinspeechrecognition,”ProceedingsoftheIEEE,vol.77,no.2,pp.257–286,1989.[21]D.A.Cohn,Z.Ghahramani,andM.I.Jordan,“Active
learningwithstatisticalmodels,”JournalofArtificialIn-telligenceResearch,vol.4,pp.129–145,1996.[22]V.Fedorov,TheoryofOptimalExperiments.
Academic
Press,1972.
[23]B.Wittenmark,“Adaptivedualcontrolmethods:an
overview,”inProceedingsofthe5thIFACSymposiumonAdaptiveSystemsinControlandSignalProcessing,Bu-dapest,Hungary,1995,pp.67–72.[24]L.Breiman,“Statisticalmodeling:thetwocultures,”Sta-tisticalScience,vol.16,no.3,pp.199–231,2001.[25]R.Mar´ee,P.Geurts,J.Piater,andL.Wehenkel,“Random
subwindowsforrobustimageclassification,”inProceed-ingsoftheIEEEInternationalConferenceonComputerVisionandPatternRecognition,CVPR2005,vol.1,2005,pp.34–40.[26]P.GeurtsandL.Wehenkel,“Segmentandcombineap-proachfornon-parametrictime-seriesclassification,”inProceedingsofthe9thEuropeanConferenceonPrinci-plesandPracticeofKnowledgeDiscoveryinDatabases(PKDD),October2005.[27]C.OlaruandL.Wehenkel,“Datamining,”IEEEComputer
ApplicationsinPower,vol.12,no.3,pp.19–25,July1999.[28]G.Saporta,“Dataminingandofficialstatistics,”inPro-ceedingsoftheQuintaConferenzaNationalediStatistica,ISTAT,2000,pp.1–4.[29]L.Wehenkel,MachineLearningApproachestoPowerSys-temSecurityAssessment.UniversityofLi`ege-Coll.Fac-ult´edesSciencesappliqu´ees,1994.[30]——,AutomaticLearningTechniquesinPowerSystems.KluwerAcademic,1998.[31]D.ErnstandL.Wehenkel,“FACTSdevicescontrolledby
meansofreinforcementlearningalgorithms,”inProceed-ingsofthe14thPowerSystemsComputationConference,PSCC02,Sevilla,Spain,June2002.[32]L.Wehenkel,M.Glavic,andD.Ernst,“Newdevelopments
intheapplicationofautomaticlearningtopowersystemcontrol,”inProceedingsofthe15thPowerSystemsCom-putationConference,PSCC05,2005.[33]D.Ernst,M.Glavic,F.Capitanescu,andL.Wehenkel,
“Modelpredictivecontrolandreinforcementlearningastwocomplementaryframeworks,”inProceedingsofthe13thIFACWorkshoponControlApplicationsofOptimi-sation,2006,(toappear).[34]A.DelAngel,M.Glavic,andL.Wehenkel,“Usingarti-ficialneuralnetworkstoestimaterotoranglesandspeedsfromphasormeasurements,”inProceedingsofIntelligentSystemsApplicationstoPowerSystems,ISAP03,2003.
因篇幅问题不能全部显示，请点此查看更多更全内容