寮哄寲瀛︿範锛圧einforcementLearning锛孯L锛夋槸鏈哄櫒瀛︿範鐨勫熀鏈寖寮忓拰鏂规硶璁轰箣涓€銆傛湰鏂囪瘯鍥句粙缁嶅己鍖栧涔犵殑鏍稿績鎬濇兂鍜屽師鐞嗭紝璁╄鑰呭揩閫熸帉鎻″己鍖栧涔犵殑鍩虹鐭ヨ瘑锛屼粠鑰屾洿濂藉湴寮€鍚笅涓€闃舵鐨勫涔犮€?1浠€涔堟槸寮哄寲瀛︿範锛熷己鍖栧涔狅紙RL锛夛紝涔熺О涓烘縺鍔卞涔犮€佽瘎浼板涔犳垨寮哄寲瀛︿範锛屾槸鏈哄櫒瀛︿範鐨勮寖寮忓拰鏂规硶璁轰箣涓€锛涚敤浜庢弿杩板拰瑙ｅ喅涓€涓唬鐞嗭紙agent锛変笌鐜浜や簰鐨勮繃绋嬨€傞€氳繃瀛︿範绛栫暐鏈€澶у寲濂栧姳鎴栧疄鐜扮壒瀹氱洰鏍囩殑闂銆傛満鍣ㄥ涔犵殑涓夌被锛歛銆傜洃鐫ｅ涔燘.鏃犵洃鐫ｅ涔燙.寮哄寲瀛︿範鍥?鏈哄櫒瀛︿範鐨勪笁澶х被鎴戜滑鍙互閫氳繃璁粌鐙楁潵澶ц嚧浣撲細寮哄寲瀛︿範锛氬鏋滅嫍鎵ц浜嗘垜浠殑鎸囦护锛坅ction锛夛紝濡傛灉鐙椾笉鎵ц鎴戜滑鐨勬寚浠わ紙action锛夛紝鎴戜滑灏辩粰浜堥澶寸瓑濂栧姳锛坮eward锛夛紝鎴戜滑浼氱粰浜堜竴瀹氱殑鎯╃綒锛堣礋濂栧姳锛夌粡杩囦竴瀹氭椂闂寸殑寮哄寲锛堥噸澶嶏級璁粌锛岀嫍灏变細瀛︿細瀵瑰緟鎸囦护鐨勬墽琛屽浘2鐙楃殑璁粌瀵逛簬鐙楃殑璁粌锛屽彲浠ョ湅浣滄槸鍗曟锛坰tep锛塷rsingle-episode锛坋pisode锛夊己鍖栧涔犺繃绋嬶紝鑰岀湡瀹炲満鏅殑RL搴旇鏄痬ulti-episode锛堟垨infinite锛夛級銆傛敞锛氬湪寮哄寲瀛︿範涓紝姣忎竴姝ョ殑淇℃伅鍙互鐢?state,action,reward>涓夊厓缁勮〃绀猴紝姣忎釜鍦烘櫙鎸囩殑鏄繃绋嬬粓姝㈠墠鎵€鏈夋楠ょ殑淇℃伅锛堜篃鍙建杩癸級锛屾弿杩板湪鍚庨潰鐨勭珷鑺備腑浼氳缁嗕粙缁嶃€?2MarkovNature&MarkovDecisionProcess椹皵鍙か鍐崇瓥杩囩▼锛圡arkovdecisionprocess锛孧DP锛夋槸寮哄寲瀛︿範鐨勪竴涓噸瑕佹蹇点€傝浣跨敤寮哄寲瀛︿範鏉ヨВ鍐充竴涓疄闄呴棶棰橈紝闇€瑕佸皢闂鎶借薄涓轰竴涓┈灏斿彲澶喅绛栬繃绋嬨€傞┈灏斿彲澶喅绛栬繃绋嬬鍚堥┈灏斿彲澶€ц川銆?.1椹皵鍙か鎬т粈涔堟槸椹皵鍙か鎬э紵褰撲竴涓殢鏈鸿繃绋嬶紝鏌愪竴鏃跺埢鐨勭姸鎬佸彧渚濊禆浜庡墠涓€鏃跺埢鐨勭姸鎬佹椂锛屾垜浠杩欎釜闅忔満杩囩▼鍏锋湁椹皵鍙か鐗规€э紙Markovproperty锛夛紝琛ㄧず涓烘崲鍙ヨ瘽璇达紝缁欏畾褰撳墠鐘舵€侊紝瀹冪殑鏈潵鍜岃繃鍘荤殑鐘舵€佹槸鏈夋潯浠剁嫭绔嬬殑銆傚叿鏈夐┈灏斿彲澶€ц川鐨勯殢鏈鸿繃绋嬩篃绉颁负椹皵鍙か杩囩▼銆傚浘3椹皵鍙か鎬ц川2.2椹皵鍙か鍐崇瓥杩囩▼锛圡arkovdecisionprocess锛夐┈灏斿彲澶喅绛栬繃绋嬪湪椹皵鍙か鎬ц川鐨勫熀纭€涓婂鍔犱簡鍐崇瓥鑰呮帶鍒讹紱瀹冧负鍙楀喅绛栬€呮帶鍒跺奖鍝嶇殑缁撴灉閮ㄥ垎闅忔満閮ㄥ垎妯″瀷鍐崇瓥鎻愪緵浜嗕竴涓暟瀛︽鏋躲€傛煇涓€鏃跺埢鐨勭姸鎬佸彇鍐充簬鍓嶄竴鏃跺埢鐨勭姸鎬佸拰閲囧彇鐨勫姩浣滐紝鐢ㄥ浘4涓殑鍏紡琛ㄧず銆傞┈灏斿彲澶喅绛栬繃绋嬪己鍖栧涔犻棶棰樺彲浠ユ娊璞′负椹皵鍙か鍐崇瓥杩囩▼锛屼絾姣忛噰鍙栦竴涓姩浣滐紝闄や簡浼撮殢鐨勭姸鎬佽浆绉讳箣澶栵紝浣犺繕浼氳幏寰楀鍔便€?3寮哄寲瀛︿範闂鐨勬瀯鎴愯绱犱笂鏂囨彁鍒帮紝寮哄寲瀛︿範闂鍦ㄩ┈灏斿彲澶喅绛栬繃绋嬬殑鍩虹涓婂紩鍏ヤ簡鐘舵€佽浆鎹㈢殑濂栧姳銆傚浘5寮哄寲瀛︿範鐨勫熀鏈厓绱犲熀浜庝笂鍥撅紝鎴戜滑鍙互灏哛L闂鎴栬€呰鍦烘櫙鍏冪礌鍒嗕负涓夌被锛?.鍩烘湰鍏冪礌锛屽彲浠ョ悊瑙ｄ负姣旇緝瀹炰綋鐨勫厓绱燼銆傜壒宸.鐜C.Goal锛堝疄鐜?.涓昏瑕佺礌锛屽熀浜庨┈灏斿彲澶繃绋嬬殑鎶借薄锛屾瀯閫犳眰瑙ｇ殑鏉′欢a.State锛堝寘鎷綋鍓嶇姸鎬乻鍜屼笅涓€涓椂闂存鐨勭姸鎬乻'锛塨.Actionc.Reward3.鏍稿績瑕佺礌鏄竴涓猂L闂瑙ｅ喅鏂规鐨勬牳蹇僡.Value锛堜环鍊硷級b.Policy锛堢瓥鐣ワ級閭ｄ箞瑕佺敤寮哄寲瀛︿範鏉ヨВ鍐抽棶棰橈紝闇€瑕佸policy鍜寁alue鏈夎冻澶熺殑鐞嗚В04Policy锛堢瓥鐣ワ級&浠峰€硷紙Value锛夊厛鏄庣‘鍑犱釜鍚嶈瘝锛氾慨锛堝崟姝ワ級濂栧姳濂栧姳锛岀敤r琛ㄧず銆傜敱涓€涓ぇ鍐欑殑饾惡銆傪潗哄彲鑳芥潵婧愪簬Gain鎴朑lobalReturn鐨勯瀛楁瘝锟姸鎬佸彉閲忕┖闂村拰鍔ㄤ綔鍙橀噺绌洪棿鍒嗗埆绠椾綔饾挳鍜岎潚溿€?.1绛栫暐锛圥olicy锛変粠鏈哄櫒瀛︿範鐨勮搴︽潵鐪嬶紝鎴戜滑棣栧厛闇€瑕佷负浠ｇ悊閲囩敤鐨勭瓥鐣ワ紙Policy锛夋娊璞′竴涓嚱鏁帮紝浠庤€屾墽琛宻ubs搴旀€ヨВ鍐虫柟妗堬紱Policy鍙互浣跨敤鍑芥暟锛氭潵琛ㄧずagent鍦ㄧ姸鎬乻鐨凙lpha姒傜巼涓嬮噰鍙栬鍔ㄣ€備笉鍚岀殑绛栫暐鍑芥暟浼氬奖鍝嶄唬鐞嗕汉瀵瑰姩浣滅殑閫夋嫨锛岃€屽姩浣滀細褰卞搷鐘舵€佺殑杞崲銆佽幏寰楃殑濂栧姳浠ュ強鏈€缁堢殑濂栧姳銆傜瓥鐣ュ垎涓虹‘瀹氭€х瓥鐣ュ拰闅忔満鎬х瓥鐣ャ€傜‘瀹氭€х瓥鐣ュ彧杈撳嚭0鍜?锛屼細鏈夋槑纭殑鍔ㄤ綔鎸囦护锛岃涔堟墽琛岋紝瑕佷箞涓嶆墽琛屻€傞殢鏈虹瓥鐣ヤ細杈撳嚭涓€涓鐜囧€硷紝鏄惁閲囧彇鏌愰」鍔ㄤ綔闇€瑕侀€氳繃閲囨牱寰楀埌锛屽洜姝ら殢鏈虹瓥鐣ュ叿鏈夋洿濂界殑鎺㈢储鑳藉姏銆?.2浠峰€硷紙Value锛変环鍊兼槸浠庢煇涓姸鎬乻鍒板闆嗕换鍔＄粨鏉熸墍鑳藉緱鍒扮殑鍥炴姤鐨勬暟瀛︽湡鏈涳紙鍥犱负瀹冩槸瀵规湭鏉ョ殑浼拌锛夈€備环鍊煎嚱鏁伴潪甯搁噸瑕侊紝鍥犱负鏅鸿兘浣撳湪鐘舵€乻涓殑琛ㄧ幇鍙互杈惧埌鈥滃濂解€濓紝杩欎釜鈥滃濂解€濇槸閫氳繃浠峰€煎嚱鏁版潵琛￠噺鐨勩€備负浜嗘柟渚挎眰瑙ｉ棶棰橈紝灏嗕环鍊煎嚱鏁板垎涓虹姸鎬?浠峰€煎嚱鏁板拰鐘舵€?鍔ㄤ綔-浠峰€煎嚱鏁颁袱绉嶃€?.2.1鐘舵€佸€煎嚱鏁版垜浠皢绛栫暐饾湅涓嬬姸鎬侌潙犵殑鍊煎嚱鏁拌涓簐蟺(s)锛屽嵆浠庣姸鎬侌潙犲紑濮嬶紝agent閬靛惊绛栫暐饾湅鍙互寰楀埌濂栧姳鐨勬暟瀛︽湡鏈涖€?.2.2鐘舵€?鍔ㄤ綔浠峰€煎嚱鏁板悓鏍凤紝鎴戜滑灏嗗湪绛栫暐饾湅涓嬩粠鐘舵€侌潙犻噰鍙栬鍔潙庣殑浠峰€煎嚱鏁拌涓簈蟺(s,a)锛岃繖鎰忓懗鐫€浠庣姸鎬乻寮€濮嬶紝閲囧彇琛屽姩饾憥鍙互寰楀埌鏁板涓婃湡鏈涚殑濂栧姳銆傛樉鐒讹紝鎴戜滑鍙互鍐欏嚭涓よ€呯殑鐩镐簰琛ㄧず锛氭湁浜嗕环鍊煎嚱鏁颁箣鍚庯紝鎴戜滑灏变細鍙備笌浠峰€煎嚱鏁扮殑浼樺寲锛岃繖鏃跺€欏氨闇€瑕佹帹瀵艰礉灏旀浖鏂圭▼銆?5璐濆皵鏇兼柟绋嬧€滆礉灏旀浖鏂圭▼锛圔ellmanEquation锛夆€濅篃鍙€滃姩鎬佽鍒掓柟绋嬶紙DynamicProgrammingEquation锛夆€濓紝鐢辩悊鏌ュ痉路璐濆皵鏇硷紙RichardBellman锛夊彂鐜般€傝礉灏旀浖鏂圭▼鏄姩鎬佽鍒掞紙DynamicProgramming锛夋暟瀛︿紭鍖栨柟娉曞疄鐜版渶浼樺寲鐨勫繀瑕佹潯浠躲€傝绛夊紡鏍规嵁鈥滃垵濮嬮€夋嫨鐨勬敹鐩婂拰鍒濆閫夋嫨寰楀嚭鐨勫喅绛栭棶棰樼殑浠峰€尖€濇潵琛ㄨ揪鈥滅壒瀹氭椂闂寸偣鍐崇瓥闂鐨勪环鍊尖€濄€傝繖鏍凤紝鍔ㄦ€佷紭鍖栭棶棰樺氨琚畝鍖栦负鏇寸畝鍗曠殑瀛愰棶棰橈紝杩欎簺瀛愰棶棰樻湇浠庤礉灏旀浖鎻愬嚭鐨勨€滀紭鍖栧師鍒欌€濄€傚紩鑷細缁村熀鐧剧-璐濆皵鏇兼柟绋嬨€傜畝鍗曠殑璇达紝璐濆皵鏇兼柟绋嬪皢鍑芥暟琛ㄧず涓洪€掓帹鐨勫舰寮忥紝浠庤€屽彲浠ラ€氳繃DP绛夋墜娈垫眰瑙ｃ€?.1浠峰€煎嚱鏁扮殑璐濆皵鏇兼柟绋嬫垜浠鍏堟帹瀵煎嚭鐘舵€佷环鍊煎嚱鏁皏蟺(s)鐨勮礉灏旀浖鏂圭▼褰㈠紡锛氬鏋滀粠鐘舵€侌潙犲埌鐘舵€侌潙犫€樼殑濂栧姳饾憻鏄‘瀹氭€х殑锛屽垯鍙互杩涗竴姝ョ畝鍖栥€備粠涓婇潰鐨勫叕寮忥紝鎴戜滑鍙互寰楀埌涓ゆ潯淇℃伅锛氣€蟺(s)鍙互鍦ㄤ笅涓€鏃跺埢閫掑綊鍦拌〃绀轰负v蟺(s')鈥㈠浜庢瘡涓笁鍏冪粍(饾憼,饾憥,饾憻)锛屽厛璁＄畻饾湅(饾憥鈭狆潙?饾憹(饾憼',饾憻鈭狆潙?饾憥),鐒跺悗瀵瑰€艰繘琛屽姞鏉冨緱鍒般€傚悓鐞嗭紝鎴戜滑涔熷彲浠ュ緱鍒扮姸鎬?鍔ㄤ綔-浠峰€煎嚱鏁扮殑璐濆皵鏇兼柟绋嬪舰寮忥細5.2鏈€浼樹环鍊煎嚱鏁扮殑璐濆皵鏇兼柟绋嬪己鍖栧涔犵殑浼樺寲鐩爣鏄壘鍒颁竴涓渶浼樼瓥鐣ハ€锛屼娇寰楁櫤鑳戒綋鑳藉寰楀埌鏈€澶э紙棰勬湡鏀剁泭銆傚湪鍏锋湁鏈夐檺鐘舵€佸拰鍔ㄤ綔闆嗗悎鐨凪DP涓紝鑷冲皯鏈変竴涓瓥鐣ヤ紭浜庢墍鏈夊叾浠栫瓥鐣ユ垨鑷冲皯鏈変竴涓瓥鐣ヤ笉宸簬鎵€鏈夊叾浠栫瓥鐣ワ紝姝ょ瓥鐣ヤ负鏈€浼樼瓥鐣ワ紙optimalpolicy锛夈€傛渶浼樼瓥鐣ュ彲鑳芥湁寰堝锛屾垜浠兘璁颁负蟺(s)銆傛垜浠彲浠ュ緱鍒颁袱涓渶浼樺€煎嚱鏁扮殑琛ㄨ揪寮忥細optimal-state-valuefunctionoptimal-state-action-valuefunction鎴戜滑鍐欏嚭璐濆皵鏇兼柟绋嬬殑褰㈠紡锛氭帴涓嬫潵鐨勯棶棰樻槸濡備綍寰楀埌鏈€浼樺€煎嚱鏁癘ptimalpolicy蟺*銆?6姹傝В鏈€浼樼瓥鐣ュ鏋滃凡鐭ョ姸鎬佽浆绉诲嚱鏁板拰濂栧姳鍑芥暟锛屽垯鍙互浣跨敤鍔ㄦ€佽鍒掓眰瑙ｆ渶浼樼瓥鐣ワ紱浣嗗疄闄呬笂鍦ㄥ緢澶氬満鏅笅锛屾槸鏃犳硶寰楀埌鐘舵€佽浆绉诲嚱鏁扮殑锛岄偅涔堣繖鏃跺€欎竴鑸細浣跨敤钂欑壒鍗℃礇鏂规硶锛岀敤瓒冲澶氱殑鏍锋湰鏉ラ€艰繎姒傜巼鍒嗗竷鎴栬€呮湡鏈涖€備絾鏄紝瀹屾暣鐨勮挋鐗瑰崱娲涙柟娉曢渶瑕佹湁瓒冲鐨勬牱鏈紝灏ゅ叾鏄浜庨珮绾害鐨勬暟鎹垎甯冿紝姣忎竴姝ョ殑鏁版嵁閲忔槸闈炲父鑷村懡鐨勶紝浼氬鑷存眰瑙ｆ晥鐜囦綆涓嬨€傛垜浠彲浠ヤ粠闅忔満姊害涓嬮檷鎴栧皬鎵归噺姊害涓嬮檷鐨勬€濇兂涓壘鍒颁竴浜涘惎鍙戙€傛垜浠笉闇€瑕佸畬鏁寸殑钂欑壒鍗℃礇閲囨牱銆傛垜浠彲浠ョ粨鍚堝姩鎬佽鍒掑拰钂欑壒鍗℃礇鎬濇兂鏉ラ€愭闄嶄綆姹傝В闂鐨勪笉纭畾鎬э紝杩欑鏂规硶涔熷彨鏃跺樊娉曘€備妇涓€涓畝鍗曠殑渚嬪瓙鏉ヨ鏄庢椂宸殑鎬濇兂銆傛垜浠兂棰勬祴浠庡寳浜埌涓婃捣鐨勮窛绂火潗?饾憫Beijing-Shanghai)銆傚鏋滄垜浠粠鍖椾含寮€杞﹀埌涓婃捣锛屾垜浠竴瀹氳兘寰楀埌鏇村噯纭殑璺濈銆傞偅涔堝鏋滄垜鍙紑杞﹀幓娴庡崡锛屾垜浠嶇劧鍙互鏇存柊鎴戜滑鐨勷潗?饾憫鍖椾含-涓婃捣)鏉ュ噺灏戝畠鐨勪笉纭畾鎬ю潗?饾憫鍖椾含-涓婃捣)=d鍖椾含-娴庡崡+饾惛(饾憫娴庡崡-涓婃捣)銆?.1鍔ㄦ€佽鍒掑姩鎬佽鍒掔畻娉曡捣浣滅敤鐨勫墠鎻愭槸鎴戜滑闇€瑕佺煡閬揗DP鐜鐨勫畬鏁存ā鍨嬶細蟺(a|s)锛屼篃灏辨槸褰撳墠鐨勭瓥鐣ヰ潙?饾憼',饾憻鈭狆潙?饾憥),涔熷氨鏄姸鎬佽浆绉绘ā鍨嬪拰濂栧姳鍑芥暟涓ょ鍏稿瀷鐨勫姩鎬佽鍒掔畻娉曪紝杩欎袱绉嶅姩鎬佽鍒掔畻娉曠殑鎬濇兂鍦═D涓緷鐒跺彲浠ュ鏁堛€?.1.1PolicyiterationPolicyIteration绛栫暐杩唬绠楁硶锛屾垜浠鍏堥€夋嫨涓€涓殢鏈虹殑绛栫暐饾湅锛岀劧鍚庨€氳繃绛栫暐璇勪及鍜岀瓥鐣ユ敼杩涗笉鏂凯浠ｏ紝浣跨瓥鐣ュ彲浠ユ敹鏁涘埌蟺t+1=蟺t銆傚浘7Policyiteration涓婇潰鎻愬埌浜嗕袱涓叧閿瘝锛宻trategyevaluation鍜宻trategyimprovement锛涚瓥鐣ヨ瘎浼版槸鐢ㄦ潵璇勪环涓€涓瓥鐣ョ殑鏁堟灉锛岃€岀瓥鐣ユ敼杩涙槸鍦ㄥ師鏈夌瓥鐣ョ殑鍩虹涓婂皾璇曞鎵炬洿濂界殑绛栫暐锛岄€愭鎵惧埌鏈€浼樼瓥鐣ャ€?.涓€鑸儏鍐典笅锛岀姸鎬佸€煎嚱鏁板彲浠ヤ綔涓吼潨?饾憼)鐨勭瓥鐣ヨ瘎浠峰嚱鏁帮細2.鍙互璇佹槑璐績绛栫暐饾湅'婊¤冻绛栫暐鎻愬崌瀹氱悊鐨勬潯浠讹紝鎵€浠ョ瓥鐣ヰ潨?鍙互鏄瘮鏀跨瓥饾湅濂斤紝鎴栬€呰嚦灏戝拰鏀跨瓥涓€鏍峰ソ銆傛垜浠彲浠ョ洿鎺ヨ椽濠湴閫夋嫨姣忎釜鐘舵€佷笅鍔ㄤ綔浠峰€兼渶楂樼殑鍔ㄤ綔锛屽嵆锛?.鏁翠綋杩囩▼涓?.1.2ValueIterationValueIteration浠峰€艰凯浠ｇ畻娉曟洿鐩存帴锛屾渶浼榮tate-value-function涓哄厛閫氳繃杩唬饾懀寰楀埌锛岀劧鍚庣洿鎺ラ€氳繃鏈€浼樼姸鎬佸€煎嚱鏁版仮澶嶆渶浼樼瓥鐣ハ€銆傚亣璁炬垜浠緱鍒颁簡饾懀*(s)锛岄偅涔堟垜浠湁锛氭渶浼樼瓥鐣ヰ潙?(s)鐨勮В涔熷緢绠€鍗曪細鏈€浼樼姸鎬佸€煎嚱鏁扮殑璐濆皵鏇兼柟绋嬶細b锛庡崟姝ラ€掑綊鍏紡锛歝锛庝笉鏂€掑綊涓婅堪涓ゆ锛岀洿鍒版壘鍒拌礉灏旀浖鏂圭▼鐨勪笉鍔ㄧ偣6.1.3PolicyIterationPolicyIterationPolicyIterationvsValueIterationValueIteration绠€鍗曟敹鏁涘揩锛屾敹鏁涙參銆?.2鏃堕棿宸垎娉曠殑鍔ㄦ€佽鍒掗渶瑕佺煡閬揗DP鐜鐨勫畬鏁存ā鍨嬨€傛椂搴忓樊鍒嗘硶缁撳悎浜嗚挋鐗瑰崱娲涘拰鍔ㄦ€佽鍒掔畻娉曠殑鎬濇兂锛屽彲浠ヤ粠鏍锋湰鏁版嵁涓涔犲拰浼拌绛栫暐鐨勪环鍊煎嚱鏁帮紙钂欑壒鍗℃礇锛夛紝鐒跺悗鍙互鍒╃敤DP鐨勮凯浠ｆ€濇兂姹傝В鏈€浼樹环鍊煎嚱鏁板拰绛栫暐锛堝姩鎬佽鍒掞級銆傝繖绉嶄笉闇€瑕佺煡閬撴ā鍨嬬殑寮哄寲瀛︿範鏂规硶涔熺О涓簃odel-free銆傛帴涓嬫潵鎴戜滑鐪嬩竴涓嬫椂搴忓樊鍒嗙殑浠峰€煎嚱鏁颁及璁℃柟娉曘€備及璁″嚭v蟺(s)鍚庯紝灏卞彲浠ュ绛栫暐杩涜璇勪及锛岃繘鑰屽绛栫暐杩涜鏀硅繘銆侻onteCarlo鏂规硶浼拌鍊煎嚱鏁板氨鏄敤绛栫暐饾湅鍦∕DP涓婇噰鏍峰緢澶氬簭鍒楋紝璁＄畻浠庤繖涓姸鎬佺殑return锛岀劧鍚庢眰鏈熸湜銆?.浣跨敤绛栫暐饾湅瀹屽叏閲囨牱锛堢洿鍒拌繘绋嬬粨鏉燂級锛涘洜涓烘槸涓€涓畬鏁寸殑杩囩▼閲囨牱锛屽彲浠ュ緱鍒版渶缁堢殑濂栧姳饾惡锛岄偅涔堟垜浠氨鍙互寰堝鏄撶殑寰楀埌姣忎竴涓椂闂存涔嬪悗鐨勫鍔盙t銆傚浘8杩斿洖G鐨勮挋鐗瑰崱缃椾及璁?.鏇存柊姣忎竴姝ラ噰鏍凤紙绫讳技浜庢搴︿笂鍗囷級銆侴t琛ㄧず鐘舵€侌潙犲湪鏃堕棿姝ラ暱饾憽鑾峰緱鐨勫鍔便€?.鏃跺簭宸垎绠楁硶灏嗗綋鍓嶅鍔卞姞涓婁笅涓€鐘舵€佺殑浠峰€间及璁′綔涓哄綋鍓嶇姸鎬佸皢鑾峰緱鐨勫鍔憋紙鍖椾含鍒颁笂娴风殑璺濈浼拌绀轰緥锛夛紝鍗筹細rt+纬v(st+1)鈭抳(st)绉颁负鏃堕棿宸宸€傚彟澶栵紝钂欑壒鍗℃礇鏈韩鏄棤鍋忎及璁★紝浣嗘槸鍦ㄦ洿鏂拌繃绋嬩腑澶氭浼拌鏃堕棿搴忓垪鐨勫樊寮傦紝鏄湁鍋忎及璁★紝鎵€浠oubleQ-learning杩欑閫氳繃鏍锋湰鍒掑垎鏉ヤ慨姝ｅ亸鍊氱殑瀛︿範鏂规硶灏卞嚭鐜颁簡.6.3绛栫暐姊害娉曞湪鍓嶉潰浠嬬粛鐨勭瓥鐣ヨ凯浠ｅ拰浠峰€艰凯浠ｄ腑锛屼环鍊肩殑寤烘ā鏄繀涓嶅彲灏戠殑銆傜浉姣斾箣涓嬶紝鏈妭浠嬬粛鐨勭瓥鐣ユ搴︽硶鍒欏彟杈熻箠寰勶紝鐩存帴璁＄畻绛栫暐鐨勬搴︽潵浼樺寲绛栫暐銆備粠缁熻瀛︾殑瑙掑害鏉ョ湅锛岀瓥鐣ユ搴︽硶鐨勪紭鍖栫洰鏍囨槸浠峰€煎嚱鏁板叧浜庣姸鎬佺殑鏈熸湜锛歊L璁粌鐨勬湰璐ㄦ槸瑙ｅ喅濡備笅浼樺寲闂锛堝叾涓镐负绛栫暐缃戠粶鐨勫弬鏁帮級锛氬湪姣忔璁粌杩唬涓紝濡傛灉鎴戜滑鐭ラ亾浼樺寲鐩爣鐩稿浜庣綉缁滃弬鏁扮殑姊害锛氶偅涔堟垜浠氨鍙互鍒╃敤杩欎釜姊害鏉ユ敼杩涙垜浠殑浼樺寲鐩爣锛岃繘鑰屼娇缃戠粶鍙傛暟鎺ヨ繎鎴戜滑鏈熸湜鐨勬渶缁堢瓥鐣ュ弬鏁般€傛牴鎹瓥鐣ユ搴﹀畾鐞嗭紙涓嶇簿纭殑琛ㄨ揪寮忥級锛屾搴﹀彲浠ヨ绠楀涓嬶細锛堝鎺ㄥ杩囩▼鎰熷叴瓒ｇ殑璇昏€呭彲浠ュ弬鑰冨叾浠栬祫鏂欙紝濡傛枃鐚甗5]鐨勭浉鍏崇珷鑺傦級鐩磋涓婏紝绛栫暐姊害鍙互鐞嗚В涓哄畾鐞嗭細浠庣粰瀹氱殑涓€瀵圭姸鎬乻鍜屽姩浣滒潙庯紝濡傛灉鍙互鑾峰緱姝ｇ殑绱Н鍥炴姤锛堝嵆q蟺(s,饾憥)涓烘锛夛紝閭ｄ箞缁欏畾鐨勫姩浣滒潙庢槸鈥滄湁鐩婄殑鈥濓紝鍗虫搴︾殑鏇存柊搴旇鏈濈潃澧炲姞鍔ㄤ綔杈撳嚭姒傜巼蟺(饾憥|s;胃)鐨勬柟鍚戣繘琛屻€傜敱浜庤嚜鐒跺鏁板嚱鏁扮殑鍗曡皟鎬э紝杩欑浉褰撲簬鎻愰珮ln蟺(a鈭﹕;胃)銆傚熀浜庢搴︿俊鎭紝浼樺寲鍣ㄥ彲浠ヤ娇鐢ㄤ换浣曞熀浜庢搴︾殑浼樺寲鏂规硶鐩存帴浼樺寲绛栫暐缃戠粶鐨勫弬鏁般€傚湪娌℃湁瀵逛环鍊煎嚱鏁板缓妯＄殑鎯呭喌涓嬶紝鐘舵€?鍔ㄤ綔-浠峰€煎嚱鏁皅蟺(s,饾憥)鍙互鐢ㄨ挋鐗瑰崱娲涙潵閫艰繎锛屽嵆瀹屾垚涓€涓猠pisode(episode)鎺㈢储锛屽熀浜庢墍鏈夊崟姝ュ洖鎶ワ紝璁＄畻episode涓瘡涓€姝ョ殑绱Н濂栧姳锛屽苟鐢ㄨ鍔ㄤ綔鐨勭疮绉鍔辨潵杩戜技杩戜技鐘舵€?鍔ㄤ綔-浠峰€煎嚱鏁般€傝繖绉嶆柟娉曠О涓篟EINFORCE銆傚€煎緱娉ㄦ剰鐨勬槸锛岃挋鐗瑰崱娲涜繎浼煎瓨鍦ㄦ柟宸繃澶х殑椋庨櫓锛屽嵆姊害鐨勬柟鍚戝彲鑳藉湪鐩搁偦鐨勪紭鍖栨楠や箣闂村墽鐑堟尝鍔ㄣ€備负浜嗙紦瑙ｈ繖涓棶棰橈紝鍙互灏嗙瓥鐣ユ搴︿笌鍓嶉潰浠嬬粛鐨勬柟娉曠粨鍚堣捣鏉ャ€傛瘮濡傚湪actor-critic鏂规硶涓紝浣跨敤绁炵粡缃戠粶瀵箆alue杩涜寤烘ā锛屽彲浠ョ悊瑙ｄ负policygradient鍜寁aluelearning鐨勪簰琛ャ€備娇鐢ㄧ缁忕綉缁滃缓妯＄殑鍊煎嚱鏁帮紝鐢变簬绁炵粡缃戠粶鐨勫弬鏁颁笉浼氬彂鐢熷彉寮傦紝鍥犳杈撳嚭涔熶笉浼氬彂鐢熷彉寮傦紝浠庤€岀紦瑙ｄ簡涓婅堪鏂瑰樊闂銆傛渶鍚庨渶瑕佹寚鍑虹殑涓€鐐规槸锛屽湪绛栫暐姊害瀹氱悊鐨勬帹瀵间腑锛屽亣璁剧瓥鐣ョ綉缁滅殑鍙傛暟鏄疄鏃舵洿鏂扮殑銆傚洜姝わ紝鏈妭浠嬬粛鐨勫熀浜庣瓥鐣ユ搴﹀畾鐞嗙殑鏂规硶灞炰簬鍚岃建绛栫暐锛坥n-policy锛夛紝鍗筹細瀵逛簬瀹屾垚鎺㈢储鐨勮涓虹瓥鐣ュ拰杩涜鍙傛暟浼樺寲鐨勭洰鏍囩瓥鐣ユ槸鍏锋湁鐩稿悓鍙傛暟鐨勭綉缁溿€傜浉鍚岀殑绛栫暐涓嶈兘浣跨敤绂荤嚎鏀堕泦鐨勬暟鎹紝鑰岀浉鍙嶇殑绂荤瓥鐣ワ紙off-policy锛夊彲浠ャ€?7缁忓吀RL绠楁硶7.1Q-learning鍥?Q-learningQ-learning鏄憲鍚嶇殑鍩轰簬鏃堕棿宸垎鍜屾煡琛ㄦ硶鐨勫己鍖栧涔犳柟娉曪紝灞炰簬纭畾鎬х瓥鐣ワ紙璐績锛夌殑鏂规硶锛屽叾绛栫暐鏄彇state-action-value鍑芥暟鐨勬渶澶у姩浣滐紝鍏剁瓥鐣ユ洿鏂扮浉褰撲簬饾憺(饾憼,饾憥)鐨勬洿鏂般€傜‘瀹氭€х瓥鐣ョ殑闂鏄細澶卞幓鎺㈢储鑳藉姏锛屾墍浠ヤ竴鑸噰鐢潨€鈭掟潙旔潙燄潙掟潙掟潙戰潙︾瓥鐣ワ紝浠ュ皬姒傜巼饾渶闅忔満閫夋嫨鍔ㄤ綔銆俀-learning棣栧厛鏇存柊饾憺(饾憼t,饾憥t)銆愯缁冦€戯紝鐒跺悗浣跨敤饾憯饾憻饾憭饾憭饾憫饾懄閫夋嫨鍔ㄤ綔銆愬喅绛栥€戯紱杩欑璁粌鍜屽喅绛栧簭鍒椾笉涓€鑷寸殑鏂规硶灞炰簬off-policy锛坥ff-trackstrategy锛夛紱濡傛灉璁粌鍜屽喅绛栧簭鍒椾竴鑷达紝鍒欏睘浜巓n-policy锛堝湪杞ㄧ瓥鐣ユ垨鍚岃建绛栫暐锛夈€係arsa鏄竴绉嶇被浼间簬Q-learning鐨刼n-policy绠楁硶銆?.2SarsaSarsa鍜孮-learning鍩烘湰涓€鏍凤紝鍞竴涓嶅悓鐨勬槸閫氳繃饾渶鈭掟潙旔潙燄潙掟潙掟潙戰潙﹂€夋嫨涓嬩竴涓姩浣滒潙巘+1锛岀劧鍚庢牴鎹繖涓姩浣滄洿鏂癚-Table锛堣缁冿級銆傝缁冨簭鍒椾笌鍐崇瓥搴忓垪涓€鑷淬€?.3鍏朵粬缁忓吀绠楁硶/楂樼骇鎶€鏈杩板熀浜嶲-Table鐨勬柟妗堝彧鑳藉绾虫湁闄愮殑鐘舵€佸拰鍔ㄤ綔锛涘鏋滄湁寰堝鍔ㄤ綔鍜岀姸鎬侊紝鍩轰簬Q-Table鐨勬柟妗堜篃浼氬け鍘讳竴浜涙硾鍖栬兘鍔涖€傚洜姝わ紝鍙互鍒╃敤DNN鐨勬嫙鍚堝拰娉涘寲鑳藉姏锛岀洿鎺ラ€氳繃鐘舵€佸姩浣滃涔犫€淨-Table鈥濄€傚浘10DQNQ-learning鐨勬洿鏂拌鍒欐槸锛氬畠鐨勭洰鏍囨槸璁╁畠瓒冲鎺ヨ繎锛岄偅涔圖QN鐨凩ossfunction鍙互璁句负涓よ€呯殑鍧囨柟璇樊褰㈠紡锛氬彲浠ョ湅鍒拌宸洰鏍囧寘鎷簡杈撳嚭绁炵粡缃戠粶锛屽苟鏇存柊缃戠粶鍙傛暟锛屽悓鏃朵紭鍖栫洰鏍囦笉鏂彉鍖栵紝瀹规槗閫犳垚绁炵粡缃戠粶璁粌涓嶇ǔ瀹氥€備负浜嗚В鍐宠繖涓棶棰橈紝DQN浣跨敤浜唗argetnetwork鐨勬€濇兂锛涘嵆浣跨敤涓ょ粍Q缃戠粶锛堣缁冪綉缁滃拰鐩爣缃戠粶锛夛紝鍒嗗埆浣跨敤鐩爣缃戠粶璁＄畻閮ㄥ垎锛屾瘡N姝ヨ绠楃洰鏍囩綉缁滅殑鍙傛暟銆傚皢涓庤缁冪綉缁滃悓姝ヤ竴娆°€俀-learning銆丼arsa銆丏QN閮藉睘浜庡厛瀛︿範浠峰€煎嚱鏁帮紝鐒跺悗鏍规嵁浠峰€煎嚱鏁帮紙閫氬父鏄熀浜巈-greedy锛夊緱鍒版渶浼樼瓥鐣ャ€傝鏂规硶灞炰簬浠ュ€间负鏈殑鏂规硶锛涗篃鍙互鐩存帴瀛︿範绛栫暐鍑芥暟蟺胃锛岄偅涔堝畠灞炰簬policy-base锛宲olicy-base姣攙alue-base鏈夋洿濂界殑鎺㈢储鑳藉姏銆?.4鍏朵粬缁忓吀绠楁硶/楂橀樁鎶€鏈杩伴櫎浜嗙粡鍏哥殑RL绠楁硶锛屽己鍖栧涔犵爺绌剁ぞ鍖?琛屼笟杩樻彁鍑轰簡璁稿鏇撮珮绾у埆鐨勬妧鏈€傝繖浜涢珮闃舵妧鏈紝缁撳悎缁忓吀/鏂版彁鍑虹殑绠楁硶妗嗘灦锛岀紦瑙ｄ簡寮哄寲瀛︿範涓殑鍚勭闂锛堣缁冪洰鏍囨姈鍔ㄣ€佷环鍊奸珮浼般€佺█鐤忓鍔辩瓑锛夛紝鏈夊姪浜庡己鍖栧涔犵爺绌剁殑杩涙鍜屽湪璇ラ鍩熺殑鎴愬姛瀹炶返銆傝涓氥€傝础鐚簡銆傞檺浜庣瘒骞咃紝涓嬮潰绠€鍗曞垪涓句竴浜涙湁浠ｈ〃鎬х殑鍐呭锛氱畻娉?鎶€宸у拰鏂规硶鍦ㄨ缁冭繃绋嬩腑寮曞叆baselineREINFORCE鍜宐aselineA2C锛屽噺灏戦殢鏈烘搴︾殑鏂瑰樊锛岀ǔ瀹氳缁冪殑澶氬€肩綉缁淭D3缂撹В楂樹及闂鍦ㄩ€氳繃浼樺寲澶氱粍浠峰€肩綉缁滐紙鐩稿悓缁撴瀯锛屼笉鍚屼笖鐙珛鐨勫弬鏁帮級璁＄畻TD鐩爣鏃讹紝鍦ㄦ墍鏈夛紙鐩爣锛変环鍊肩綉缁滀腑浣跨敤鏈€灏忓€肩喌姝ｅ垯SAC鏉ラ紦鍔卞湪涓坊鍔犱竴涓喌璁粌闃舵鐨別xplorationobjectivefunction锛堟帹瀵煎悗璁粌杩囩▼浼氶€氳繃sub-networkloss锛宲olicygradientextraitems绛夋柟寮忚繘鍏ヨ缁冭繃绋嬶級RandomactionTD3榧撳姳璁粌闃舵鐨別xploration銆傚湪鐢熸垚璁粌搴忓垪鐨勮繃绋嬩腑锛屽湪绛栫暐缃戠粶杈撳嚭鐨勫姩浣滀腑鍔犲叆鍣０杩涜鎵板姩锛岄紦鍔辨帰绱㈡洿澶氱殑鐘舵€併€傜┖闂村弬鑰?鏉愭枡[1]銆係utton銆丷ichardS.鍜孉ndrewG.Barto銆傗€滃己鍖栧涔狅細浠嬬粛銆傗€漅obotica17.2(1999)锛?29-235銆俒2].https://www.analyticsvidhya.c...[3].https://arshren.medium.com/su...[4]銆俬ttps://hrl.boyuai.com/chapte...[5].銆婃繁搴﹀己鍖栧涔犮€嬬帇鏍戞．绛夛紝https://github.com/wangshusen...
