当前位置: 首页 > Linux

记一次用gdb诊断gc问题的全过程

时间:2023-04-06 04:57:09 Linux

鍘熷垱锛氭寜閽棩璁帮紙寰俊鍏紬鍙稩D锛歝odelogs锛夛紝娆㈣繋鍒嗕韩锛岃浆杞借淇濈暀鍑哄銆傚墠瑷€鍦ㄨВ鍐充簡涓婃GC鑰楁椂闀跨殑闂鍚庯紝绯荤粺绋冲畾浜嗗緢澶氥€傝繖鏄箣鍓嶇殑鏂囩珷銆奊C鑰楁椂楂橈紝鍘熷洜绔熸槸鏈嶅姟娴侀噺灏忥紵銆嬩笉杩囪繃浜嗕竴娈垫椂闂达紝鎴戝湪鏌ョ湅GC鏃ュ織鐨勬椂鍊欙紝鍙堝彂鐜颁簡涓€涓狦C闂锛屽涓嬶細浠庤繖涓浘涓彲浠ュ彂鐜帮紝鎴戜滑鐨凣C鏈変竴浜涘皷宄帮紝鏈夋椂澶у唴瀛樺垎閰嶉噺绐佺劧鍙戠敓銆傛煡鐪婫C鏃ュ織锛屽彂鐜版湁澶у璞″垎閰嶇殑璁板綍锛屽涓嬶細$grep'concurrenthumongousallocation'gc.log|awk'match($0,/allocationrequest:(\w+)bytes/,a){printa[1]}'|sort-nr19418357841889656848鍙互鐪嬪嚭涓€娆″ぇ瀵硅薄鍒嗛厤鐨勫垎閰嶅ぇ灏忎负1.9G銆傝繖璋佽兘鎶楁嫆锛乤sync-profiler瀹氫綅澶у璞″垎閰嶄笂鏂囨彁鍒帮紝浣跨敤async-profiler鍙互杞绘澗瀹氫綅澶у璞″垎閰嶇殑璋冪敤鏍堬紝鏂规硶濡備笅锛?/profiler.shstart--all-user-eG1CollectedHeap::humongous_obj_allocate-f./humongous.jfrjps鐒跺悗鐢╦mc鎵撳紑humongous.jfr鏂囦欢锛岃皟鐢ㄦ爤濡備笅锛氳繖鏄竴涓猼hrift鍙嶅簭鍒楀寲鎿嶄綔锛岃皟鐢ㄤ簡TCompactProtocol.readDouble鏂规硶锛屾柟娉曚唬鐮佸涓?浣嗘槸杩欓噷鍙垱寤轰簡8涓猙ytes鏁扮粍涓嶅彲鑳藉垎閰?.9G鍐呭瓨锛岀湡鏄鎬紒缁忚繃涓€鐣簡瑙o紝杩欐槸鍥犱负async-profiler鏄€氳繃AsyncGetCallTrace鑾峰彇璋冪敤鍫嗘爤鐨勶紝鑰孉syncGetCallTrace鑾峰彇鐨勫爢鏍堟湁鏃舵槸涓嶅噯纭殑銆侸ava绀惧尯宸茬粡鍙嶉浜嗚繖涓棶棰橈紝鐩墠杩樻病鏈夎В鍐炽€傞棶棰橀摼鎺ワ細https://bugs.openjdk.org/brow...瀵绘壘鍏朵粬鐨則racerlinux涓婃湁寰堝鍐呮牳妯″紡鐨則racer锛屾瘮濡俻erf,bcc,systemtap锛屼絾鏄兘闇€瑕乺oot鏉冮檺锛屼笉鍙兘涓烘垜鐢宠杩欎釜Permission馃様鍦ㄧ敤鎴锋ā寮忎笅锛屾湁鍩轰簬ptrace绯荤粺璋冪敤瀹炵幇鐨剆trace鍜宭trace銆傝瘯杩囦簡锛屾病娉曠洿鎺ヨ拷韪埌G1涓殑澶у璞″垎閰嶅嚱鏁癎1CollectedHeap::humongous_obj_allocate銆備篃鍦ㄧ綉涓婃壘浜嗗ソ鍑犲ぉ锛屽笇鏈涜兘鎵惧埌涓€涓ソ鐢ㄧ殑绾敤鎴锋€乼racer锛屽彲鎯滄壘浜嗗嚑澶╅兘娌℃壘鍒般€傛渶鍚庡彧鑳介拡瀵筩/c++璋冭瘯宸ュ叿gdb銆傛垜鎯筹紝鏃㈢劧gdb鏄竴涓皟璇曞伐鍏凤紝閭h偗瀹氭槸鍙互鏌ョ湅鎸囧畾鍑芥暟鐨勮皟鐢ㄥ弬鏁板拰璋冪敤鏍堢殑锛屽彧瑕佹壘鍒板搴旂殑鐢ㄦ硶灏卞彲浠ヤ簡锛佸湪缂栧啓gdb鑴氭湰鏂归潰缁忚繃涓€鐣涔犲拰鎽哥储锛圥S锛氬叾瀹炴垜鑺变簡灏嗚繎2鍛煒傦級锛岀粓浜庡啓鍑轰簡涓€涓疄鐢ㄧ殑gdb鑴氭湰锛屽涓嬶細handleallnostopnoprintpasshandleSIGINTstopprintnopassbreak*(_ZN15G1CollectedHeap22humongous_obj_allocateEmh+0x58c06f-0x58c060)while1continue#濡傛灉鏄疌trl+c,exitif$_siginfoif$_siginfo.si_signo==2detachquitendendprintf"word_sizeis%d\n",$rsiif$rsi>100*1024*1024/8#鎵撳嵃褰撳墠鏃堕棿shelldate+%FT%T#鎵撳嵃褰撳墠绾跨▼thread#鎵撳嵃褰撳墠璋冪敤鏍坆tpythonimportsubprocess#鍚慾vm鍙戦€乲ill-3淇″彿锛屽嵆SIGQUIT淇″彿pythonproc=subprocess.Popen(['kill','-3',str(gdb.selected_inferior().pid)],stdout=subprocess.PIPE,stderr=subprocess.PIPE,bufsize=1,universal_newlines=True)pythonstdout,stderr=proc.communicate()pythonnprint(stdout)pythonprint(stderr)detachquitendend娌″杩噂db鐨勫悓瀛﹀彲鑳界湅涓嶆噦锛屾病鍏崇郴锛屾斁杞绘澗銆俬andleallnostopnoprintpasshandleSIGINTstopprintnopass杩欎袱涓彞鏌勬槸鐢ㄦ潵澶勭悊Linux淇″彿鐨勩€傜敱浜庢垜浠笉闇€瑕佽皟璇曚俊鍙烽棶棰橈紝鎵€浠ヨgdb涓嶅鐞嗕俊鍙凤紝鎸塁trl+c鏃朵繚鎸丼IGINT閫€鍑篻db鑴氭湰銆俠reak*(_ZN15G1CollectedHeap22humongous_obj_allocateEmh+0x58c06f-0x58c060)杩欎釜break鏄粰G1涓殑澶у璞″垎閰嶅嚱鏁癎1CollectedHeap::humongous_obj_allocate璁剧疆鏂偣銆傝鏂规硶鐨勬簮鐮佸涓嬶細word_size鍙傛暟琛ㄧず鍐呭瓨鍒嗛厤浜嗗灏戜釜瀛楋紝鑰屽湪64浣嶆満鍣ㄤ笂锛?涓瓧绛変簬8涓瓧鑺傦紝鎵€浠ュ鏋滆兘杩借釜鍒拌繖涓弬鏁扮殑鍊硷紝鍙互鐭ラ亾姣忔鍒嗛厤鐨勫ぇ瀵硅薄鐨勫ぇ灏忋€傜敱浜嶫VM鏄敤C++鍐欑殑锛岃€孋++缂栬瘧鍣ㄤ細閲嶅啓鍑芥暟鍚嶏紙mangle锛変互鍏煎C鐨凙BI锛屾墍浠ョ紪璇戝悗鐨勫嚱鏁板悕鍙樻垚浜嗗鎬殑_ZN15G1CollectedHeap22humongous_obj_allocateEmh锛岄€氳繃nm鏌ヨ浜岃繘鍒舵枃浠剁殑绗﹀彿琛紝寰楀埌鍚嶅瓧銆?whichjava/usr/local/jdk/jdk1.8.0_202/bin/java#jvm-relatedimplementationsareallinthelibjvm.sodynamiclibrary$find/usr/local/jdk/jdk1.8.0_202|greplibjvm.so/usr/local/jdk/jdk1.8.0_202/jre/lib/amd64/server/libjvm.so$nm/usr/local/jdk/jdk1.8.0_202/jre/lib/amd64/server/libjvm.sogrephumongous_obj_allocate000000000058c060t_ZN15G1CollectedHeap22humongous_obj_allocateEmh000000000058b1a0t_ZN15G1CollectedHeap41humongous_obj_allocate_initialize_regionsEjjmh鍐嶇湅鍥炰箣鍓嶈缃柇鐐圭殑鑴氭湰浠g爜锛歜reak*(_ZN15G1CollectedHeap22humongous_obj_allocateEmh+0x58c06f-0x58c060)+0x58c06f-0x58c060杩欎釜鏄湪鍋氬湴鍧€鍋忕Щ鎿嶄綔锛屼簡瑙h繃姹囩紪鐨勫悓瀛﹀簲璇ユ竻妤氾紝璋冪敤鍑芥暟鍚庯紝鍑芥暟Someassemblyinstructionsatthebeginningaregenerallyrelatedtoparameterregisteroperations.Thex86parameterregistersareasfollows:rdiindicatesthefirstparameterrsiindicatesthesecondparameterrdxindicatesthethirdparameterrcxindicatesthefourthparameterr8indicatesthefifthparameterr9Itmeansthatthesixthparametercanuseobjdumptodisassemblelibjvm.so,andlookattheassemblycodetodeterminewhichlineofinstructionthebreakpointshouldbeshiftedto.Theofficialsholdon,andtheassemblyrelatedcontentisasfollows馃榿$objdump-d/usr/local/jdk/jdk1.8.0_202/jre/lib/amd64/server/libjvm.so|less-Sandthensearchforthe_ZN15G1CollectedHeap22humongous_obj_allocateEmhfunctioninless,asfollows:Thereasonforaddingtheoffsetisbecauseitisat+0x58c06f-Afterthepositionof0x58c060,thersiregister(thesecondparameter)willhaveavalue.Thereasonwhythevalueofeachsecondparameterisobtainedis鍥犱负鍦–++瀵硅薄缂栫▼涓紝绗竴涓弬鏁板氨鏄痶his锛屽悗闈㈢殑閫昏緫寰堝鏄撶悊瑙o紝濡備笅锛氶鍏堟槸寰幆锛岀劧鍚巆ontinue灏辨槸璁╃▼搴忚繍琛屻€傚綋绋嬪簭閬囧埌鏂偣鏃讹紝灏嗘墽琛宑ontinue銆備腑闂存槸淇″彿澶勭悊锛屼富瑕佹槸璁〤trl+c閫€鍑哄惊鐜€傛渶鍚庨€氳繃print鎵撳嵃鍑簉si鐨勫€硷紝浠庤€岃窡韪埌word_size鍙傛暟鐨勫€笺€傜劧鍚庢墦鍗扮嚎绋嬪拰璋冪敤鏍堜俊鎭紝濡備笅锛氬綋鍒嗛厤鐨勫唴瀛樺ぇ浜?00M鏃讹紝鎵撳嵃褰撳墠鏃堕棿銆佸綋鍓嶇嚎绋嬪拰褰撳墠璋冪敤鏍堛€備絾鏄痝db鐨刡t鍛戒护鎵撳嵃鐨勮皟鐢ㄦ爤鏄繖鏍风殑锛氬洜涓簀ava鏄В閲婃墽琛岀殑锛屾墍浠ヨ幏鍙栦笉鍒癹ava閮ㄥ垎鐨勮皟鐢ㄦ爤bt銆傛病鏈塲ava璋冪敤鍫嗘爤锛岃繖涓窡韪剼鏈槸韫╄剼鐨勩€傚湪杩欓噷鍗′簡寰堜箙锛岃瘯浜嗗緢澶氭柟娉曘€傜啛鎮塲ava鐨勫簲璇ラ兘鐭ラ亾jvm鏈変竴涓殣钘忕殑璇婃柇鍔熻兘銆傚鏋滅粰jvm杩涚▼鍙戦€丼IGQUIT淇″彿锛宩vm浼氬湪鏍囧噯杈撳嚭涓墦鍗扮嚎绋嬫爤淇℃伅锛岃€孲IGQUIT淇″彿鍙互閫氳繃kill-3鍙戦€侊紝鎵€浠ュ氨鏈変簡涓嬮潰鐨勪唬鐮侊細gdb鐪熷帀瀹筹紝鍐呭祵python鎵╁睍锛岄€氳繃python鐨剆ubprocess鍖咃紝鍙互鎵цkill-3鍛戒护銆備笅闈㈢殑detach鍜宷uit鏄敤鏉ラ€€鍑篻db鐨勶紝灏变笉娣辩┒浜嗐€傝繍琛実db鑴氭湰璺熻釜澶у璞″皢涓婇潰鐨刧db鑴氭湰鍛藉悕涓簍race.gdb锛岀劧鍚庝娇鐢╣db鍛戒护杩愯锛屽涓嬶細$gdb-q--batch-xtrace.gdb-p`pgrepjava`wherepgrepjava鐢ㄤ簬鑾峰彇java杩涚▼鐨勮繘绋婭D銆傛敞鎰忥細gdb鏈川涓婃槸涓€涓熀浜巔trace绯荤粺璋冪敤鐨勮皟璇曞櫒銆傚綋鍛戒腑鏂偣鏃讹紝杩涚▼鐨勫垏鎹㈠紑閿€寰堝ぇ锛屾墍浠ヨ繖绉嶆柟娉曞彧鑳借窡韪笉缁忓父璋冪敤鐨勫嚱鏁般€傝繍琛屽悗锛岃窡韪埌鐨勫弬鏁板拰绾跨▼淇℃伅濡備笅锛歀WP鍚庣殑166涓虹嚎绋嬪彿锛岃浆鎴?6杩涘埗涓?xa6銆傜劧鍚庡幓java杩涚▼鐨勬爣鍑嗚緭鍑烘棩蹇椾腑鎵惧埌杩欎釜绾跨▼鐨凧ava璋冪敤鏍堬紝濡備笅锛氬ぇ瀵硅薄鍒嗛厤鏄敱readBinary鍑芥暟鍙戣捣鐨勶紝璋冭瘯杩欎釜鍑芥暟锛屽涓嬶細澶╁摢锛屽畠鍒涘缓浜嗕竴涓法澶х殑瀛楄妭鏁扮粍锛岄毦鎬細鏈?.9G鐨勫ぇ瀵硅薄鍒嗛厤锛乺eadBinary鐨勮皟鐢ㄧ敱浠ヤ笅浠g爜瑙﹀彂锛歍ProtocolFactoryfactory=newTCompactProtocol.Factory();TDeserializer瑙d覆鍣?newTDeserializer(宸ュ巶);deserializer.deserialize(deserializeObj,sourceBytes);section鏁扮粍琚弽搴忓垪鍖栦负deserializeObj瀵硅薄銆傚綋sourceBytes鏄粠deserializeObj瀵硅薄搴忓垪鍖栬繃鏉ョ殑鏃跺€欙紝鍙嶅簭鍒楀寲鏄病鏈夐棶棰樼殑銆傚綋sourceBytes娌℃湁琚玠eserializeObj瀵硅薄搴忓垪鍖栨椂锛屽綋鍙嶅簭鍒楀寲浠g爜浠巗ourceBytes涓В鏋愬嚭瀛楁闀垮害锛坙ength锛夋椂锛屽畠鍙兘鏄换鎰忓€硷紝杩欏彲鑳戒細瀵艰嚧鍒涘缓涓€涓潪甯稿ぇ鐨勫瓧鑺傛暟缁勩€備絾鏄垜浠啓杩欐浠g爜鏄负浜嗘娴媠ourceBytes鏄惁琚玠eserializeObj搴忓垪鍖栵紝鎵€浠ourceBytes鏈夊彲鑳芥病鏈夎deserializeObj搴忓垪鍖栵紒绠€鍗曠湅浜嗕竴浼歵hrift浠g爜锛屽彂鐜板彲浠ラ檺鍒跺瓧娈电殑鏈€澶ч暱搴︼紝濡備笅锛氭兂涓€鎯筹紝涓€涓弽搴忓垪鍖栫殑瀛楁闀垮害鑲畾涓嶄細鍜屾暣涓弽搴忓垪鍖栨暟鎹竴鏍烽暱锛屾墍浠ヤ娇鐢╯ourceBytes.length鏉ラ檺鍒跺畠銆俆ProtocolFactoryfactory=newTCompactProtocol.Factory(sourceBytes.length,sourceBytes.length);TDeserializer瑙d覆鍣?newTDeserializer(宸ュ巶);deserializer.deserialize(deserializeObj,sourceBytes);鍙嶅簭鍒楀寲寮傚父锛岃鏄庡綋鍓峴ourceBytes娌℃湁琚玠eserializeObj搴忓垪鍖栥€傛€荤粨涓€涓嬶紝鍐欒繖涓猤db鑴氭湰鐪熺殑鏄姳浜嗘垜寰堝鏃堕棿锛屽洜涓烘垜浜嬪厛涓嶇煡閬揼db鑳戒笉鑳藉仛杩欎釜锛岃€屼笖鎴戝張涓嶆槸C/C++绋嬪簭鍛橈紝鎵€浠ュ姹囩紪鐩稿叧鐨勪笉鐔熸倝鐭ヨ瘑锛屼腑闂存湁濂藉嚑娆℃兂鏀惧純馃槪杩樺ソ鏈€鍚庢垚鍔熶簡锛岃鎴戣蛋涓婁簡涓€鏉℃柊鐨勮В棰樿矾寰勶紝杩樻槸寰堝€煎緱鐨勷煒?/p>