鏈哄櫒浜哄崗璁満鍣ㄤ汉鍗忚锛屽張绉版満鍣ㄤ汉鍗忚锛屼富瑕佺敤浜庢悳绱㈠紩鎿庢姄鍙栫綉绔欓〉闈€傪煍峳obots.txt鏂囦欢涓€鑸瓨鏀惧湪缃戠珯鐨勬牴鐩綍涓嬨€傝鍗忚涓昏鏄秷闄ょ埇鍙栦笉蹇呰鐨勭綉绔欒矾寰勩€傚浜庢湁閽堝鎬х殑鐖鍔ㄧ墿銆傪煒勬病鏈変换浣曟剰涔夈€傝繖灏卞儚鍛婅瘔灏忓伔涓嶈鍋蜂綘鐨勪笢瑗夸竴鏍枫€傚熀浜庨伒寰猂obots鍗忚鐨勭埇铏紝瀹冮鍏堜細妫€鏌ョ珯鐐规牴鐩綍涓嬫槸鍚︽湁robots.txt鏂囦欢锛屽鏋滃瓨鍦ㄥ垯鏍规嵁鍏朵腑瀹氫箟鐨勭埇琛岃寖鍥磋繘琛岀埇鍙栥€傚鏋滄病鏈夛紝鍒欑洿鎺ヨ闂椤甸潰銆俁obotsSpecificationUserAgentDirectivesuseuser-agentdirectives鎸囧畾閫傜敤浜庢墍鏈夌埇铏殑瑙勫垯锛?Disallow:/Userdisallow鎸囧畾url鍚庣紑鍚庨潰璺?User鐨勪細琚睆钄姐€俛llow鍛戒护鍙互閫氳繃allow鍛戒护閬垮厤琚玠isallow灞忚斀鐨勯摼鎺ワ細User-agent:*Allow:/User/007Disallow:/Userisallowedtocrawl/User/007鍦╠isallow鎸囧畾鐨剈rl鍚庣紑鍚庤窡/鐨勯〉闈箣鍚庣敤鎴疯闃绘銆傞摼鎺ュ湴鍧€銆係itemap鎸囦护涓昏鐢ㄤ簬璇嗗埆绔欑偣鍦板浘锛歎ser-agent:*Disallow:/wp-admin/Allow:/wp-admin/admin-ajax.phpSitemap:https://www.lubaogui.com/wp-sitemap.xml鍦ㄥ悇涓悳绱㈠紩鎿庣殑绔欓暱宸ュ叿涓婇兘鏈変娇鐢紝鏂逛究鎼滅储寮曟搸鏀跺綍馃攳銆俁obots浣跨敤robotparser妯″潡妫€鏌ユ槸鍚﹂伒寰猺obots鍗忚锛屼唬鐮佸涓嬶細fromurllib.robotparserimportRobotFileParserfromurllib.requestimporturlopenrp=RobotFileParser()rp.parse(urlopen('https://www.lubaogui.com/robots.txt')read()decode('utf-8').split('n'))print(rp.can_fetch('*','https://www.lubaogui.com/96'))===鎵撳嵃缁撴灉===True馃懆鈥嶐煉昏櫧鐒剁埇铏竴鑸笉浼氶伒寰猺obots.txt鍗忚锛屼絾杩樻槸寤鸿澶у鍦ㄧ埇鍙栫殑鏃跺€欏悎鐞嗗鐞嗘彁鍙栨晥鐜囥€傞伩鍏嶅奖鍝嶇洰鏍囩珯鐐圭殑璐熻浇銆?/p>
