缁村熀鐧剧浠嬬粛锛氳瘝娉曞垎鏋愭槸璁＄畻鏈虹瀛︿腑灏嗗瓧绗﹀簭鍒楄浆鎹负鏍囪搴忓垪鐨勮繃绋嬨€傛墽琛岃瘝娉曞垎鏋愮殑绋嬪簭鎴栧嚱鏁扮О涓鸿瘝娉曞垎鏋愬櫒銆傚師绋嬪簭浠ｇ爜濡備笅add_result=1+2锛岄€氳繃璇嶆硶鍒嗘瀽锛屽緱鍒板涓嬬粨鏋滐細NAME`add_result`0,0SYMBOL`=`0,11INT`1`0,13SYMBOL`+`0,15INT`2`0,17鎺掑垪鎴愯〃鏍糉ormTagTypeLiteralLineNumberColumnNumberNAMEadd_result00SYMBOL=011INT1013SYMBOL+015INT2017鎴戜滑鍙互鐢℅o璇█杞绘澗瀹炵幇鍙敤鐨勮瘝娉曞垎鏋愬櫒馃槂锔廏o璇█瀹炵幇璇嶆硶鍒嗘瀽鍣ㄥ寘mainimport("fmt""regexp""unicode/utf8""os")varexprs=[]string{"\\d+","[\\p{L}\\d_]+","[\\+\\-=]"}鍙橀噺鍚嶇О=[]string{"INT","NAME","SYMBOL"}funcmain(){rules:=[]*regexp.Regexp{}fori,expr:=rangeexprs{rule,_:=regexp.Compile("^"+expr)rules=append(rules,rule)fmt.Println(names[i],rule)}fmt.Println("-------------------------------")瀵逛簬琛岋紝浠ｇ爜锛?鑼冨洿os.Args[1:]{浣嶇疆锛?0col锛?0瀵逛簬鐪焮瀵逛簬浣嶇疆=len(code){break}source:=""tokenType:=-1fori,rule:=rangerules{source=rule.FindString(code[position:])ifsource!=""{tokenType=ibreak}}iftokenType>=0{fmt.Printf("%s\t`%s`\t%d\t%d\n",names[tokenType],source,row,col)浣嶇疆+=len(source)col+=utf8.RuneCountInString(source)}else{fmt.Printf("errorin:%d,%d\n",row,col)break}}}}鍦ㄥ懡浠よ杩愯娴嬭瘯鉃済orunlexer.go"Value=PI+100"INT^\d+NAME^[\p{L}\d_]+SYMBOL^[\+-=]----------------------------NAME`value`00SYMBOL`=`03NAME`PI`05SYMBOL`+`08INT`100`010Go璇█浠ｇ爜璇存槑闇€瑕佸鍏ュ埌鍖咃細鍖呬富瑕佸鍏ワ紙鈥渇mt鈥濃€渞egexp""unicode/utf8""os")fmt鐢ㄤ簬鎵撳嵃鍑簉egexp姝ｅ垯琛ㄨ揪寮弖nicode/utf8缁熻utf8涓鏂囩殑涓暟os鑾峰彇鐢ㄦ埛杈撳叆鎸囧畾姝ｅ垯琛ㄨ揪寮忓拰瀛楁绫诲瀷鍚嶇О锛歷arexprs=[]string{"\\d+","[\\p{L}\\d_]+","[\\+\\-=]"}varnames=[]string{"INT","NAME","SYMBOL"}鍒涘缓涓や釜瀛楃涓叉暟缁勫瓨鏀炬鍒欒〃杈惧紡鍜屽搴旂殑瀛楁绫诲瀷鍚嶏紝鍒濆鍖栧瓧娈靛尮閰嶈鍒欙細funcmain(){rules:=[]*regexp.Regexp{}fori,expr:=rangeexprs{rule,_:=regexp.Compile("^"+expr)rules=append(rules,rule)fmt.Println(names[i],rule)}闇€瑕佹敞鎰忕殑鏄紝鍦ㄦ瘡涓鍒欒〃杈惧紡鐨勫ご閮ㄤ箣鍓嶅繀椤绘彃鍏鐢ㄤ簬纭繚鍖归厤鐨勫瓧绗︿覆鍖呭惈鏈€宸﹁竟鐨勫瓧绗︼紝閬垮厤鈥滆烦杩囧尮閰嶁€濄€傚惊鐜尮閰嶅瓧娈碉細forrow,code:=rangeos.Args[1:]{position:=0col:=0fortrue{forposition=len(code){break}source:=""tokenType:=-1fori,rule:=range瑙勫垯{source=rule.FindString(code[position:])ifsource!=""{tokenType=ibreak}}iftokenType>=0{fmt.Printf("%s\t`%s`\t%d\t%d\n",names[tokenType],source,row,col)position+=len(source)col+=utf8.RuneCountInString(source)}else{fmt.Printf("errorin:%d,%d\n",row,col)break}}}浣跨敤閬嶅巻os.Args[1:]鐨勬柟娉曞皢鐢ㄦ埛杈撳叆鐨勬瘡涓弬鏁颁綔涓轰竴琛屼唬鐮佽繘琛岃瘝娉曞垎鏋愩€傝烦杩嘯蹇界暐]绌哄瓧绗︼細forposition=len(code){break}閬嶅巻鍖归厤瑙勫垯骞跺皾璇曞尮閰嶏細source:=""tokenType:=-1fori,rule:=rangerules{source=rule.FindString(code[position:])ifsource!=""{tokenType=ibreak}}寰幆閬嶅巻璁剧疆鐨勮鍒欒繘琛屽尮閰嶏紝鎴愬姛鍒欒缃笅鏍囦负tokenType鐨勫€硷紝濡傛灉涓嶅尮閰嶅垯tokenType榛樿鍒?1銆傛牴鎹尮閰嶇粨鏋滃垽鏂悗缁姩浣滐細iftokenType>=0{fmt.Printf("%s\t`%s`\t%d\t%d\n",names[tokenType],source,row,col)position+=len(source)col+=utf8.RuneCountInString(source)}else{fmt.Printf("errorin:%d,%d\n",row,col)break}濡傛灉tokenType涓嶆槸-1锛岄偅涔坢atch濡傛灉鎴愬姛锛屼細鎵撳嵃瀛楁鍚嶃€佸瓧闈㈠€笺€佽鍒椾俊鎭紝骞惰缃烦杩囧綋鍓嶅瓧娈电殑浣嶇疆銆傞渶瑕佹敞鎰忕殑鏄紝浠庝笅涓€涓瓧娈靛紑濮嬬殑鍒楀彿col鐨勫閲忥紝闇€瑕佷娇鐢╱tf8绗︽枃璁℃暟鏂瑰紡鑾峰彇锛屽惁鍒欎細閬囧埌涓€浜泆nicode/utf8缂栫爜鏃犳硶姝ｇ‘鎸囧悜銆侾ython浣跨敤鑰呬篃鍙互杞绘澗鐨勫疄鐜梆煒侊笍Python璇嶆眹鍒嗘瀽鍣╥mportreimportsysexprs=['\\d+','\\w+','[\\+\\-=]']names=['INT','NAME','SYMBOL']defmain():rules=[]fori,exprinenumerate(exprs):rules.append(re.compile('^'+expr))print(names[i],rules[-1].pattern)print('-'*32)forrow,codeinenumerate(sys.argv[1:]):position=0whileTrue:whileposition=len(code):breaksource=''tokenType=-1fori,ruleinenumerate(rules):缁撴灉=rule.findall(code[position:])濡傛灉len(result)>0:source=result[0]tokenType=ibreak濡傛灉tokenType>=0:print(f'{names[tokenType]}\t`{source}`\t{琛寎\t{浣嶇疆}')position+=len(source)else:print(f'errorin{row},{position}')breakif__name__=="__main__":main()浣滀负琛ュ厖鍐呭杩欓噷涔熸彁渚汣++鏂规馃槅锔廋++鐜拌瘝娉曞垎鏋愬櫒#include#include#include#include#includestd::vectorexprs{L"\\d+",L"\\w+",L[\\+\\-=]"};std::vectornames{"INT","NAME","SYMBOL"};intmain(intargc,char*argv[]){std::locale鏃х殑锛泂td::locale::global(std::locale("en_US.UTF-8"));std::wstring_convert>codecvt_utf8锛泂td::vector瑙勫垯锛涘浜?size_ti=0,count=exprs.size();i=0){autosource=match.str(0);printf("%s\t`%s`\t%d\t%ld\n",names[tokenType].c_str(),codecvt_utf8.to_bytes(source).c_str(),琛?浣嶇疆);浣嶇疆+=source.size();}鍒殑{printf("閿欒:%d,%ld\n",琛?浣嶇疆);浼戞伅;}}}std::locale::global(鏃?;return0;}涓嬩竴绡囥€婁娇鐢˙KLexer杩涜璇嶆硶鍒嗘瀽銆?/p>
