大家好，我是程序员二！如果有一个文件，里面有100,000个url，需要向每个url发送一个http请求，并打印请求结果的状态码，如何编写代码尽快完成这些任务？Python中并发编程的方法有很多种。多线程标准库threading、concurrency、coroutineasyncio，当然还有异步库grequests，每一个都可以达到上面的需求，下面我们用代码来一一实现。本文代码可以直接Run，供你以后并发编程参考：队列+多线程定义一个队列，大小为400，然后启动200个线程，每个线程不断从队列中获取和访问url。主线程读取文件中的url放入队列，然后等待队列中的元素全部接收并处理。代码如下：fromthreadingimportThreadimportsysfromqueueimportQueueimportrequestsconcurrent=200defdoWork():whileTrue:url=q.get()status,url=getStatus(url)doSomethingWithResult(status,url)q.task_done()defgetStatus(ourl):try:res=requests.get(ourl)returnres.status_code,ourlexcept:return"error",ourldefdoSomethingWithResult(status,url):print(status,url)q=Queue(并发*2)foriinrange(concurrent):t=Thread(target=doWork)t.daemon=Truet.start()try:forurlinopen("urllist.txt"):q.put(url.strip())q.join()exceptKeyboardInterrupt:sys.exit(1)运行结果如下：有没有得到什么新技能？线程池如果使用线程池，建议使用更高级的concurrent.futures库：importconcurrent.futuresimportrequestsout=[]CONNECTIONS=100TIMEOUT=5urls=[]withopen("urllist.txt")asreader：对于阅读器中的url：urls.append(url.strip())defload_url(url,timeout):ans=requests.get(url,timeout=timeout)returnans.status_codewithconcurrent.futures.ThreadPoolExecutor(max_workers=CONNECTIONS)作为执行人：future_to_url=(executor.submit(load_url,url,TIMEOUT)forurlinurls)forfutureinconcurrent.futures.as_completed(future_to_url):try:data=future.result()exceptExceptionasexc:data=str(type(exc))finally:out.append(data)print(data)协程+aiohttp协程也是并发非常常用的工具了，importasynciofromaiohttpimportClientSession,ClientConnectorErrorasyncdeffetch_html(url:str,session:ClientSession,**kwargs)->tuple:try:resp=awaitsession.request(method="GET",url=url,**kwargs)除了ClientConnectorError:return(url,404)return(url,resp.status)asyncdefmake_requests(urls:set,**kwargs)->None:asyncwithClientSession()assession:tasks=[]forurlinurls:tasks.append(fetch_html(url=url,session=session,**kwargs))结果=等待asyncio.gather(*tasks)for结果结果：print(f'{result[1]}-{str(result[0])}')if__name__=="__main__":importsysassertsys.version_info>=(3,7),"ScriptrequiresPython3.7+."withopen("urllist.txt")asinfile:urls=set(map(str.strip,infile))asyncio.run(make_requests(urls=urls))grequests[1]这个是第三方库，目前有3.8Kstar，就是Requests+Gevent[2]、让异步http请求更简单Gevent的本质是协程。使用前：pipinstallgrequests使用起来相当简单：importgrequestsurls=[]withopen("urllist.txt")asreader:forurlinreader:urls.append(url.strip())rs=(grequests.get(u)foruinurls)forresultingrequests.map(rs):print(result.status_code,result.url)请注意，grequests.map(rs)是同时执行的。运行结果如下：也可以加入异常处理：>>>defexception_handler(request,exception):...print("Requestfailed")>>>reqs=[...grequests.get('http//httpbin.org/delay/1',timeout=0.001),...grequests.get('http://fakedomain/'),...grequests.get('http://httpbin.org/status/500')]>>>grequests.map(reqs,exception_handler=exception_handler)RequestfailedRequestfailed[None,None,
