python3.6.5 + pycharm
注意:
一、python3里的 urllib2 已经没有了,改为了 urllbi.request,因此,直接导入 import urllib.request 即可。
二、必须对正则表达式里的引用变量进行格式变换 .decode(‘utf-8‘),否则会报错说 不能在一个字节类的对象上使用字符串格式。
如下代码所示。
### 一、 网站地图爬虫,控制用户代理的设定,可捕获异常、重试下载并设置用户代理。wswp: web scraping with python import urllib.request ## -- written by LiSongbo def Rocky_dnload(url,user_agent=‘wswp‘,num_retries = 2): print(‘Downloading:‘,url) LiSongbo_he={‘User-agent‘:user_agent} request = urllib.request.Request(url, headers=LiSongbo_he) try: ## -- written by LiSongbo html = urllib.request.urlopen(request).read() except urllib.request.URLError as e: ## -- written by LiSongbo print(‘Download error:‘,e.reason) html = None if num_retries > 0: ## -- written by LiSongbo if hasattr(e,‘code‘) and 500 <= e.code < 600: return Rocky_dnload(url,user_agent,num_retries-1) ## retry 5xx HTTP errors return html import re ## -- written by LiSongbo def Rocky_crawl_sitemap(url): ## -- written by LiSongbo sitemap = Rocky_dnload(url) ## download the sitmap file # sitemap = sitemap.decode(‘utf-8‘) ## must add this . links = re.findall(‘<loc>(.*?)</loc>‘, sitemap) ## extract the sitemap links from flag loc for link in links: ## download each link html = Rocky_dnload(link) ## crape html here Rocky_crawl_sitemap(‘http://example.webscraping.com/sitemap.xml‘)
运行结果报错:
Downloading: http://example.webscraping.com/sitemap.xml
Traceback (most recent call last):
File "C:/Users/klooa/my_env/book9/test.py", line 25, in <module>
Rocky_crawl_sitemap(‘http://example.webscraping.com/sitemap.xml‘)
File "C:/Users/klooa/my_env/book9/test.py", line 22, in Rocky_crawl_sitemap
links = re.findall(‘<loc>(.*?)</loc>‘, sitemap) ## extract the sitemap links from flag loc
File "C:\Users\klooa\AppData\Local\Programs\Python\Python36\lib\re.py", line 222, in findall
return _compile(pattern, flags).findall(string)
TypeError: cannot use a string pattern on a bytes-like object
必须在 sitemap的下一行加上
sitemap = sitemap.decode(‘utf-8‘)
修改后的运行结果为:
Downloading: http://example.webscraping.com/sitemap.xml
Downloading: http://example.webscraping.com/places/default/view/Afghanistan-1
Downloading: http://example.webscraping.com/places/default/view/Aland-Islands-2
Downloading: http://example.webscraping.com/places/default/view/Albania-3
Downloading: http://example.webscraping.com/places/default/view/Algeria-4
Downloading: http://example.webscraping.com/places/default/view/American-Samoa-5
Downloading: http://example.webscraping.com/places/default/view/Andorra-6
Downloading: http://example.webscraping.com/places/default/view/Angola-7
Downloading: http://example.webscraping.com/places/default/view/Anguilla-8
Downloading: http://example.webscraping.com/places/default/view/Antarctica-9
Downloading: http://example.webscraping.com/places/default/view/Antigua-and-Barbuda-10
Downloading: http://example.webscraping.com/places/default/view/Argentina-11
Downloading: http://example.webscraping.com/places/default/view/Armenia-12
Download error: TOO MANY REQUESTS
Downloading: http://example.webscraping.com/places/default/view/Aruba-13
Download error: TOO MANY REQUESTS
……
## -- written by LiSongbo
原文地址:https://www.cnblogs.com/LiSongbo/p/9245449.html