一、urllib2简单获取html页面
#!/usr/bin/env python # -*- coding:utf-8 -*- import urllib2 response = urllib2.urlopen(‘http://www.baidu.com‘); html = response.read(); print html
简单的几行代码就能拿到html页面，接下来局势html的解析工作了。
想象很美好，实际操作就出问题了。baidu没有禁止机器人抓取可以正常抓取到页面，但是比如：https://b.ishadow.tech/是禁止机器人抓取的，简单模拟浏览器头部信息也不行。
然后想找个GitHub上的爬虫来试验一下行不行，因此找到了https://github.com/scrapy/scrapy，看样子好像比较叼。
按照readme安装了一下，安装失败了，仔细看了一下文档。
pip install scrapy
官方建议安装在Python的虚拟环境里，描述大概如下（https://doc.scrapy.org/en/latest/intro/install.html#intro-using-virtualenv）：
Using a virtual environment (recommended) TL;DR: We recommend installing Scrapy inside a virtual environment on all platforms. Python packages can be installed either globally (a.k.a system wide), or in user-space. We do not recommend installing scrapy system wide. Instead, we recommend that you install scrapy within a so-called “virtual environment” (virtualenv). Virtualenvs allow you to not conflict with already-installed Python system packages (which could break some of your system tools and scripts), and still install packages normally with pip (without sudo and the likes).
然后决定安装一个Python虚拟环境，命令如下：
$ sudo pip install virtualenv
查看基本使用
$virtualenv -h Usage: virtualenv [OPTIONS] DEST_DIR
只需要 virtualenv加目标目录就可以了。
因此新建虚拟环境
$virtualevn e27 New python executable in ~/e27/bin/python Installing setuptools, pip, wheel...done.
启用环境
$source ./bin/activate
注意切换环境成功后当前目录会有标识，如下
? e27 source ./bin/activate (e27) ? e27
退出环境
$deactivate
现在开始干正事，在虚拟环境中安装爬虫（https://github.com/scrapy/scrapy）
pip install scrapy
大约三分钟后安装完成，之前直接在全局环境安装尽然还失败了。成功后shell输出如下：
......Successfully built lxml PyDispatcher Twisted pycparser Installing collected packages: lxml, PyDispatcher, zope.interface, constantly, incremental, attrs, Automat, Twisted, ipaddress, asn1crypto, enum34, idna, pycparser, cffi, cryptography, pyOpenSSL, queuelib, w3lib, cssselect, parsel, pyasn1, pyasn1-modules, service-identity, scrapy Successfully installed Automat-0.5.0 PyDispatcher-2.0.5 Twisted-17.1.0 asn1crypto-0.22.0 attrs-16.3.0 cffi-1.10.0 constantly-15.1.0 cryptography-1.8.1 cssselect-1.0.1 enum34-1.1.6 idna-2.5 incremental-16.10.1 ipaddress-1.0.18 lxml-3.7.3 parsel-1.1.0 pyOpenSSL-16.2.0 pyasn1-0.2.3 pyasn1-modules-0.0.8 pycparser-2.17 queuelib-1.4.2 scrapy-1.3.3 service-identity-16.0.0 w3lib-1.17.0 zope.interface-4.3.3
安装好scrapy后尝试一个简单的连接
(e27) ? e27 scrapy shell ‘http://quotes.toscrape.com/page/1/‘
得到一堆结果如下
2017-03-30 22:08:42 [scrapy.core.engine] INFO: Spider opened
2017-03-30 22:08:43 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/page/1/> (referer: None)
[s] Available Scrapy objects:
[s] scrapy scrapy module (contains scrapy.Request, scrapy.Selector, etc)
[s] crawler <scrapy.crawler.Crawler object at 0x1100bab50>
[s] item {}
[s] request <GET http://quotes.toscrape.com/page/1/>
[s] response <200 http://quotes.toscrape.com/page/1/>
[s] settings <scrapy.settings.Settings object at 0x1100baad0>
[s] spider <DefaultSpider ‘default‘ at 0x11037ebd0>
[s] Useful shortcuts:
[s] fetch(url[, redirect=True]) Fetch URL and update local objects (by default, redirects are followed)
[s] fetch(req) Fetch a scrapy.Request and update local objects
[s] shelp() Shell help (print this help)
[s] view(response) View response in a browser
证明是可以工作的，然后试一下连接：https://b.ishadow.tech/
(e27) ? e27 scrapy shell ‘https://b.ishadow.tech/‘
结果如下：
2017-03-30 22:10:21 [scrapy.middleware] INFO: Enabled item pipelines: [] 2017-03-30 22:10:21 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023 2017-03-30 22:10:21 [scrapy.core.engine] INFO: Spider opened 2017-03-30 22:11:36 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://b.ishadow.tech/> (failed 1 times): TCP connection timed out: 60: Operation timed out. 2017-03-30 22:12:51 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://b.ishadow.tech/> (failed 2 times): TCP connection timed out: 60: Operation timed out. 2017-03-30 22:14:07 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying <GET https://b.ishadow.tech/> (failed 3 times): TCP connection timed out: 60: Operation timed out. Traceback (most recent call last):
爬去超时了，看来是被识别出来是机器人爬取内容被拒绝的（当然此时站点通过浏览器是可以访问的），厉害了我的哥！到这里你是不是已经猜到我的真实目的了，没有的话请打开我爬取得连接看看就知道了。??
及时没有爬取成功有时间研究一下这个爬虫还是挺好的，文档连接：https://doc.scrapy.org/en/latest/intro/overview.html
接下来慢慢研究怎么突破封锁。
时间： 2024-12-21 09:05:24