前提:操作平台-WIN7
一.首先你得python,我安装的是python2.7.9
二.其次,你得安装pyLibCurl,安装方式:http://pycurl.sourceforge.net/
三.最后,你得编写测试用例test.py:(当然,从代码中可以看出你电脑得有E盘,否则改一下代码,然后我抓取的数据是google一下test的数据)
#! /usr/bin/env python # -*- coding: utf-8 -*- # vi:ts=4:et import sys import pycurl class Test: def __init__(self): self.contents = '' def body_callback(self, buf): self.contents = self.contents + buf f = open('e:\\tmp\\tmp.html', 'w') print f f.write(self.contents) sys.stderr.write("Testing %s\n" % pycurl.version) t = Test() c = pycurl.Curl() c.setopt(c.URL, 'https://www.google.com/search?q=testx') c.setopt(c.USERAGENT, "Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0) in my heart of heart.") c.setopt(c.HEADER, True) c.setopt(c.REFERER, "https://www.google.com/search?q=testx") c.setopt(c.COOKIEFILE, "./COOKIE.txt") c.setopt(c.COOKIEJAR, "./COOKIE.txt") c.setopt(c.WRITEFUNCTION, t.body_callback) c.perform() c.close() print(t.contents)
延伸:
是不是你正常发起http请求google会告诉你“302 Moved”,ok,仔细研究一下这段代码,也会解决你的问题
参考:
http://superuser.com/questions/482470/google-302-moved-in-firefox
http://stackoverflow.com/questions/22570970/php-search-by-image-google-curl-return-302-moved
时间: 2024-10-05 22:58:50