python爬虫：案例二:携程网酒店价格信息

这个案例可能不太智能，有个朋友和我说他们公司让他爬携程的酒店价格信息，我当时去看了一下，发现携程的信息爬起来挺麻烦，城市是必输项，酒店名是选输项，跳转的url中城市后面带一个数字，对于这个每个城市表示的数字的规则我不知道，这样我只能定向爬一个城市，或者就是模拟浏览器之类，觉得挺麻烦，到了酒店页面又有挺多东西看着头疼，我对他说这个挺麻烦的，分析花的时间会很久，后来他说他们公司是人工输入酒店价格详情的url到数据库，然后直接从一个页面获取价格数据

#coding=utf-8
import sys
reload(sys)
sys.setdefaultencoding( "utf-8" )
import urllib
from selenium import webdriver

urls=['http://hotels.ctrip.com/hotel/848702.html#ctm_ref=hod_sr_lst_dl_n_2_1']
＃假设一堆url
class Xc():
    def pc(seif):
        for url in urls:
            driver = webdriver.PhantomJS()
            driver.get(url)
            fangx_1=driver.find_element_by_class_name('room_unfold').text.split('\n')[0]
            jiage_1=driver.find_element_by_class_name('base_price').text
            driver.quit
            return fangx_1+'|'+jiage_1
            ＃房型和对应的价格

s=Xc()
print s.pc()

结果：

单人房（无窗）｜￥237

上面的代码只是简单的例子，而且所有房型价额需要一个一个解析，太麻烦了，后来我发现源码最下面居然有一段json，里面的内容就是房型，价格这些，于是我改了一下代码

#coding=utf-8
import sys
reload(sys)
sys.setdefaultencoding( "utf-8" )
import urllib
from selenium import webdriver

urls=['http://hotels.ctrip.com/hotel/848702.html#ctm_ref=hod_sr_lst_dl_n_2_1']

class Xc():
    def pc(seif):
        for url in urls:
            driver = webdriver.PhantomJS()
            driver.get(url)
            #fangx_1=driver.find_element_by_class_name('room_unfold').text.split('\n')[0]
            #jiage_1=driver.find_element_by_class_name('base_price').text
            json=driver.find_element_by_xpath('//*[@id="htl_detail_htl_hotel"]').get_attribute('value')
            driver.quit
            #return fangx_1+'|'+jiage_1
            return json

s=Xc()
print s.pc()

结果：

pageid=102003;ht=848702;checkin=2016-05-09;checkout=2016-05-10;rmlist=[{"rm":"30665921","shadowid":"0","rpfq":"0.0","rpfh":"219","pt":"FG","mt":"0.0","pn":"0.0","promotiontype":"0","iscomfirm":"F","bedtype":"大床","breakfast":"0","policy":"免费取消","guaranteetype":"F","bk":"T","isgift":"F","isgroup":"F"},{"rm":"30265080","shadowid":"0","rpfq":"0.0","rpfh":"263","pt":"FG","mt":"0.0","pn":"0.0","promotiontype":"0","iscomfirm":"F","bedtype":"大床","breakfast":"0","policy":"不可取消","guaranteetype":"T","bk":"T","isgift":"F","isgroup":"F"},{"rm":"24125027","shadowid":"0","rpfq":"0.0","rpfh":"294","pt":"FG","mt":"0.0","pn":"0.0","promotiontype":"0","iscomfirm":"F","bedtype":"大床","breakfast":"0","policy":"免费取消","guaranteetype":"F","bk":"T","isgift":"F","isgroup":"F"},{"rm":"8684722","shadowid":"0","rpfq":"0.0","rpfh":"294","pt":"FG","mt":"0.0","pn":"0.0","promotiontype":"0","iscomfirm":"F","bedtype":"大床","breakfast":"0","policy":"不可取消","guaranteetype":"T","bk":"F","isgift":"F","isgroup":"F"},{"rm":"30265081","shadowid":"0","rpfq":"0.0","rpfh":"219","pt":"FG","mt":"0.0","pn":"0.0","promotiontype":"0","iscomfirm":"F","bedtype":"双床","breakfast":"0","policy":"不可取消","guaranteetype":"T","bk":"T","isgift":"F","isgroup":"F"},{"rm":"8684723","shadowid":"0","rpfq":"0.0","rpfh":"294","pt":"FG","mt":"0.0","pn":"0.0","promotiontype":"0","iscomfirm":"F","bedtype":"双床","breakfast":"0","policy":"不可取消","guaranteetype":"T","bk":"F","isgift":"F","isgroup":"F"},{"rm":"30265075","shadowid":"0","rpfq":"0.0","rpfh":"237","pt":"FG","mt":"0.0","pn":"0.0","promotiontype":"0","iscomfirm":"F","bedtype":"单人床","breakfast":"0","policy":"不可取消","guaranteetype":"T","bk":"T","isgift":"F","isgroup":"F"},{"rm":"24125024","shadowid":"0","rpfq":"0.0","rpfh":"265","pt":"FG","mt":"0.0","pn":"0.0","promotiontype":"0","iscomfirm":"F","bedtype":"单人床","breakfast":"0","policy":"免费取消","guaranteetype":"F","bk":"T","isgift":"F","isgroup":"F"},{"rm":"2890470","shadowid":"0","rpfq":"0.0","rpfh":"265","pt":"FG","mt":"0.0","pn":"0.0","promotiontype":"0","iscomfirm":"F","bedtype":"单人床","breakfast":"0","policy":"不可取消","guaranteetype":"T","bk":"F","isgift":"F","isgroup":"F"},{"rm":"30265074","shadowid":"0","rpfq":"0.0","rpfh":"254","pt":"FG","mt":"0.0","pn":"0.0","promotiontype":"0","iscomfirm":"F","bedtype":"大床","breakfast":"0","policy":"不可取消","guaranteetype":"T","bk":"T","isgift":"F","isgroup":"F"},{"rm":"24125041","shadowid":"0","rpfq":"0.0","rpfh":"284","pt":"FG","mt":"0.0","pn":"0.0","promotiontype":"0","iscomfirm":"F","bedtype":"大床","breakfast":"0","policy":"免费取消","guaranteetype":"F","bk":"T","isgift":"F","isgroup":"F"},{"rm":"2890480","shadowid":"0","rpfq":"0.0","rpfh":"284","pt":"FG","mt":"0.0","pn":"0.0","promotiontype":"0","iscomfirm":"F","bedtype":"大床","breakfast":"0","policy":"不可取消","guaranteetype":"T","bk":"F","isgift":"F","isgroup":"F"},{"rm":"30265072","shadowid":"0","rpfq":"0.0","rpfh":"280","pt":"FG","mt":"0.0","pn":"0.0","promotiontype":"0","iscomfirm":"F","bedtype":"双床","breakfast":"0","policy":"不可取消","guaranteetype":"T","bk":"T","isgift":"F","isgroup":"F"},{"rm":"24125016","shadowid":"0","rpfq":"0.0","rpfh":"313","pt":"FG","mt":"0.0","pn":"0.0","promotiontype":"0","iscomfirm":"F","bedtype":"双床","breakfast":"0","policy":"免费取消","guaranteetype":"F","bk":"T","isgift":"F","isgroup":"F"},{"rm":"2525661","shadowid":"0","rpfq":"0.0","rpfh":"313","pt":"FG","mt":"0.0","pn":"0.0","promotiontype":"0","iscomfirm":"F","bedtype":"双床","breakfast":"0","policy":"不可取消","guaranteetype":"T","bk":"F","isgift":"F","isgroup":"F"},{"rm":"30265079","shadowid":"0","rpfq":"0.0","rpfh":"305","pt":"FG","mt":"0.0","pn":"0.0","promotiontype":"0","iscomfirm":"F","bedtype":"大床","breakfast":"0","policy":"不可取消","guaranteetype":"T","bk":"T","isgift":"F","isgroup":"F"},{"rm":"2525665","shadowid":"0","rpfq":"0.0","rpfh":"341","pt":"FG","mt":"0.0","pn":"0.0","promotiontype":"0","iscomfirm":"F","bedtype":"大床","breakfast":"0","policy":"不可取消","guaranteetype":"T","bk":"F","isgift":"F","isgroup":"F"},{"rm":"30265077","shadowid":"0","rpfq":"0.0","rpfh":"305","pt":"FG","mt":"0.0","pn":"0.0","promotiontype":"0","iscomfirm":"F","bedtype":"双床","breakfast":"0","policy":"不可取消","guaranteetype":"T","bk":"T","isgift":"F","isgroup":"F"},{"rm":"24125021","shadowid":"0","rpfq":"0.0","rpfh":"341","pt":"FG","mt":"0.0","pn":"0.0","promotiontype":"0","iscomfirm":"F","bedtype":"双床","breakfast":"0","policy":"免费取消","guaranteetype":"F","bk":"T","isgift":"F","isgroup":"F"},{"rm":"8684720","shadowid":"0","rpfq":"0.0","rpfh":"341","pt":"FG","mt":"0.0","pn":"0.0","promotiontype":"0","iscomfirm":"F","bedtype":"双床","breakfast":"0","policy":"不可取消","guaranteetype":"T","bk":"T","isgift":"F","isgroup":"F"}]

rpfh是价格

bedtype是房型

时间： 2024-10-05 05:39:24

python爬虫：案例二:携程网酒店价格信息的相关文章

Python爬虫进阶二之PySpider框架安装配置

关于 首先,在此附上项目的地址,以及官方文档 PySpider 官方文档 安装 1. pip 首先确保你已经安装了pip,若没有安装,请参照 pip安装 2. phantomjs PhantomJS 是一个基于 WebKit 的服务器端 JavaScript API.它全面支持web而不需浏览器支持,其快速.原生支持各种Web标准:DOM 处理.CSS 选择器.JSON.Canvas 和 SVG. PhantomJS 可以用于页面自动化.网络监测.网页截屏以及无界面测试等. 安装 以上附有官方安

Python爬虫实战二之爬取百度贴吧帖子

大家好,上次我们实验了爬取了糗事百科的段子,那么这次我们来尝试一下爬取百度贴吧的帖子.与上一篇不同的是,这次我们需要用到文件的相关操作. 前言 亲爱的们,教程比较旧了,百度贴吧页面可能改版,可能代码不好使,八成是正则表达式那儿匹配不到了,请更改一下正则,当然最主要的还是帮助大家理解思路. 2016/12/2 本篇目标 1.对百度贴吧的任意帖子进行抓取 2.指定是否只抓取楼主发帖内容 3.将抓取到的内容分析并保存到文件 1.URL格式的确定 首先,我们先观察一下百度贴吧的任意一个帖子. 比如:ht

Python爬虫利器二之Beautiful Soup的用法

上一节我们介绍了正则表达式,它的内容其实还是蛮多的,如果一个正则匹配稍有差池,那可能程序就处在永久的循环之中,而且有的小伙伴们也对写正则表达式的写法用得不熟练,没关系,我们还有一个更强大的工具,叫Beautiful Soup,有了它我们可以很方便地提取出HTML或XML标签中的内容,实在是方便,这一节就让我们一起来感受一下Beautiful Soup的强大吧. 1. Beautiful Soup的简介 简单来说,Beautiful Soup是python的一个库,最主要的功能是从网页抓取数据.官

转 Python爬虫入门二之爬虫基础了解

静觅 » Python爬虫入门二之爬虫基础了解 2.浏览网页的过程 在用户浏览网页的过程中,我们可能会看到许多好看的图片,比如 http://image.baidu.com/ ,我们会看到几张的图片以及百度搜索框,这个过程其实就是用户输入网址之后,经过DNS服务器,找到服务器主机,向服务器发出一个请求,服务器经过解析之后,发送给用户的浏览器 HTML.JS.CSS 等文件,浏览器解析出来,用户便可以看到形形色色的图片了. 因此,用户看到的网页实质是由 HTML 代码构成的,爬虫爬来的便是这些内容

转 Python爬虫实战二之爬取百度贴吧帖子

静觅 » Python爬虫实战二之爬取百度贴吧帖子 大家好,上次我们实验了爬取了糗事百科的段子,那么这次我们来尝试一下爬取百度贴吧的帖子.与上一篇不同的是,这次我们需要用到文件的相关操作. 本篇目标 1.对百度贴吧的任意帖子进行抓取 2.指定是否只抓取楼主发帖内容 3.将抓取到的内容分析并保存到文件

python爬虫（二）：向网页提交数据

python爬虫(二):向网页提交数据 回忆一下,我们有的时候在看一些网站的时候,是否遇见过一些网站里面的信息开始显示一部分,然后当我们把鼠标滑轮向下拉动后,又显示出一些信息.这就是异步加载.我的上一篇文章python爬虫百度贴吧标题数据爬取的所有标题都是页面已经加载好的.但是对于这种开始没有加载好的数据我们应该如何爬取呢? 接下来我们先介绍下一些概念: 异步加载:举个简单的例子就是说,假如老师判作业,有两种情况,第一种就是无论哪个学生先写完,都等待到所有的同学全部写完,老师讲所有的作业都收齐后

Python 爬虫入门(二)——爬取妹子图

Python 爬虫入门 听说你写代码没动力?本文就给你动力,爬取妹子图.如果这也没动力那就没救了. GitHub 地址: https://github.com/injetlee/Python/blob/master/%E7%88%AC%E8%99%AB%E9%9B%86%E5%90%88/meizitu.py 公众号:[智能制造专栏].欢迎关注,分享智能制造与编程那些事. 爬虫成果 当你运行代码后,文件夹就会越来越多,如果爬完的话会有2000多个文件夹,20000多张图片.不过会很耗时间,可以在

python 爬虫（二）

python 爬虫 Advanced HTML Parsing 1. 通过属性查找标签:基本上在每一个网站上都有stylesheets,针对于不同的标签会有不同的css类于之向对应在我们看到的标签可能是像下面这样的 <span class="green" ></span> <span class="red"></span> 通过标签的class我们的程序能够简单的将他们分辨开来. 1 from urllib.reque

python爬虫（二）

python爬虫之urllib 在python2和python3中的差异 在python2中,urllib和urllib2各有各个的功能,虽然urllib2是urllib的升级版,但是urllib2还是不能完全替代urllib,但是在python3中,全部封装成一个类urllib. Urllib2可以接受一个Request对象,并以此可以来设置一个URL的headers,但是urllib只接受一个URL.这就意味着你不能通过urllib伪装自己的请求头.Urllib模板可以提供运行urlenco