Scrapy爬取伯乐在线文章

首先搭建虚拟环境,创建工程

scrapy startproject ArticleSpider
cd ArticleSpider
scrapy genspider jobbole blog.jobbole.com

修改start_urls = [‘http://blog.jobbole.com/all-posts/‘]

获取网页信息

ArticleSpider/spiders/jobbole.py

# -*- coding: utf-8 -*-
import datetime
import re
from urllib import parse
import scrapy
from scrapy import Request

from ArticleSpider.items import JobBoleArticleItem
from ArticleSpider.utils.common import get_md5

class JobboleSpider(scrapy.Spider):
    name = ‘jobbole‘
    allowed_domains = [‘blog.jobbole.com‘]
    start_urls = [‘http://blog.jobbole.com/all-posts/‘]

    def parse(self, response):
        """
        1. 从文章列表中获取文章链接交给scrapy下载 再进行解析
        2. 获取下一页的链接并交给scrapy下载, 下载完成后在使用parse函数进行解析
        :param response:
        :return:
        """
        post_nodes = response.css("#archive .floated-thumb .post-thumb a")
        for post_node in post_nodes:
            image_url = post_node.css("img::attr(src)").extract_first("")
            post_url = post_node.css("::attr(href)").extract_first("")
            yield Request(url=parse.urljoin(response.url, post_url), meta={"front_image_url": image_url},
                          callback=self.parse_detail)

        # 提取下一页
        next_url = response.css(".next.page-numbers::attr(href)").extract_first()
        if next_url:
            yield Request(url=parse.urljoin(response.url, next_url), callback=self.parse)

    def parse_detail(self, response):
        """
        获取具体字段
        :param response:
        :return:
        """
        article_item = JobBoleArticleItem()

        # 通过css选择器提取字段
        front_image_url = response.meta.get("front_image_url", "")  #文章封面图
        title = response.css(".entry-header h1::text").extract()[0]
        create_date = response.css("p.entry-meta-hide-on-mobile::text").extract()[0].strip().replace("·","").strip()
        praise_nums = response.css(".vote-post-up h10::text").extract()[0]
        fav_nums = response.css(".bookmark-btn::text").extract()[0]
        match_re = re.match(".*?(\d+).*", fav_nums)
        if match_re:
            fav_nums = int(match_re.group(1))
        else:
            fav_nums = 0

        comment_nums = response.css("a[href=‘#article-comment‘] span::text").extract()[0]
        match_re = re.match(".*?(\d+).*", comment_nums)
        if match_re:
            comment_nums = int(match_re.group(1))
        else:
            comment_nums = 0

        content = response.css("div.entry").extract()[0]

        tag_list = response.css("p.entry-meta-hide-on-mobile a::text").extract()
        tag_list = [element for element in tag_list if not element.strip().endswith("评论")]
        tags = ",".join(tag_list)

        article_item["url_object_id"] = get_md5(response.url)
        article_item["title"] = title
        article_item["url"] = response.url
        try:
            create_date = datetime.datetime.strptime(create_date, "%Y/%m/%d").date()
        except Exception as e:
            create_date = datetime.datetime.now().date()
        article_item["create_date"] = create_date
        article_item["front_image_url"] = [front_image_url]
        article_item["praise_nums"] = praise_nums
        article_item["comment_nums"] = comment_nums
        article_item["fav_nums"] = fav_nums
        article_item["tags"] = tags
        article_item["content"] = content
        yield article_item

ArticleSpider/items.py

class JobBoleArticleItem(scrapy.Item):
    title = scrapy.Field()
    create_date = scrapy.Field()
    url = scrapy.Field()
    url_object_id = scrapy.Field()
    front_image_url = scrapy.Field()
    front_image_path = scrapy.Field()
    praise_nums = scrapy.Field()
    comment_nums = scrapy.Field()
    fav_nums = scrapy.Field()
    tags = scrapy.Field()
    content = scrapy.Field()

ArticleSpider/pipelines.py

from scrapy.pipelines.images import ImagesPipeline

class ArticlespiderPipeline(object):
    def process_item(self, item, spider):
        return item

class ArticleImagePipeline(ImagesPipeline):
    def item_completed(self, results, item, info):
        if "front_image_url" in item:
            for ok, value in results:
                image_file_path = value["path"]
            item["front_image_path"] = image_file_path

        return item

ArticleSpider/settings.py

创建一个文件夹ArticleSpider/images,用来保存图片

# Configure item pipelines
# See https://doc.scrapy.org/en/latest/topics/item-pipeline.html
ITEM_PIPELINES = {
    ‘ArticleSpider.pipelines.ArticlespiderPipeline‘: 300,
    # ‘scrapy.pipelines.images.ImagesPipeline‘: 1,
    ‘ArticleSpider.pipelines.ArticleImagePipeline‘: 1,
}

IMAGES_URLS_FIELD = "front_image_url"
project_dir = os.path.abspath(os.path.dirname(__file__))
IMAGES_STORE = os.path.join(project_dir, ‘images‘)

数据入库

/*
Navicat MySQL Data Transfer

Source Server         : 2233
Source Server Version : 50723
Source Host           : localhost:3306
Source Database       : article_spider

Target Server Type    : MYSQL
Target Server Version : 50723
File Encoding         : 65001

Date: 2018-10-15 11:19:07
*/

SET FOREIGN_KEY_CHECKS=0;

-- ----------------------------
-- Table structure for jobbole_article
-- ----------------------------
DROP TABLE IF EXISTS `jobbole_article`;
CREATE TABLE `jobbole_article` (
  `title` varchar(255) NOT NULL,
  `create_date` date DEFAULT NULL,
  `url` varchar(255) NOT NULL,
  `url_object_id` varchar(50) NOT NULL,
  `front_image_url` varchar(255) DEFAULT NULL,
  `front_image_path` varchar(255) DEFAULT NULL,
  `praise_nums` int(11) NOT NULL DEFAULT ‘0‘,
  `comment_nums` int(11) NOT NULL DEFAULT ‘0‘,
  `fav_nums` int(11) NOT NULL DEFAULT ‘0‘,
  `tags` varchar(255) DEFAULT NULL,
  `content` longtext NOT NULL,
  PRIMARY KEY (`url_object_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

编写pipline

import MySQLdb
import MySQLdb.cursors
from scrapy.pipelines.images import ImagesPipeline
from twisted.enterprise import adbapi

from ArticleSpider import settings
from ArticleSpider.settings import MYSQL_DBNAME, MYSQL_PASSWORD, MYSQL_USER, MYSQL_HOST

class ArticlespiderPipeline(object):
    def process_item(self, item, spider):
        return item

class ArticleImagePipeline(ImagesPipeline):
    def item_completed(self, results, item, info):
        if "front_image_url" in item:
            for ok, value in results:
                image_file_path = value["path"]
            item["front_image_path"] = image_file_path

        return item

class MysqlPipeline(object):
    # 采用同步的机制写入mysql
    def __init__(self):
        self.conn = MySQLdb.connect(MYSQL_HOST, MYSQL_USER, MYSQL_PASSWORD, MYSQL_DBNAME, charset="utf8",
                                    use_unicode=True)
        self.cursor = self.conn.cursor()

    def process_item(self, item, spider):
        insert_sql = """
            insert into jobbole_article(title, url,url_object_id, create_date, fav_nums)
            VALUES (%s, %s, %s, %s, %s)
        """
        self.cursor.execute(insert_sql,
                            (item["title"], item["url"], item["url_object_id"], item["create_date"], item["fav_nums"]))
        self.conn.commit()

class MysqlTwistedPipline(object):
    def __init__(self, dbpool):
        self.dbpool = dbpool

    @classmethod
    def from_settings(cls, settings):
        dbparms = dict(
            host=settings["MYSQL_HOST"],
            db=settings["MYSQL_DBNAME"],
            user=settings["MYSQL_USER"],
            passwd=settings["MYSQL_PASSWORD"],
            charset=‘utf8‘,
            cursorclass=MySQLdb.cursors.DictCursor,
            use_unicode=True,
        )
        dbpool = adbapi.ConnectionPool("MySQLdb", **dbparms)

        return cls(dbpool)

    def process_item(self, item, spider):
        # 使用twisted将mysql插入变成异步执行
        query = self.dbpool.runInteraction(self.do_insert, item)
        query.addErrback(self.handle_error, item, spider)  # 处理异常

    def handle_error(self, failure, item, spider):
        # 处理异步插入的异常
        print(failure)

    def do_insert(self, cursor, item):
        # 执行具体的插入
        # 根据不同的item 构建不同的sql语句并插入到mysql中
        insert_sql = """
                    insert into jobbole_article(title, url,url_object_id, create_date, front_image_url, front_image_path, praise_nums, comment_nums,fav_nums, tags,content )
                    VALUES (%s, %s, %s, %s,%s,%s, %s, %s, %s,%s,%s)
                """
        cursor.execute(insert_sql,
                       (item["title"], item["url"], item["url_object_id"], item["create_date"], item["front_image_url"],
                        item["front_image_path"], item["praise_nums"], item["comment_nums"], item["fav_nums"], item["tags"],
                        item["content"]))

settings.py

ITEM_PIPELINES = {
    # ‘ArticleSpider.pipelines.ArticlespiderPipeline‘: 300,
    # ‘scrapy.pipelines.images.ImagesPipeline‘: 1,
    ‘ArticleSpider.pipelines.ArticleImagePipeline‘: 1,
    ‘ArticleSpider.pipelines.MysqlTwistedPipline‘: 2
}

item loader

ArticleSpider/spiders/jobbole.py

        # 通过item loader加载item
        front_image_url = response.meta.get("front_image_url", "")  # 文章封面图
        item_loader = ArticleItemLoader(item=JobBoleArticleItem(), response=response)
        item_loader.add_css("title", ".entry-header h1::text")
        item_loader.add_value("url", response.url)
        item_loader.add_value("url_object_id", get_md5(response.url))
        item_loader.add_css("create_date", "p.entry-meta-hide-on-mobile::text")
        item_loader.add_value("front_image_url", [front_image_url])
        item_loader.add_css("praise_nums", ".vote-post-up h10::text")
        item_loader.add_css("comment_nums", "a[href=‘#article-comment‘] span::text")
        item_loader.add_css("fav_nums", ".bookmark-btn::text")
        item_loader.add_css("tags", "p.entry-meta-hide-on-mobile a::text")
        item_loader.add_css("content", "div.entry")

        article_item = item_loader.load_item()

        yield article_item

ArticleSpider/items.py

```python

class ArticleItemLoader(ItemLoader):

# 自定义itemloader

default_output_processor = TakeFirst()

def date_convert(value):

try:

create_date = datetime.datetime.strptime(value, "%Y/%m/%d").date()

except Exception as e:

create_date = datetime.datetime.now().date()

return create_date

def get_nums(value):

match_re = re.match(".?(\d+).", value)

if match_re:

nums = int(match_re.group(1))

else:

nums = 0

return nums

def remove_comment_tags(value):

# 去掉tag中提取的评论

if "评论" in value:

return ""

else:

return value

def return_value(value):

return value

class JobBoleArticleItem(scrapy.Item):

title = scrapy.Field()

create_date = scrapy.Field(

input_processor=MapCompose(date_convert),

)

url = scrapy.Field()

url_object_id = scrapy.Field()

front_image_url = scrapy.Field(

output_processor=MapCompose(return_value)

)

front_image_path = scrapy.Field()

praise_nums = scrapy.Field(

input_processor=MapCompose(get_nums)

)

comment_nums = scrapy.Field(

input_processor=MapCompose(get_nums)

)

fav_nums = scrapy.Field(

input_processor=MapCompose(get_nums)

)

tags = scrapy.Field(

input_processor=MapCompose(remove_comment_tags),

output_processor=Join(",")

)

content = scrapy.Field()

```

原文地址：https://www.cnblogs.com/gaoyongjian/p/9905371.html

时间： 2024-10-16 19:29:42

Scrapy爬取伯乐在线文章的相关文章

Scrapy分布式爬虫打造搜索引擎——（二） scrapy 爬取伯乐在线

1.开发环境准备 1.爬取策略 目标:爬取“伯乐在线”的所有文章 策略选择:由于“伯乐在线”提供了全部文章的索引页 ,所有不需要考虑url的去重方法,直接在索引页开始,一篇文章一篇文章地进行爬取,一直进行到最后一页即可. 索引页地址:http://blog.jobbole.com/all-posts/ 2. 搭建python3虚拟环境 打开cmd,进入命令行,输入workon,查看当前存在的虚拟环境:  workon 为爬虫项目,新建python3虚拟环境: mkvirtualenv -p py

Python爬虫-爬取伯乐在线美女邮箱

爬取伯乐在线美女邮箱 1.登录界面的进入,设置url,cookie,data,headers 2.进入主页,点击邮箱链接,需要重新设置url,cookie(读取重新保存的cookie),data,headers 1 ''' 2 爬取伯乐在线的美女联系方式 3 需要: 4 1. 登录 5 2. 在登录和相应声望值的前提下,提取对方的邮箱 6 ''' 7 8 from urllib import request, error, parse 9 from http import cookiejar 1

scrapy爬取中关村在线手机频道

1 # -*- coding: utf-8 -*- 2 import scrapy 3 from pyquery import PyQuery as pq 4 5 from zolphone.items import ZolphoneItem 6 7 8 class PhoneSpider(scrapy.Spider): 9 name = "phone" 10 # allowed_domains = ["www.zol.com.cn"] 11 # start_url

python抓取伯乐在线的所有文章，对标题分词后存入mongodb中

依赖包: 1.pymongo 2.jieba # -*- coding: utf-8 -*- """ @author: jiangfuqiang """ from HTMLParser import HTMLParser import urllib2 import sys import pymongo import time import jieba import traceback default_encoding = 'utf-8' if s

Scrapy爬取美女图片 (原创)

有半个月没有更新了,最近确实有点忙.先是华为的比赛,接着实验室又有项目,然后又学习了一些新的知识,所以没有更新文章.为了表达我的歉意,我给大家来一波福利... 今天咱们说的是爬虫框架.之前我使用python爬取慕课网的视频,是根据爬虫的机制,自己手工定制的,感觉没有那么高大上,所以我最近玩了玩 python中强大的爬虫框架Scrapy. Scrapy是一个用 Python 写的 Crawler Framework ,简单轻巧,并且非常方便.Scrapy 使用 Twisted 这个异步网络库来处理

Scrapy爬取美女图片续集 (原创)

上一篇咱们讲解了Scrapy的工作机制和如何使用Scrapy爬取美女图片,而今天接着讲解Scrapy爬取美女图片,不过采取了不同的方式和代码实现,对Scrapy的功能进行更深入的运用. 在学习Scrapy官方文档的过程中,发现Scrapy自身实现了图片和文件的下载功能,不需要咱们之前自己实现图片的下载(不过原理都一样). 在官方文档中,我们可以看到下面一些话:Scrapy为下载item中包含的文件(比如在爬取到产品时,同时也想保存对应的图片)提供了一个可重用的 item pipelines .

scrapy爬取网易新闻内容

最近在写毕业论文,是做个文本分类相关的题目.想抓取网易新闻里那些新闻内容作为分析的数据,于是就又照着scrapy的文档照做了一遍... 感觉主要就只是两个文件items.py和spiders文件夹下的爬取规则的文件,我这里爬取技术类文档的直接叫tech.py了 scrapy的入门教程已经写的很详细了,大家可以先看看.地址为http://scrapy-chs.readthedocs.org/zh_CN/0.24/intro/tutorial.html 下面是我做的情况 1.创建项目    运行命令

Scrapy爬取美女图片第三集 代理ip(上) (原创)

首先说一声,让大家久等了.本来打算520那天进行更新的,可是一细想,也只有我这样的单身狗还在做科研,大家可能没心思看更新的文章,所以就拖到了今天.不过忙了521,522这一天半,我把数据库也添加进来了,修复了一些bug(现在肯定有人会说果然是单身狗). 好了,废话不多说,咱们进入今天的主题.上两篇 Scrapy爬取美女图片 的文章,咱们讲解了scrapy的用法.可是就在最近,有热心的朋友对我说之前的程序无法爬取到图片,我猜应该是煎蛋网加入了反爬虫机制.所以今天讲解的就是突破反爬虫机制的上篇 代理

【转载】教你分分钟学会用python爬虫框架Scrapy爬取心目中的女神

原文:教你分分钟学会用python爬虫框架Scrapy爬取心目中的女神 本博文将带领你从入门到精通爬虫框架Scrapy,最终具备爬取任何网页的数据的能力.本文以校花网为例进行爬取,校花网:http://www.xiaohuar.com/,让你体验爬取校花的成就感. Scrapy,Python开发的一个快速,高层次的屏幕抓取和web抓取框架,用于抓取web站点并从页面中提取结构化的数据.Scrapy用途广泛,可以用于数据挖掘.监测和自动化测试. Scrapy吸引人的地方在于它是一个框架,任何人都可