爬虫3 html解析器 html_parser.py

#coding:utf8
import urlparse
from bs4 import BeautifulSoup
import re

__author__ = ‘wang‘

class HtmlParser(object):
    def parse(self, page_url, html_cont):
        if page_url is None or html_cont is None:
            return

        soup = BeautifulSoup(html_cont, ‘html.parser‘, from_encoding = ‘utf-8‘)
        new_urls = self._get_new_urls(page_url, soup)
        new_data = self._get_new_data(page_url, soup)
        return new_urls, new_data;

    def _get_new_urls(self, page_url, soup):
        new_urls = set()
        links = soup.find_all(‘a‘, href=re.compile(r"/view/\d+\.htm"))

        for link in links:
            new_url = link[‘href‘]
            new_full_url = urlparse.urljoin(page_url, new_url);
            new_urls.add(new_full_url)

        return new_urls

    def _get_new_data(self, page_url, soup):
        res_data = {}
        res_data[‘url‘] = page_url
        title_node = soup.find(‘dd‘, class_=‘lemmaWgt-lemmaTitle-title‘).find("h1")
        res_data[‘title‘] = title_node.get_text()
        summary_node = soup.find(‘div‘, class_ = ‘lemma-summary‘)
        res_data[‘summary‘] = summary_node.get_text()

        return res_data

时间： 2024-10-11 19:41:02

爬虫3 html解析器 html_parser.py的相关文章

爬虫学习——网页解析器Beautiful Soup

一.Beautiful Soup的安装与测试官方网站:https://www.crummy.com/software/BeautifulSoup/ Beautiful Soup安装与使用文档: https://www.crummy.com/software/BeautifulSoup/bs4/doc/ 1.首先测试一下bs4模块是否已经存在,若不存在再安装即可,我用的是kali测试发现bs4模块已经存在,下面介绍如何测试与安装新建python文档输入以下代码 1 import bs4 2

爬虫5 html下载器 html_downloader.py

#coding:utf8 import urllib2 __author__ = 'wang' class HtmlDownloader(object): def download(self, url): if url is None: return None response = urllib2.urlopen(url) if response.getcode() != 200: return None return response.read()

爬虫2 url管理器 url_manager.py

#coding:utf8 class UrlManager(object): def __init__(self): self.new_urls = set() self.old_urls = set() def add_new_url(self, url): if url is None: return if url not in self.new_urls and url not in self.old_urls: self.new_urls.add(url) def add_new_url

爬虫4 html输出器 html_outputer.py

#coding:utf8 __author__ = 'wang' class HtmlOutputer(object): def __init__(self): self.datas = []; def collect_data(self, data): if data is None: return print data self.datas.append(data) def output_html(self): fout = open('output.html', 'w') fout.wri

python爬虫主要就是五个模块：爬虫启动入口模块，URL管理器存放已经爬虫的URL和待爬虫URL列表，html下载器，html解析器，html输出器同时可以掌握到urllib2的使用、bs4（BeautifulSoup）页面解析器、re正则表达式、urlparse、python基础知识回顾（set集合操作）等相关内容。

本次python爬虫百步百科,里面详细分析了爬虫的步骤,对每一步代码都有详细的注释说明,可通过本案例掌握python爬虫的特点: 1.爬虫调度入口(crawler_main.py) # coding:utf-8from com.wenhy.crawler_baidu_baike import url_manager, html_downloader, html_parser, html_outputer print "爬虫百度百科调度入口" # 创建爬虫类class SpiderMai

爬虫3 html解析器 html_parser.py