python3.7 Scrapy在Windows平台的安装

1.安装python并配置好环境变量

2.更新pip ,这步操作时遇到了问题,更新超时了,所以加上--default-timeout来解决超时问题:python -m pip --default-timeout=100 install --upgrade pip

3.安装lxml :pip intsall lxml

4.安装wheel: pip install wheel

5.在python lib网站上以下安装包,然后使用pip install xxx 安装:

Twisted-18.9.0-cp37-cp37m-win_amd64.whl

pywin32-224-cp37-cp37m-win_amd64.whl

Scrapy-1.6.0-py2.py3-none-any.whl

Python Lib网站:https://www.lfd.uci.edu/~gohlke/pythonlibs/

6.Scrapy调试,验证是否安装成功:

C:\Users\admin>scrapy
Scrapy 1.6.0 - no active project

Usage:
scrapy <command> [options] [args]

Available commands:
bench Run quick benchmark test
fetch Fetch a URL using the Scrapy downloader
genspider Generate new spider using pre-defined templates
runspider Run a self-contained spider (without creating a project)
settings Get settings values
shell Interactive scraping console
startproject Create new project
version Print Scrapy version
view Open URL in browser, as seen by Scrapy

[ more ] More commands available when run from project directory

Use "scrapy <command> -h" to see more info about a command

部分安装日志如下:

C:\Users\admin>pip install pywin32-224-cp37-cp37m-win_amd64.whl
Processing c:\users\admin\pywin32-224-cp37-cp37m-win_amd64.whl
Installing collected packages: pywin32
Successfully installed pywin32-224

C:\Users\admin>pip install Scrapy-1.6.0-py2.py3-none-any.whl
Processing c:\users\admin\scrapy-1.6.0-py2.py3-none-any.whl
Collecting PyDispatcher>=2.0.5 (from Scrapy==1.6.0)
Downloading https://files.pythonhosted.org/packages/cd/37/39aca520918ce1935bea9c356bcbb7ed7e52ad4e31bff9b943dfc8e7115b/PyDispatcher-2.0.5.tar.gz
Collecting pyOpenSSL (from Scrapy==1.6.0)
Downloading https://files.pythonhosted.org/packages/01/c8/ceb170d81bd3941cbeb9940fc6cc2ef2ca4288d0ca8929ea4db5905d904d/pyOpenSSL-19.0.0-py2.py3-none-any.whl (53kB)
100% |████████████████████████████████| 61kB 64kB/s
Collecting cssselect>=0.9 (from Scrapy==1.6.0)
Downloading https://files.pythonhosted.org/packages/7b/44/25b7283e50585f0b4156960691d951b05d061abf4a714078393e51929b30/cssselect-1.0.3-py2.py3-none-any.whl
Collecting w3lib>=1.17.0 (from Scrapy==1.6.0)
Downloading https://files.pythonhosted.org/packages/81/43/9dcf92a77f5f0afe4f4df2407d7289eea01368a08b64bda00dd318ca62a6/w3lib-1.20.0-py2.py3-none-any.whl
Requirement already satisfied: Twisted>=13.1.0 in d:\python37\lib\site-packages (from Scrapy==1.6.0) (18.9.0)
Collecting service-identity (from Scrapy==1.6.0)
Downloading https://files.pythonhosted.org/packages/e9/7c/2195b890023e098f9618d43ebc337d83c8b38d414326685339eb024db2f6/service_identity-18.1.0-py2.py3-none-any.whl
Collecting queuelib (from Scrapy==1.6.0)
Downloading https://files.pythonhosted.org/packages/4c/85/ae64e9145f39dd6d14f8af3fa809a270ef3729f3b90b3c0cf5aa242ab0d4/queuelib-1.5.0-py2.py3-none-any.whl
Requirement already satisfied: six>=1.5.2 in d:\python37\lib\site-packages (from Scrapy==1.6.0) (1.12.0)
Requirement already satisfied: lxml in d:\python37\lib\site-packages (from Scrapy==1.6.0) (4.3.1)
Collecting parsel>=1.5 (from Scrapy==1.6.0)
Downloading https://files.pythonhosted.org/packages/96/69/d1d5dba5e4fecd41ffd71345863ed36a45975812c06ba77798fc15db6a64/parsel-1.5.1-py2.py3-none-any.whl
Collecting cryptography>=2.3 (from pyOpenSSL->Scrapy==1.6.0)
Downloading https://files.pythonhosted.org/packages/cd/ab/06deda5e26c4290d9973bbf2d53d09d09898f6f972576807c8e76ff13e76/cryptography-2.5-cp37-cp37m-win_amd64.whl (1.5MB)
100% |████████████████████████████████| 1.5MB 17kB/s
Requirement already satisfied: attrs>=17.4.0 in d:\python37\lib\site-packages (from Twisted>=13.1.0->Scrapy==1.6.0) (18.2.0)
Requirement already satisfied: hyperlink>=17.1.1 in d:\python37\lib\site-packages (from Twisted>=13.1.0->Scrapy==1.6.0) (18.0.0)
Requirement already satisfied: zope.interface>=4.4.2 in d:\python37\lib\site-packages (from Twisted>=13.1.0->Scrapy==1.6.0) (4.6.0)
Requirement already satisfied: constantly>=15.1 in d:\python37\lib\site-packages (from Twisted>=13.1.0->Scrapy==1.6.0) (15.1.0)
Requirement already satisfied: incremental>=16.10.1 in d:\python37\lib\site-packages (from Twisted>=13.1.0->Scrapy==1.6.0) (17.5.0)
Requirement already satisfied: Automat>=0.3.0 in d:\python37\lib\site-packages (from Twisted>=13.1.0->Scrapy==1.6.0) (0.7.0)
Requirement already satisfied: PyHamcrest>=1.9.0 in d:\python37\lib\site-packages (from Twisted>=13.1.0->Scrapy==1.6.0) (1.9.0)
Collecting pyasn1-modules (from service-identity->Scrapy==1.6.0)
Downloading https://files.pythonhosted.org/packages/da/98/8ddd9fa4d84065926832bcf2255a2b69f1d03330aa4d1c49cc7317ac888e/pyasn1_modules-0.2.4-py2.py3-none-any.whl (66kB)
100% |████████████████████████████████| 71kB 18kB/s
Collecting pyasn1 (from service-identity->Scrapy==1.6.0)
Downloading https://files.pythonhosted.org/packages/7b/7c/c9386b82a25115cccf1903441bba3cbadcfae7b678a20167347fa8ded34c/pyasn1-0.4.5-py2.py3-none-any.whl (73kB)
100% |████████████████████████████████| 81kB 13kB/s
Collecting cffi!=1.11.3,>=1.8 (from cryptography>=2.3->pyOpenSSL->Scrapy==1.6.0)
Downloading https://files.pythonhosted.org/packages/6b/a4/4d9ba87108405ef0612907c1da4595a2cb4fba91a18105e68c1b7751bfe3/cffi-1.12.0-cp37-cp37m-win_amd64.whl (170kB)
100% |████████████████████████████████| 174kB 11kB/s
Collecting asn1crypto>=0.21.0 (from cryptography>=2.3->pyOpenSSL->Scrapy==1.6.0)
Downloading https://files.pythonhosted.org/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl (101kB)
100% |████████████████████████████████| 102kB 18kB/s
Requirement already satisfied: idna>=2.5 in d:\python37\lib\site-packages (from hyperlink>=17.1.1->Twisted>=13.1.0->Scrapy==1.6.0) (2.8)
Requirement already satisfied: setuptools in d:\python37\lib\site-packages (from zope.interface>=4.4.2->Twisted>=13.1.0->Scrapy==1.6.0) (39.0.1)
Collecting pycparser (from cffi!=1.11.3,>=1.8->cryptography>=2.3->pyOpenSSL->Scrapy==1.6.0)
Downloading https://files.pythonhosted.org/packages/68/9e/49196946aee219aead1290e00d1e7fdeab8567783e83e1b9ab5585e6206a/pycparser-2.19.tar.gz (158kB)
100% |████████████████████████████████| 163kB 16kB/s
Building wheels for collected packages: PyDispatcher, pycparser
Building wheel for PyDispatcher (setup.py) ... done
Stored in directory: C:\Users\admin\AppData\Local\pip\Cache\wheels\88\99\96\cfef6665f9cb1522ee6757ae5955feedf2fe25f1737f91fa7f
Building wheel for pycparser (setup.py) ... done
Stored in directory: C:\Users\admin\AppData\Local\pip\Cache\wheels\f2\9a\90\de94f8556265ddc9d9c8b271b0f63e57b26fb1d67a45564511
Successfully built PyDispatcher pycparser
Installing collected packages: PyDispatcher, pycparser, cffi, asn1crypto, cryptography, pyOpenSSL, cssselect, w3lib, pyasn1, pyasn1-modules, service-identity, queuelib, parsel, Scrapy
Successfully installed PyDispatcher-2.0.5 Scrapy-1.6.0 asn1crypto-0.24.0 cffi-1.12.0 cryptography-2.5 cssselect-1.0.3 parsel-1.5.1 pyOpenSSL-19.0.0 pyasn1-0.4.5 pyasn1-modules-0.2.4 pycparser-2.19 queuelib-1.5.0 service-identity-18.1.0 w3lib-1.20.0

原文地址:https://www.cnblogs.com/jwr810/p/10387151.html

时间: 2024-11-05 19:28:02

python3.7 Scrapy在Windows平台的安装的相关文章

[TimLinux] scrapy 在Windows平台的安装

1. 安装Python 这个不去细说,官网直接下载,安装即可,我自己选择的版本是 Python 3.6.5 x86_64bit windows版本. 2. 配置PATH 我用的windows 10系统,操作步骤,'此电脑' 上鼠标右键,选择 '属性', 在弹出的面板中,选择 '高级系统设置', 新窗口中点击 '高级' 标签页,然后点击 '环境变量', 在用户环境变量中,选中 path(没有就添加),然后把:C:\Python365\Scripts;C:\Python365;添加到该变量值中即可.

windows平台下安装与配置mysql5.7

博主QQ:819594300 博客地址:http://zpf666.blog.51cto.com/ 有什么疑问的朋友可以联系博主,博主会帮你们解答,谢谢支持! 在windows上安装mysql5.7需要具有系统的管理员权限. Windows平台下提供两种安装方式: 1.mysql二进制分发版(.msi安装文件) 2.免安装版(.zip压缩文件) 一般来讲,我们使用二进制分发版,因为该版本比其他的分发版本使用起来要简单,不再需要其他工具来启动就可以运行mysql. 本例以window7平台为例进行

Windows平台上安装LaTex工具以及LaTex常见的文件类型汇总

LaTex广泛用于学术报告.paper.学术PPT的撰写和制作,熟练掌握LaTex是在学术界生存的必备基本技能之一.本文简要介绍了在Windows平台上安装LaTex工具的基本方法和流程以及LaTeX常见的文件类型. 1.下载MiKTeX,下载地址为:http://miktex.org/download.MiKTeX的正确读音为/'mik'tek/. 2.安装MikTeX.安装完毕后,在开始菜单会出现"Previewer"和"TeXworks"程序图标.Previe

MongoDB学习总结(一) —— Windows平台下安装

一.基本概念 MongoDB是一个基于分布式文件存储的开源数据库系统,皆在为WEB应用提供可扩展的高性能数据存储解决方案.MongoDB将数据存储为一个文档,数据结构由键值key=>value组成.文档类似JSON对象.字段值可以包含其他文档,数组及文档数组. MongoDB数据库的几个基本概念: 1)一个 MongoDB 实例中,可以有零个或多个database(数据库). 2)数据库中可以有零个或多个 collections (集合).相当于传统关系数据库中table(表). 3)集合是由零

windows 平台下 安装解密 openssl

1 在openssl 官网下载 openssl 安装, 本机是 64位 win 8.1 系统 http://slproweb.com/products/Win32OpenSSL.html 下载:Win64 OpenSSL v0.9.8zb Visual C++ 2008 Redistributables (x64) 2 下载ActivePerl 5.10.1.1007(最新的版本或较低的版本也可以): 下载地址:http://www.activestate.com/activeperl/down

[转]Windows平台下安装Hadoop

1.安装JDK1.6或更高版本 官网下载JDK,安装时注意,最好不要安装到带有空格的路径名下,例如:Programe Files,否则在配置Hadoop的配置文件时会找不到JDK(按相关说法,配置文件中的路径加引号即可解决,但我没测试成功). 2.安装Cygwin Cygwin是Windows平台下模拟Unix环境的工具,需要在安装Cygwin的基础上安装Hadoop,下载地址:http://www.cygwin.com/ 根据操作系统的需要下载32位或64的安装文件. 1).双击下载好的安装文

windows平台mongoDB安装配置

一.首先安装mongodb 1.官网下载mongoDB:http://www.mongodb.org/downloads,选择windows平台.安装时,一路next就可以了.我安装在了F:\mongoDB目录下. 2.建立工作目录: 在mongoDB目录下新建数据存放目录:F:\mongoDB\data\db 在mongoDB目录下新建日志文件:F:\mongoDB\log\mongodb.log 3.运行cmd.exe(以管理员身份打开),进入到F:\mongoDB\bin目录下,执行以下命

Arduino可穿戴开发入门教程Windows平台下安装Arduino IDE

Arduino可穿戴开发入门教程Windows平台下安装Arduino IDE Windows平台下安装Arduino IDE Windows操作系统下可以使用安装向导和压缩包形式安装.下面详细讲解这两种方式. 1.Arduino IDE Windows安装向导方式 Windows安装向导方式安装和绝大多数Windows应用程序安装类似,只需要直接运行下载的.exe文件即可.例如1.6.0版本的安装向导方式的文件名为arduino-1.6.0-windows.exe.直接双击该文件即可开始安装.

Windows平台下安装PhoenixSuit要点

在上手问题这个板块经常看到烧写固件失败的求助帖,这个帖子主要整理一下Windows平台下安装PhoenixSuit和刷机的要点,让您在5分钟内刷入任何CubieBoard的固件.       1.首先你要明确你的固件类型和板子类型是否一致,截止到2014.5.5,官网推出的3套固件:CB1/CB2/CB3(Cubietruck),另外,固件有分卡固件和Nand固件,2者是不能够通用的.       2.固件下载推荐这个官网ftp服务器 ,所有的固件更新都会第一时间推送到这个服务器,按照产品类型C