这里面我们介绍一下python中操作mangodb的第三方库pymongo的使用，以及简单的使用requests库作爬虫。人情冷暖正如花开花谢，不如将这种现象，想成一种必然的季节。
pymongo的安装及前期准备
一、mangodb的安装以及启动
测试机器：win10， mangodb版本v3.4.0，python版本3.6.3。
mangodb的安装目录：D:\Database\DataBase\Mongo。数据的存放目录：E:\data\database\mango\data。首先我们启动mangodb服务器的：可以看到在本地27017端口成功启动server。
D:\Database\DataBase\Mongo\Server\3.4\bin>mongod --dbpath E:\data\database\mango\data
2017-11-21T20:48:38.458+0800 I CONTROL [initandlisten] MongoDB starting : pid=20484 port=27017 dbpath=E:\data\database\mango\data 64-bit host=Linux
2017-11-21T20:48:38.461+0800 I CONTROL [initandlisten] targetMinOS: Windows 7/Windows Server 2008 R2
2017-11-21T20:48:38.462+0800 I CONTROL [initandlisten] db version v3.4.0
2017-11-21T20:48:38.463+0800 I CONTROL [initandlisten] git version: f4240c60f005be757399042dc12f6addbc3170c1
2017-11-21T20:48:38.464+0800 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1t-fips 3 May 2016
2017-11-21T20:48:38.465+0800 I CONTROL [initandlisten] allocator: tcmalloc
2017-11-21T20:48:38.466+0800 I CONTROL [initandlisten] modules: none
2017-11-21T20:48:38.466+0800 I CONTROL [initandlisten] build environment:
2017-11-21T20:48:38.467+0800 I CONTROL [initandlisten] distmod: 2008plus-ssl
2017-11-21T20:48:38.468+0800 I CONTROL [initandlisten] distarch: x86_64
2017-11-21T20:48:38.469+0800 I CONTROL [initandlisten] target_arch: x86_64
2017-11-21T20:48:38.469+0800 I CONTROL [initandlisten] options: { storage: { dbPath: "E:\data\database\mango\data" } }
2017-11-21T20:48:38.491+0800 I - [initandlisten] Detected data files in E:\data\database\mango\data created by the ‘wiredTiger‘ storage engine, so setting the active storage engine to ‘wiredTiger‘.
2017-11-21T20:48:38.493+0800 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=5573M,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2017-11-21T20:48:39.931+0800 I CONTROL [initandlisten]
2017-11-21T20:48:39.933+0800 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database.
2017-11-21T20:48:39.936+0800 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted.
2017-11-21T20:48:39.940+0800 I CONTROL [initandlisten]
2017-11-21T20:48:41.253+0800 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory ‘E:/data/database/mango/data/diagnostic.data‘
2017-11-21T20:48:41.259+0800 I NETWORK [thread1] waiting for connections on port 27017
mangodb客户端的启动：D:\Database\DataBase\Mongo\Server\3.4\bin\mongo.exe。双击即可运行
MongoDB shell version v3.4.0 connecting to: mongodb://127.0.0.1:27017 MongoDB server version: 3.4.0 Server has startup warnings: 2017-11-21T20:48:39.931+0800 I CONTROL [initandlisten] 2017-11-21T20:48:39.933+0800 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database. 2017-11-21T20:48:39.936+0800 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted. 2017-11-21T20:48:39.940+0800 I CONTROL [initandlisten] >
二、python中pymongo的安装
pip install pymongo
这里简单的介绍一下pymongo的使用，这里面的代码是选自github的入门例子。
>>> import pymongo
>>> client = pymongo.MongoClient("localhost", 27017)
>>> db = client.test
>>> db.name
u‘test‘
>>> db.my_collection
Collection(Database(MongoClient(‘localhost‘, 27017), u‘test‘), u‘my_collection‘)
>>> db.my_collection.insert_one({"x": 10}).inserted_id
ObjectId(‘4aba15ebe23f6b53b0000000‘)
>>> db.my_collection.insert_one({"x": 8}).inserted_id
ObjectId(‘4aba160ee23f6b543e000000‘)
>>> db.my_collection.insert_one({"x": 11}).inserted_id
ObjectId(‘4aba160ee23f6b543e000002‘)
>>> db.my_collection.find_one()
{u‘x‘: 10, u‘_id‘: ObjectId(‘4aba15ebe23f6b53b0000000‘)}
>>> for item in db.my_collection.find():
... print(item["x"])
...
10
8
11
>>> db.my_collection.create_index("x")
u‘x_1‘
>>> for item in db.my_collection.find().sort("x", pymongo.ASCENDING):
... print(item["x"])
...
8
10
11
>>> [item["x"] for item in db.my_collection.find().limit(2).skip(1)]
[8, 11]
pymongo的使用例子
一、python爬虫以及pymongo存储数据
import requests
import pymongo
import json
def requestData():
url = ‘http://****.com/*.do‘
data = {
‘projectId‘: 90,
‘myTaskFlag‘: 1,
‘userId‘: 40
}
json_data = requests.post(url, data=json.dumps(data)).json()
return json_data
def output_data(json_data):
client = pymongo.MongoClient(host=‘localhost‘, port=27017)
db = client.test
collection = db.tasks
tasks_data = json_data.get(‘taskList‘)
collection.insert(tasks_data)
if __name__ == ‘__main__‘:
json_data = requestData()
output_data(json_data)
我们把得到的数据存放在tasks集合中，这里使用的是mangodb默认的test数据库。运行完程序，我们可以通过mangodb的客户端查看数据，运行：db.tasks.find().pretty()可以查询tasks集合的所有数据。
{
"_id" : ObjectId("5a1427a2edc9f04be40bc02d"),
"taskId" : 1,
"summary" : "PC版“个人信息”页面优化",
"status" : 8,
"categoryId" : 3,
"creatorId" : 7,
"projectId" : 1,
"dateSubmit" : NumberLong("1481105108000"),
"level" : 1,
"handlerId" : 2,
"ViewState" : 2,
"priority" : 2
} {
"_id" : ObjectId("5a1427a2edc9f04be40bc02e"),
"taskId" : 2,
"summary" : "PC版“添加新任务”界面字体太大",
"status" : 8,
"categoryId" : 3,
"creatorId" : 7,
"projectId" : 1,
"dateSubmit" : NumberLong("1481105195000"),
"level" : 1,
"handlerId" : 2,
"ViewState" : 2,
"priority" : 1
}
友情链接
时间： 2024-10-29 04:15:25