MongoDB部署实验系列文章
MongoDB做为NoSQL数据库，最近几年持续升温，越来越多的企业都开始尝试用MongoDB代替原有Database做一些事情。MongoDB也在集群，分片，复制上也有相当不错的的表现。我通过将做各种MongoDB的部署实验进行介绍。
原网址：
http://blog.fens.me/mongodb-shard/
第三篇 MongoDB 自动分片 auto sharding，分为6个部分
- 初始化文件目录
- 启动shard节点
- 配置shard节点
- 插入数据分片实验
- 删除主分片
- 重置主分片
系统环境介绍：
Ubuntu 12.04. LTS 64bit Server
1. 初始化文件目录
创建目录
- config1,config2,config3是配置节点
- shard1,shard2,shard3是分片节点
~ pwd /home/conan/dbs ~ mkdir config1 config2 config3 shard1 shard2 shard3 ~/dbs$ ls -l drwxrwxr-x 3 conan conan 4096 May 31 11:27 config1 drwxrwxr-x 3 conan conan 4096 May 31 11:27 config2 drwxrwxr-x 3 conan conan 4096 May 31 11:27 config3 drwxrwxr-x 3 conan conan 4096 May 31 11:28 shard1 drwxrwxr-x 3 conan conan 4096 May 31 11:29 shard2 drwxrwxr-x 3 conan conan 4096 May 31 11:29 shard3
2. 启动shard节点
启动config节点
mongod --dbpath config1 --port 20001 --fork --logpath config1.log mongod --dbpath config2 --port 20002 --fork --logpath config2.log mongod --dbpath config3 --port 20003 --fork --logpath config3.log
启动mongos节点
mongos --configdb localhost:20001,localhost:20002,localhost:20003 --port 30001 --fork --logpath mongos1.log mongos --configdb localhost:20001,localhost:20002,localhost:20003 --port 30002 --fork --logpath mongos2.log
启动shard节点
mongod --dbpath shard1 --port 10001 --fork --logpath shard1.log mongod --dbpath shard2 --port 10002 --fork --logpath shard2.log mongod --dbpath shard3 --port 10003 --fork --logpath shard3.log
查看端口
~ netstat -nlt
Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 0.0.0.0:10001 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:30001 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:10002 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:30002 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:10003 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:21 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:20001 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:20002 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:20003 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:27017 0.0.0.0:* LISTEN tcp 0 0 :::22 :::* LISTEN tcp 0 0 ::1:631 :::* LISTEN tcp 0 0 ::1:25 :::* LISTEN
3. 配置shard节点
连接mongos1，在mongos中添加分片：
# mongo localhost:30001/admin
MongoDB shell version: 3.0.5
connecting to: localhost:30001/admin
Server has startup warnings:
2015-08-12T23:37:56.997-0700 I CONTROL ** WARNING: You are running this process as the root user, which is not recommended.
2015-08-12T23:37:56.997-0700 I CONTROL
mongos> db.runCommand({addshard : "localhost:10001", allowLocal : true})
{ "shardAdded" : "shard0000", "ok" : 1 }
mongos> db.runCommand({addshard : "localhost:10002", allowLocal : true})
{ "shardAdded" : "shard0001", "ok" : 1 }
mongos> db.runCommand({addshard : "localhost:10003", allowLocal : true})
{ "shardAdded" : "shard0002", "ok" : 1 }
查看分片信息
mongos> db.runCommand({listshards:1})
{
"shards" : [
{
"_id" : "shard0000",
"host" : "localhost:10001"
},
{
"_id" : "shard0001",
"host" : "localhost:10002"
},
{
"_id" : "shard0002",
"host" : "localhost:10003"
}
],
"ok" : 1
}
启用数据库分片：fensme
mongos> db.runCommand({"enablesharding" : "fensme"})
{ "ok" : 1 }
注：一旦enable了个数据库，mongos将会把数据库里的不同数据集放在不同的分片上。只有数据集也被分片，否则一个数据集的所有数据将放在一个分片上。
启用数据集分片：fensme.users
db.runCommand({"shardcollection" : "fensme.users", "key" : {"_id" : 1,"uid":1}})
{ "collectionsharded" : "fensme.users", "ok" : 1 }
查看分片状态
mongos> db.printShardingStatus()
--- Sharding Status ---
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("55cc3b46396d4ce739e9eacc")
}
shards:
{ "_id" : "shard0000", "host" : "localhost:10001" }
{ "_id" : "shard0001", "host" : "localhost:10002" }
{ "_id" : "shard0002", "host" : "localhost:10003" }
balancer:
Currently enabled: yes
Currently running: no
Failed balancer rounds in last 5 attempts: 0
Migration Results for the last 24 hours:
No recent migrations
databases:
{ "_id" : "admin", "partitioned" : false, "primary" : "config" }
{ "_id" : "fensme", "partitioned" : true, "primary" : "shard0000" }
fensme.users
shard key: { "_id" : 1, "uid" : 1 }
chunks:
shard0000 1
{ "_id" : { "$minKey" : 1 }, "uid" : { "$minKey" : 1 } } -->> { "_id" : { "$maxKey" : 1 }, "uid" : { "$maxKey" : 1 } } on : shard0000 Timestamp(1, 0)
fensme数据库是支持shard的，主shard是shard0000，对应host是localhost:10001
再查看config数据库
mongos> use config
switched to db config
mongos> show collection
actionlog
changelog
chunks
collections
databases
lockpings
locks
mongos
settings
shards
system.indexes
tags
version
mongos> db.shards.find()
{ "_id" : "shard0000", "host" : "localhost:10001" }
{ "_id" : "shard0001", "host" : "localhost:10002" }
{ "_id" : "shard0002", "host" : "localhost:10003" }
mongos> db.chunks.find()
{ "_id" : "fensme.users-_id_MinKeyuid_MinKey", "lastmod" : Timestamp(1, 0), "lastmodEpoch" : ObjectId("55cc4b46396d4ce739e9edcf"), "ns" : "fensme.users", "min" : { "_id" : { "$minKey" : 1 }, "uid" : { "$minKey" : 1 } }, "max" : { "_id" : { "$maxKey" : 1 }, "uid" : { "$maxKey" : 1 } }, "shard" : "shard0000" }
shards配置成功，分片信息都正确
4. 插入数据分片实验
向fensme.users插入数据, 批量插入10w条记录
mongos> use fensme
switched to db fensme
mongos> for(var i=0; i<10000; i++){
... db.users.insert({_id:i*1597,uid:i});
... }
WriteResult({ "nInserted" : 1 })
mongos> db.users.find()
{ "_id" : 0, "uid" : 0 }
{ "_id" : 1597, "uid" : 1 }
{ "_id" : 33537, "uid" : 21 }
{ "_id" : 3194, "uid" : 2 }
{ "_id" : 35134, "uid" : 22 }
{ "_id" : 4791, "uid" : 3 }
{ "_id" : 36731, "uid" : 23 }
{ "_id" : 6388, "uid" : 4 }
{ "_id" : 38328, "uid" : 24 }
{ "_id" : 7985, "uid" : 5 }
{ "_id" : 39925, "uid" : 25 }
{ "_id" : 9582, "uid" : 6 }
{ "_id" : 41522, "uid" : 26 }
{ "_id" : 11179, "uid" : 7 }
{ "_id" : 43119, "uid" : 27 }
{ "_id" : 12776, "uid" : 8 }
{ "_id" : 44716, "uid" : 28 }
{ "_id" : 14373, "uid" : 9 }
{ "_id" : 46313, "uid" : 29 }
{ "_id" : 15970, "uid" : 10 }
Type "it" for more
查看数据分片存储信息
mongos> db.users.stats()
{
"sharded" : true,
"paddingFactorNote" : "paddingFactor is unused and unmaintained in 3.0. It remains hard coded to 1.0 for compatibility only.",
"userFlags" : 1,
"capped" : false,
"ns" : "fensme.users",
"count" : 10000,
"numExtents" : 6,
"size" : 480048,
"storageSize" : 712704,
"totalIndexSize" : 711312,
"indexSizes" : {
"_id_" : 302512,
"_id_1_uid_1" : 408800
},
"avgObjSize" : 48.0048,
"nindexes" : 2,
"nchunks" : 3,
"shards" : {
"shard0000" : {
"ns" : "fensme.users",
"count" : 9979,
"size" : 479040,
"avgObjSize" : 48,
"numExtents" : 4,
"storageSize" : 696320,
"lastExtentSize" : 524288,
"paddingFactor" : 1,
"paddingFactorNote" : "paddingFactor is unused and unmaintained in 3.0. It remains hard coded to 1.0 for compatibility only.",
"userFlags" : 1,
"capped" : false,
"nindexes" : 2,
"totalIndexSize" : 678608,
"indexSizes" : {
"_id_" : 286160,
"_id_1_uid_1" : 392448
},
"ok" : 1
},
"shard0001" : {
"ns" : "fensme.users",
"count" : 1,
"size" : 48,
"avgObjSize" : 48,
"numExtents" : 1,
"storageSize" : 8192,
"lastExtentSize" : 8192,
"paddingFactor" : 1,
"paddingFactorNote" : "paddingFactor is unused and unmaintained in 3.0. It remains hard coded to 1.0 for compatibility only.",
"userFlags" : 1,
"capped" : false,
"nindexes" : 2,
"totalIndexSize" : 16352,
"indexSizes" : {
"_id_" : 8176,
"_id_1_uid_1" : 8176
},
"ok" : 1
},
"shard0002" : {
"ns" : "fensme.users",
"count" : 20,
"size" : 960,
"avgObjSize" : 48,
"numExtents" : 1,
"storageSize" : 8192,
"lastExtentSize" : 8192,
"paddingFactor" : 1,
"paddingFactorNote" : "paddingFactor is unused and unmaintained in 3.0. It remains hard coded to 1.0 for compatibility only.",
"userFlags" : 1,
"capped" : false,
"nindexes" : 2,
"totalIndexSize" : 16352,
"indexSizes" : {
"_id_" : 8176,
"_id_1_uid_1" : 8176
},
"ok" : 1
}
},
"ok" : 1
}
查看users数据集信息，shard0000上面9979条，shard0001上面1条，shard0002上面20条。分片很不均匀。
分别连接到shard1,shard2,shard3查看数据分片存储的情况
连接shard1
# mongo localhost:10001 MongoDB shell version: 3.0.5 connecting to: localhost:10001/test> show dbsshow dbs fensme 0.078GB local 0.078GB > use fensme switched to db fensme > show collections system.indexes users > db.users.count() 9979
连接shard2, 1条记录
# mongo localhost:10002 MongoDB shell version: 3.0.5 connecting to: localhost:10002/test> use fensme switched to db fensme > db.users.count() 1
连接shard3，20
# mongo localhost:10003 MongoDB shell version: 3.0.5 connecting to: localhost:10003/test> use fensme switched to db fensme > db.users.count() 20
注：分片数据分配不均匀，应该重新规化分片的key。
5. 删除主分片
移除Primary分片shard1，localhost:10001
mongos> db.runCommand({"removeshard":"localhost:10001"})
{
"msg" : "draining started successfully",
"state" : "started",
"shard" : "shard0000",
"note" : "you need to drop or movePrimary these databases",
"dbsToMove" : [
"fensme"
],
"ok" : 1
}
提示shard0000是主分片，要手动迁移主分片。
查看分片状态
mongos> db.runCommand({listshards:1})
{
"shards" : [
{
"_id" : "shard0000",
"host" : "localhost:10001",
"draining" : true
},
{
"_id" : "shard0001",
"host" : "localhost:10002"
},
{
"_id" : "shard0002",
"host" : "localhost:10003"
}
],
"ok" : 1
}
draining为正在迁移过程中。。。。
mongos> db.printShardingStatus()
--- Sharding Status ---
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("55cc3b46396d4ce739e9eacc")
}
shards:
{ "_id" : "shard0000", "host" : "localhost:10001", "draining" : true }
{ "_id" : "shard0001", "host" : "localhost:10002" }
{ "_id" : "shard0002", "host" : "localhost:10003" }
balancer:
Currently enabled: yes
Currently running: no
Failed balancer rounds in last 5 attempts: 0
Migration Results for the last 24 hours:
3 : Success
1 : Failed with error ‘could not acquire collection lock for fensme.users to migrate chunk [{ : MinKey },{ : MaxKey }) :: caused by :: Lock for migrating chunk [{ : MinKey }, { : MaxKey }) in fensme.users is taken.‘, from shard0000 to shard0002
databases:
{ "_id" : "admin", "partitioned" : false, "primary" : "config" }
{ "_id" : "fensme", "partitioned" : true, "primary" : "shard0000" }
fensme.users
shard key: { "_id" : 1, "uid" : 1 }
chunks:
shard0001 2
shard0002 1
{ "_id" : { "$minKey" : 1 }, "uid" : { "$minKey" : 1 } } -->> { "_id" : 1597, "uid" : 1 } on : shard0001 Timestamp(2, 0)
{ "_id" : 1597, "uid" : 1 } -->> { "_id" : 33537, "uid" : 21 } on : shard0002 Timestamp(3, 0)
{ "_id" : 33537, "uid" : 21 } -->> { "_id" : { "$maxKey" : 1 }, "uid" : { "$maxKey" : 1 } } on : shard0001 Timestamp(4, 0)
draining为正在迁移过程中。。。。
查看db.users的分片存储分布
mongos> use fensme
switched to db fensme
mongos> db.users.stats()
{
"sharded" : true,
"paddingFactorNote" : "paddingFactor is unused and unmaintained in 3.0. It remains hard coded to 1.0 for compatibility only.",
"userFlags" : 1,
"capped" : false,
"ns" : "fensme.users",
"count" : 10000,
"numExtents" : 9,
"size" : 480048,
"storageSize" : 1400832,
"totalIndexSize" : 711312,
"indexSizes" : {
"_id_" : 302512,
"_id_1_uid_1" : 408800
},
"avgObjSize" : 48.0048,
"nindexes" : 2,
"nchunks" : 3,
"shards" : {
"shard0000" : {
"ns" : "fensme.users",
"count" : 0,
"size" : 0,
"numExtents" : 4,
"storageSize" : 696320,
"lastExtentSize" : 524288,
"paddingFactor" : 1,
"paddingFactorNote" : "paddingFactor is unused and unmaintained in 3.0. It remains hard coded to 1.0 for compatibility only.",
"userFlags" : 1,
"capped" : false,
"nindexes" : 2,
"totalIndexSize" : 16352,
"indexSizes" : {
"_id_" : 8176,
"_id_1_uid_1" : 8176
},
"ok" : 1
},
"shard0001" : {
"ns" : "fensme.users",
"count" : 9980,
"size" : 479088,
"avgObjSize" : 48,
"numExtents" : 4,
"storageSize" : 696320,
"lastExtentSize" : 524288,
"paddingFactor" : 1,
"paddingFactorNote" : "paddingFactor is unused and unmaintained in 3.0. It remains hard coded to 1.0 for compatibility only.",
"userFlags" : 1,
"capped" : false,
"nindexes" : 2,
"totalIndexSize" : 678608,
"indexSizes" : {
"_id_" : 286160,
"_id_1_uid_1" : 392448
},
"ok" : 1
},
"shard0002" : {
"ns" : "fensme.users",
"count" : 20,
"size" : 960,
"avgObjSize" : 48,
"numExtents" : 1,
"storageSize" : 8192,
"lastExtentSize" : 8192,
"paddingFactor" : 1,
"paddingFactorNote" : "paddingFactor is unused and unmaintained in 3.0. It remains hard coded to 1.0 for compatibility only.",
"userFlags" : 1,
"capped" : false,
"nindexes" : 2,
"totalIndexSize" : 16352,
"indexSizes" : {
"_id_" : 8176,
"_id_1_uid_1" : 8176
},
"ok" : 1
}
},
"ok" : 1
}
发现shard0000的数据，都被自动迁移到了shard0001
连接node2，发现9980条数据全在这里。
# mongo localhost:10002 MongoDB shell version: 3.0.5 connecting to: localhost:10002/test> use fensme switched to db fensme > db.users.count() 9980
6. 重置主分片
重新设置主分片为shard0002
mongos> use adminuse admin
switched to db admin
mongos> db.runCommand({"moveprimary":"fensme","to":"localhost:10003"})
{ "primary " : "shard0002:localhost:10003", "ok" : 1 }
再次删除node1, shard0000
mongos> use adminuse admin
switched to db admin
mongos> db.runCommand({"removeshard":"localhost:10001"})
{
"msg" : "removeshard completed successfully",
"state" : "completed",
"shard" : "shard0000",
"ok" : 1
}
删除分片成功。
查看分片信息
mongos> use adminuse admin
switched to db admin
mongos> db.printShardingStatus()
--- Sharding Status ---
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("55cc3b46396d4ce739e9eacc")
}
shards:
{ "_id" : "shard0001", "host" : "localhost:10002" }
{ "_id" : "shard0002", "host" : "localhost:10003" }
balancer:
Currently enabled: yes
Currently running: no
Failed balancer rounds in last 5 attempts: 0
Migration Results for the last 24 hours:
3 : Success
1 : Failed with error ‘could not acquire collection lock for fensme.users to migrate chunk [{ : MinKey },{ : MaxKey }) :: caused by :: Lock for migrating chunk [{ : MinKey }, { : MaxKey }) in fensme.users is taken.‘, from shard0000 to shard0002
databases:
{ "_id" : "admin", "partitioned" : false, "primary" : "config" }
{ "_id" : "fensme", "partitioned" : true, "primary" : "shard0002" }
fensme.users
shard key: { "_id" : 1, "uid" : 1 }
chunks:
shard0001 2
shard0002 1
{ "_id" : { "$minKey" : 1 }, "uid" : { "$minKey" : 1 } } -->> { "_id" : 1597, "uid" : 1 } on : shard0001 Timestamp(2, 0)
{ "_id" : 1597, "uid" : 1 } -->> { "_id" : 33537, "uid" : 21 } on : shard0002 Timestamp(3, 0)
{ "_id" : 33537, "uid" : 21 } -->> { "_id" : { "$maxKey" : 1 }, "uid" : { "$maxKey" : 1 } } on : shard0001 Timestamp(4, 0)
fensme的数据库的primary分片，成功的改为了shard0002.
再看数据的分布情况：
mongos> use fensme
switched to db fensme
mongos> db.users.stats()
{
"sharded" : true,
"paddingFactorNote" : "paddingFactor is unused and unmaintained in 3.0. It remains hard coded to 1.0 for compatibility only.",
"userFlags" : 1,
"capped" : false,
"ns" : "fensme.users",
"count" : 10000,
"numExtents" : 5,
"size" : 480048,
"storageSize" : 704512,
"totalIndexSize" : 694960,
"indexSizes" : {
"_id_" : 294336,
"_id_1_uid_1" : 400624
},
"avgObjSize" : 48.0048,
"nindexes" : 2,
"nchunks" : 3,
"shards" : {
"shard0001" : {
"ns" : "fensme.users",
"count" : 9980,
"size" : 479088,
"avgObjSize" : 48,
"numExtents" : 4,
"storageSize" : 696320,
"lastExtentSize" : 524288,
"paddingFactor" : 1,
"paddingFactorNote" : "paddingFactor is unused and unmaintained in 3.0. It remains hard coded to 1.0 for compatibility only.",
"userFlags" : 1,
"capped" : false,
"nindexes" : 2,
"totalIndexSize" : 678608,
"indexSizes" : {
"_id_" : 286160,
"_id_1_uid_1" : 392448
},
"ok" : 1
},
"shard0002" : {
"ns" : "fensme.users",
"count" : 20,
"size" : 960,
"avgObjSize" : 48,
"numExtents" : 1,
"storageSize" : 8192,
"lastExtentSize" : 8192,
"paddingFactor" : 1,
"paddingFactorNote" : "paddingFactor is unused and unmaintained in 3.0. It remains hard coded to 1.0 for compatibility only.",
"userFlags" : 1,
"capped" : false,
"nindexes" : 2,
"totalIndexSize" : 16352,
"indexSizes" : {
"_id_" : 8176,
"_id_1_uid_1" : 8176
},
"ok" : 1
}
},
"ok" : 1
}
shard0001包括9980条数据，shard0002包括20数据，shard0000已经不存储
时间： 2024-11-11 03:32:15