Mongo实战之数据空洞的最佳实践

问题背景：某天,开发部的同事跑过来反映: mongodb数据文件太大，快把磁盘撑爆了！其中某个db占用最大(运营环境这个db的数据量其实很小)

分析: 开发环境有大量测试的增/删/改操作,而由于MongoDB顺序写的原因，在我们删除部分无用数据后，它的storageSize并不会变小，这就造成了大量的数据空洞。

解决办法

1. 使用MongoDB自带的compact命令：

  db.collectionName.runCommand(“compact”)
 这种方式是collection级别的压缩，只能去除collection内的碎片。但是MongoDB的数据分配是DB级别的，效果有限。
 这个压缩是线上压缩，磁盘IO会比较高，从而影响线上服务。

2.采用复制集的方式滚动瘦身(offline)

1.检查服务器各节点是否正常运行 (ps -ef |grep mongod)
2.登入要处理的主节点,做降权处理rs.stepDown(),并通过命令 rs.status()来查看是否降权
3.切换成功之后，停掉该节点,删除数据文件,比如： rm -fr /mongodb/data/*
4.重新启动该节点，执行重启命令，比如：如:/mongodb/bin/mongod --config /mongodb/mongodb.conf
5.通过日志查看进程以及数据同步的进度
6.数据同步完成后,在修改后的主节点上执行命令 rs.stepDown(),做降权处理。

通过这种方式,可以做到收缩率是100%,数据完全无碎片.当然也会带来运维成本的增加，并且在Replic-Set集群只有2个副本的情况下，
还会存在一段时间内的单点风险(在下面的实验中就发生了这样的情况)。通过Offline的数据收缩后,收缩前后效果非常明显.

系统环境(实验环境,为了简单只做了1个副本,请勿在生产环境操作！！！)

主库: 192.168.2.130:27017
从库: 192.168.2.138:27017

1.安装mongo

2.配置参数(两台上配置)

[[email protected] bin]# vi mongodb.conf

dbpath=/u01/mongodb/data
port=27017
oplogSize = 2048
logpath=/u01/mongodb/logs/mongodb.log
logappend = true
fork = true
nojournal = true
bind_ip=0.0.0.0
shardsvr=true
replSet=relp1

3.启动mongo

[[email protected] bin]# ./mongod --config ./mongodb.conf

4.初始化副本集

[[email protected] bin]# ./mongo  192.168.2.130:27017

> cfg = {"_id":"repl1","members":[ {"_id":0, "host":"192.168.2.130:27017"}, {"_id":1, "host":"192.168.2.138:27017"}]};
{
        "_id" : "repl1",
        "members" : [
                {
                        "_id" : 0,
                        "host" : "192.168.2.130:27017"
                },
                {
                        "_id" : 1,
                        "host" : "192.168.2.138:27017"
                }
        ]
}
>
> rs.initiate(cfg)rs.initiate(cfg)
{ "ok" : 1 }

5.插入数据

#准备工作 安装pip工具 pymongo
[[email protected] mongodb]# tar zxvf setuptools-0.6c11.tar.gz
[[email protected] mongodb]# cd setuptools-0.6c11
[[email protected] mongodb]# python setup.py install

[[email protected] mongodb]# wget "https://pypi.python.org/packages/source/p/pip/pip-1.5.4.tar.gz#md5=834b2904f92d46aaa333267fb1c922bb" --no-check-certificate
[[email protected] mongodb]# tar zxvf [[email protected] mongodb]# tar pip-1.5.4.tar.gz
[[email protected] mongodb]# cd pip-1.5.4
[[email protected] pip-1.5.4]# python setup.py install

[[email protected] mongodb]# pip install pymongo

#插入脚本
vi insert.py
#!/usr/bin/python
import random
from pymongo import MongoClient

client = MongoClient('192.168.2.137', 27017)

test = client.test
students = test.students
students_count = students.count()
print "student count is ", students_count

for i in xrange(0,5000000):
    classid = random.randint(1,4)
    age = random.randint(10, 30)
    student = {"classid":classid, "age":age,
"name":"fujun"}
    students.insert_one(student)
    print i

students_count = students.count()
print "student count is ", students_count

#执行插入,可以开多个窗口执行
[[email protected] mongodb]#python insert.py

#插入完毕 2kw条记录
repl1:PRIMARY> db.students.find().count()
21511778

#查看复制集状态
repl1:PRIMARY> rs.status()rs.status()
{
        "set" : "repl1",
        "date" : ISODate("2018-03-17T02:42:01.126Z"),
        "myState" : 1,
        "term" : NumberLong(3),
        "heartbeatIntervalMillis" : NumberLong(2000),
        "members" : [
                {
                        "_id" : 0,
                        "name" : "192.168.2.130:27017",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "uptime" : 60089,
                        "optime" : {
                                "ts" : Timestamp(1521231587, 19),
                                "t" : NumberLong(3)
                        },
                        "optimeDate" : ISODate("2018-03-16T20:19:47Z"),
                        "electionTime" : Timestamp(1521213635, 1),
                        "electionDate" : ISODate("2018-03-16T15:20:35Z"),
                        "configVersion" : 1,
                        "self" : true
                },
                {
                        "_id" : 1,
                        "name" : "192.168.2.138:27017",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 40892,
                        "optime" : {
                                "ts" : Timestamp(1521231587, 19),
                                "t" : NumberLong(3)
                        },
                        "optimeDate" : ISODate("2018-03-16T20:19:47Z"),
                        "lastHeartbeat" : ISODate("2018-03-17T02:42:01Z"),
                        "lastHeartbeatRecv" : ISODate("2018-03-17T02:42:00.376Z"),
                        "pingMs" : NumberLong(1),
                        "syncingTo" : "192.168.2.130:27017",
                        "configVersion" : 1
                }
        ],
        "ok" : 1
}

#查看storageSize和fileSize:
repl1:PRIMARY> db.stats()
{
        "db" : "test",
        "collections" : 1,
        "objects" : 21511778,
        "avgObjSize" : 60,
        "dataSize" : 1290706680,
        "storageSize" : 410132480,
        "numExtents" : 0,
        "indexes" : 1,
        "indexSize" : 217350144,
        "ok" : 1
}

#storageSize = dataSize+size(删除的文档)（文档删除后，storageSize并不会变小）。

6.制造空洞数据

#删除3/4数据
repl1:PRIMARY> db.students.remove({"classid":{"$lt":4}}) 

WriteResult({ "nRemoved" : 9050980 })

#还剩下500w
repl1:PRIMARY> db.students.find().count()db.students.find().count()
5376340

#storageSize没有变下。
repl1:PRIMARY>db.stats()
{
        "db" : "test",
        "collections" : 1,
        "objects" : 5376340,
        "avgObjSize" : 60,
        "dataSize" : 322580400,
        "storageSize" : 425127936,
        "numExtents" : 0,
        "indexes" : 1,
        "indexSize" : 219590656,
        "ok" : 1
}

7.primary降权、停止

repl1:PRIMARY> rs.setrs.sers.stepDown()rs.stepDown()
2018-03-17T13:46:49.588+0800 E QUERY    [thread1] Error: error doing query: failed: network error while attempting to run command 'replSetStepDown' on host '192.168.2.130:27017'  :
[email protected]/mongo/shell/db.js:135:1
[email protected]/mongo/shell/db.js:153:16
[email protected]/mongo/shell/utils.js:1202:12
@(shell):1:1

2018-03-17T13:46:49.600+0800 I NETWORK  [thread1] trying reconnect to 192.168.2.130:27017 (192.168.2.130) failed
2018-03-17T13:46:49.618+0800 I NETWORK  [thread1] reconnect 192.168.2.130:27017 (192.168.2.130) ok
repl1:SECONDARY> 

#停止192.168.2.130
[[email protected] mongodb]# mongod --shutdown --config=/u01/mongodb/mongodb.conf
killing process with pid: 32322

8.删除192.168.2.130 上的数据文件

[[email protected] data]# rm -rf /u01/mongodb/data/*

9.重启192.168.2.130

[[email protected] data]# mongod  --config=/u01/mongodb/mongodb.conf
about to fork child process, waiting until server is ready for connections.
forked process: 48079
child process started successfully, parent exiting

10.查看日志,跟踪数据同步过程

[[email protected] mongodb]# cd /u01/mongodb/logs/
[[email protected] logs]# tail -50f  mongodb.log

2018-03-17T13:53:31.077+0800 I REPL     [replExecDBWorker-0] Starting replication applier threads
2018-03-17T13:53:31.077+0800 I REPL     [ReplicationExecutor]
2018-03-17T13:53:31.078+0800 I REPL     [ReplicationExecutor] ** WARNING: This replica set is running without journaling enabled but the
2018-03-17T13:53:31.078+0800 I REPL     [ReplicationExecutor] **          writeConcernMajorityJournalDefault option to the replica set config
2018-03-17T13:53:31.078+0800 I REPL     [ReplicationExecutor] **          is set to true. The writeConcernMajorityJournalDefault
2018-03-17T13:53:31.078+0800 I REPL     [ReplicationExecutor] **          option to the replica set config must be set to false
2018-03-17T13:53:31.078+0800 I REPL     [ReplicationExecutor] **          or w:majority write concerns will never complete.
2018-03-17T13:53:31.078+0800 I REPL     [ReplicationExecutor]
2018-03-17T13:53:31.078+0800 I REPL     [ReplicationExecutor] New replica set config in use: { _id: "repl1", version: 1, protocolVersion: 1, members: [ { _id: 0, host: "192.168.2.130:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 1, host: "192.168.2.138:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 }, replicaSetId: ObjectId('5aab9f92276e84b589038188') } }
2018-03-17T13:53:31.078+0800 I REPL     [ReplicationExecutor] This node is 192.168.2.130:27017 in the config
2018-03-17T13:53:31.078+0800 I REPL     [ReplicationExecutor] transition to STARTUP2
2018-03-17T13:53:31.079+0800 I REPL     [rsSync] ******
2018-03-17T13:53:31.079+0800 I REPL     [rsSync] creating replication oplog of size: 2048MB...
2018-03-17T13:53:31.298+0800 I REPL     [ReplicationExecutor] Member 192.168.2.138:27017 is now in state SECONDARY
2018-03-17T13:53:31.915+0800 I STORAGE  [rsSync] Starting WiredTigerRecordStoreThread local.oplog.rs
2018-03-17T13:53:31.915+0800 I STORAGE  [rsSync] The size storer reports that the oplog contains 0 records totaling to 0 bytes
2018-03-17T13:53:31.916+0800 I STORAGE  [rsSync] Scanning the oplog to determine where to place markers for truncation
2018-03-17T13:53:34.635+0800 I REPL     [rsSync] ******
2018-03-17T13:53:34.635+0800 I REPL     [rsSync] initial sync pending
2018-03-17T13:53:37.219+0800 I REPL     [ReplicationExecutor] syncing from: 192.168.2.138:27017
2018-03-17T13:53:39.164+0800 I REPL     [rsSync] initial sync drop all databases
2018-03-17T13:53:39.165+0800 I STORAGE  [rsSync] dropAllDatabasesExceptLocal 1
2018-03-17T13:53:39.165+0800 I REPL     [rsSync] initial sync clone all databases
2018-03-17T13:53:39.281+0800 I REPL     [rsSync] fetching and creating collections for test
2018-03-17T13:53:47.061+0800 I REPL     [rsSync] initial sync cloning db: test
2018-03-17T14:19:07.020+0800 I STORAGE  [rsSync] clone test.students 70015
2018-03-17T14:19:07.021+0800 I STORAGE  [rsSync] 70105 objects cloned so far from collection test.students
2018-03-17T14:20:34.083+0800 I STORAGE  [rsSync] 139955 objects cloned so far from collection test.students
2018-03-17T14:20:34.084+0800 I STORAGE  [rsSync] clone test.students 140031
2018-03-17T14:21:38.094+0800 I STORAGE  [rsSync] 489459 objects cloned so far from collection test.students
2018-03-17T14:21:38.094+0800 I STORAGE  [rsSync] clone test.students 489471
2018-03-17T14:22:51.519+0800 I STORAGE  [rsSync] 769113 objects cloned so far from collection test.students
2018-03-17T14:22:51.519+0800 I STORAGE  [rsSync] clone test.students 769151
2018-03-17T14:24:07.410+0800 I STORAGE  [rsSync] 978790 objects cloned so far from collection test.students
2018-03-17T14:24:07.410+0800 I STORAGE  [rsSync] clone test.students 978815
2018-03-17T14:25:10.198+0800 I STORAGE  [rsSync] 1468121 objects cloned so far from collection test.students
2018-03-17T14:25:10.198+0800 I STORAGE  [rsSync] clone test.students 1468159
2018-03-17T14:25:44.018+0800 I NETWORK  [initandlisten] connection accepted from 127.0.0.1:48042 #7 (4 connections now open)
2018-03-17T14:26:01.100+0800 I NETWORK  [conn5] end connection 127.0.0.1:48032 (3 connections now open)
2018-03-17T14:26:01.100+0800 I NETWORK  [conn7] end connection 127.0.0.1:48042 (3 connections now open)
2018-03-17T14:26:16.318+0800 I STORAGE  [rsSync] 1887602 objects cloned so far from collection test.students
2018-03-17T14:26:16.318+0800 I STORAGE  [rsSync] clone test.students 1887615
2018-03-17T14:27:19.201+0800 I STORAGE  [rsSync] clone test.students 2307071
2018-03-17T14:27:19.201+0800 I STORAGE  [rsSync] 2307083 objects cloned so far from collection test.students
2018-03-17T14:28:22.898+0800 I STORAGE  [rsSync] clone test.students 3145855
2018-03-17T14:28:22.899+0800 I STORAGE  [rsSync] 3145918 objects cloned so far from collection test.students
2018-03-17T14:29:25.991+0800 I STORAGE  [rsSync] 3565272 objects cloned so far from collection test.students
2018-03-17T14:29:25.991+0800 I STORAGE  [rsSync] clone test.students 3565311
2018-03-17T14:30:28.780+0800 I STORAGE  [rsSync] 3984753 objects cloned so far from collection test.students
2018-03-17T14:30:28.780+0800 I STORAGE  [rsSync] clone test.students 3984767
2018-03-17T14:31:28.001+0800 I STORAGE  [rsSync] clone test.students 4667519
2018-03-17T14:31:29.457+0800 I STORAGE  [rsSync] 4683761 objects cloned so far from collection test.students
2018-03-17T14:32:34.884+0800 I STORAGE  [rsSync] clone test.students 5103231
2018-03-17T14:32:34.884+0800 I STORAGE  [rsSync] 5103242 objects cloned so far from collection test.students
2018-03-17T14:33:46.017+0800 I STORAGE  [rsSync] 5452746 objects cloned so far from collection test.students
2018-03-17T14:33:46.017+0800 I STORAGE  [rsSync] clone test.students 5452799
2018-03-17T14:34:47.770+0800 I STORAGE  [rsSync] clone test.students 5872127
2018-03-17T14:34:47.770+0800 I STORAGE  [rsSync] 5872227 objects cloned so far from collection test.students
2018-03-17T14:35:52.885+0800 I STORAGE  [rsSync] 6291581 objects cloned so far from collection test.students
2018-03-17T14:35:52.885+0800 I STORAGE  [rsSync] clone test.students 6291583
2018-03-17T14:36:57.050+0800 I STORAGE  [rsSync] 6571235 objects cloned so far from collection test.students
2018-03-17T14:36:57.050+0800 I STORAGE  [rsSync] clone test.students 6571263
2018-03-17T14:37:59.514+0800 I STORAGE  [rsSync] 8248905 objects cloned so far from collection test.students
2018-03-17T14:37:59.515+0800 I STORAGE  [rsSync] clone test.students 8248959
2018-03-17T14:38:37.698+0800 I INDEX    [rsSync] build index on: test.students properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "test.students" }
2018-03-17T14:38:37.698+0800 I INDEX    [rsSync]         building index using bulk method; build may temporarily use up to 500 megabytes of RAM
2018-03-17T14:38:40.000+0800 I -        [rsSync]   Index Build: 1511900/12074974 12%
2018-03-17T14:38:43.000+0800 I -        [rsSync]   Index Build: 3571300/12074974 29%
2018-03-17T14:38:46.000+0800 I -        [rsSync]   Index Build: 5640600/12074974 46%
2018-03-17T14:38:49.001+0800 I -        [rsSync]   Index Build: 7599000/12074974 62%
2018-03-17T14:38:52.001+0800 I -        [rsSync]   Index Build: 9471000/12074974 78%
2018-03-17T14:38:55.001+0800 I -        [rsSync]   Index Build: 11341800/12074974 93%
2018-03-17T14:39:27.000+0800 I -        [rsSync]   Index: (2/3) BTree Bottom Up Progress: 8961500/12074974 74%
2018-03-17T14:39:30.222+0800 I INDEX    [rsSync]         done building bottom layer, going to commit
2018-03-17T14:39:31.465+0800 I NETWORK  [initandlisten] connection accepted from 192.168.2.138:50802 #8 (3 connections now open)
2018-03-17T14:39:31.960+0800 I COMMAND  [conn1] command local.replset.election command: replSetRequestVotes { replSetRequestVotes: 1, setName: "repl1", dryRun: false, term: 212, candidateIndex: 1, configVersion: 1, lastCommittedOp: { ts: Timestamp 1521264812000|7795, t: 193 } } keyUpdates:0 writeConflicts:0 numYields:0 reslen:63 locks:{ Global: { acquireCount: { r: 4, w: 2 } }, Database: { acquireCount: { r: 1, W: 2 } }, Collection: { acquireCount: { r: 1 } } } protocol:op_command 3219ms
2018-03-17T14:39:31.983+0800 I INDEX    [rsSync] build index done.  scanned 12074974 total records. 54 secs
2018-03-17T14:39:31.984+0800 I REPL     [rsSync] initial sync data copy, starting syncup
2018-03-17T14:39:31.984+0800 I REPL     [rsSync] oplog sync 1 of 3
2018-03-17T14:39:32.797+0800 I REPL     [ReplicationExecutor] syncing from: 192.168.2.138:27017
2018-03-17T14:39:33.749+0800 I REPL     [ReplicationExecutor] Member 192.168.2.138:27017 is now in state PRIMARY
2018-03-17T14:46:16.481+0800 I REPL [rsSync] oplog sync 2 of 3
2018-03-17T14:46:16.708+0800 I REPL [rsSync] initial sync building indexes
2018-03-17T14:46:16.708+0800 I REPL [rsSync] initial sync cloning indexes for : test
2018-03-17T14:46:17.236+0800 I STORAGE  [rsSync] copying indexes for: { name: "students", options: {} }
2018-03-17T14:46:19.079+0800 I REPL [rsSync] oplog sync 3 of 3
2018-03-17T14:46:19.083+0800 I REPL [rsSync] initial sync finishing up
2018-03-17T14:46:19.483+0800 I REPL [rsSync] initial sync done
2018-03-17T14:46:19.491+0800 I REPL [rsSync] initial sync succeeded after 1 attempt(s).
2018-03-17T14:46:19.491+0800 I REPL [ReplicationExecutor] transition to RECOVERING
2018-03-17T14:46:19.495+0800 I REPL [ReplicationExecutor] transition to SECONDARY

由日志中可以看到,同步花了很长时间,原因:

1.192.168.2.130降级前,复制集的同步是没有完成的,这导致了降级后，192.168.2.138并没有升级为primary,这样就造成了单点故障！
2.192.168.2.130删除数据重启后,192.168.2.138继续做delete同步操作,130的状态为startup2
3.192.168.2.138删除数据完毕后(由于没有同步完全,130上的数据文件被删除了,导致138上最终的数据不是最新的！),
  然后才成为primary

repl1:PRIMARY> db.students.find().count()
12074974

通过查询结果,可以看到数据不是最新的,新数据应该至于500w条，也证明了mongo复制集是异步的，降级的时候应该观察副本数据库的状态！！！

后面又做了一次删除,并跟踪了副本的同步情况,这次切换时间就变得很短了,并且文件大小下降的十分可观！

11. 再次查看storageSize,下降十分明显

repl1:PRIMARY> db.stats()
{
        "db" : "test",
        "collections" : 2,
        "objects" : 5376341,
        "avgObjSize" : 60.000029946017186,
        "dataSize" : 322580621,
        "storageSize" : 102256640,
        "numExtents" : 0,
        "indexes" : 2,
        "indexSize" : 54321152,
        "ok" : 1
}

12. 192.168.2.138降权

repl1:PRIMARY>rs.stepDown()
2018-03-17T02:51:29.554-0400 E QUERY    [thread1] Error: error doing query: failed: network error while attempting to run command 'replSetStepDown' on host '127.0.0.1:27017'  :
[email protected]/mongo/shell/db.js:135:1
[email protected]/mongo/shell/db.js:153:16
[email protected]/mongo/shell/utils.js:1202:12
@(shell):1:1

2018-03-17T02:51:29.652-0400 I NETWORK  [thread1] trying reconnect to 127.0.0.1:27017 (127.0.0.1) failed
2018-03-17T02:51:29.791-0400 I NETWORK  [thread1] reconnect 127.0.0.1:27017 (127.0.0.1) ok

13. 重复130的操作收缩138的空间

原文地址：https://www.cnblogs.com/chinesern/p/8602239.html

时间： 2024-11-07 17:13:21

Mongo实战之数据空洞的最佳实践的相关文章

Citrix虚拟化数据中心反病毒最佳实践

虚拟化巨头Citrix最近在Citrix技术专区的"技术论文"部分发表了一篇深思熟虑的文章,标题为"端点安全和防病毒最佳实践",概述了与安全厂商合作以购买正确的反恶意软件工具来保护虚拟机,应用程序和桌面的逐点入门知识.Citrix专注于四个具有挑战性的领域. 一.代理注册二.签名更新三.性能优化四.防病毒排除该博客介绍了Bitdefender完全遵循Citrix最佳实践准则的情况,涵盖了GravityZone虚拟化安全,该功能可为下一代基础架构提供安全性,包括软件

Elasticsearch+Mongo亿级别数据导入及查询实践

数据方案: 在Elasticsearch中通过code及time字段查询对应doc的mongo_id字段获得mongodb中的主键_id 通过获得id再进入mongodb进行查询 1,数据情况: 全部为股票及指数的分钟K线数据(股票代码区分度较高) Elasticsearch及mongodb都未分片且未优化参数配置 mongodb数据量: Elasticsearch数据量: 2,将数据从mongo源库导入Elasticsearch import time from pymongo impor

生产SparkStreaming数据零丢失最佳实践(含代码)

MySQL创建存储offset的表格 mysql> use test mysql> create table hlw_offset( topic varchar(32), groupid varchar(50), partitions int, fromoffset bigint, untiloffset bigint, primary key(topic,groupid,partitions) ); Maven依赖包 <scala.version>2.11.8</scala

kafka数据接口定义最佳实践

1 上游业务系统向kakfa推送的数据全部应为字符串类型,使用的时候转换成需要的类型,保证数据可以类型转换2 为了便于后续 kafka数据校对,建议把推送数据来自的数据库主键id,也一起推送过来(一般数据推送流程:业务过程 --> 插入DB(返回主键id) --> 推送kafka) 3 所有金额涉及到到计算全部使用 decimal 类型,不要使用double 4 小数的位数需要明确好原文地址:https://www.cnblogs.com/wooluwalker/p/12168066.h

决胜大数据时代：Hadoop&Yarn&Spark企业级最佳实践（8天完整版脱产式培训版本）

Hadoop.Yarn.Spark是企业构建生产环境下大数据中心的关键技术,也是大数据处理的核心技术,是每个云计算大数据工程师必修课. 课程简介大数据时代的精髓技术在于Hadoop.Yarn.Spark,是大数据时代公司和个人必须掌握和使用的核心内容. Hadoop.Yarn.Spark是Yahoo!.阿里淘宝等公司公认的大数据时代的三大核心技术,是大数据处理的灵魂,是云计算大数据时代的技术命脉之所在,以Hadoop.Yarn.Spark为基石构建起来云计算大数据中心广泛运行于Yahoo!.阿

决胜大数据时代：Hadoop&Yarn&Spark企业级最佳实践(3天)

Hadoop是云计算的事实标准软件框架,是云计算理念.机制和商业化的具体实现,是整个云计算技术学习中公认的核心和最具有价值内容. Yarn是目前公认的最佳的分布式集群资源管理框架: Mahout是目前数据挖掘领域的王者: 工业和信息化部电信研究院于2014年5月发布的“大数据白皮书”中指出: “2012 年美国联邦政府就在全球率先推出“大数据行动计划(Big data initiative)”,重点在基础技术研究和公共部门应用上加大投入.在该计划支持下,加州大学伯克利分校开发了完整

Hadoop大数据时代：Hadoop&YarnSpark企业级最佳实践 (4天)

Hadoop.Yarn.Spark是企业构建生产环境下大数据中心的关键技术,也是大数据处理的核心技术,是每个云计算大数据工程师必修课. 大数据时代的精髓技术在于Hadoop.Yarn.Spark,是大数据时代公司和个人必须掌握和使用的核心内容. Hadoop.Yarn.Spark是Yahoo!.阿里淘宝等公司公认的大数据时代的三大核心技术,是大数据处理的灵魂,是云计算大数据时代的技术命脉之所在,以Hadoop.Yarn.Spark为基石构建起来云计算大数据中心广泛运行于Yahoo!.阿里淘宝.腾

大数据Hadoop最佳实践(V3)

一:课程简介: Hadoop是当下云计算大数据的王者. Hadoop不仅是一个大数据的计算框架,同时也是大数据的存储平台. 使用Hadoop,用户可以在不了解分布式底层细节的情况下开发出分布式程序,从而可以使用众多廉价的计算设备的集群的威力来高速的运算和存储,而且Hadoop的运算和存储是可靠的.高效的.可伸缩的,能够使用普通的社区服务器出来PB级别的数据,是分布式大数据处理的存储的理想选择使用Hadoop可以主要完成: 1,构建离线处理平台,完成海量离线数据的存储分析,相对于传统的关系型数据

Scala 深入浅出实战经典第65讲：Scala中隐式转换内幕揭秘、最佳实践及其在Spark中的应用源码解析

王家林亲授<DT大数据梦工厂>大数据实战视频 Scala 深入浅出实战经典(1-87讲)完整视频.PPT.代码下载:百度云盘:http://pan.baidu.com/s/1c0noOt6 腾讯微云:http://url.cn/TnGbdC 360云盘:http://yunpan.cn/cQ4c2UALDjSKy 访问密码 45e2土豆:http://www.tudou.com/programs/view/NGgUD5FBQaA/优酷:http://v.youku.com/v_show/id_