使用elasticdump迁移数据到新es集群

参考文档 

https://blog.csdn.net/u012731379/article/details/79856113
https://blog.csdn.net/u010466329/article/details/79209236
https://blog.csdn.net/laoyang360/article/details/65449407

迁移方法

  • 通过logstash的input和output配置迁移（配置灵活适用于长期数据同步等）
  • 通过迁移工具如elasticdump等（适用于备份一次性小量数据操作）
  • 通过elasticsarch自带快照功能（适用于一次性迁移大量数据）

操作步骤

1、安装elasticdump

[[email protected]_8_24_centos ~]# yum install nodejs npm
[[email protected]_8_24_centos ~]# npm install elasticdump
/root
└─┬ [email protected]
  ├── [email protected]
  ├─┬ [email protected]
  │ ├─┬ [email protected]
  │ │ ├── [email protected]
  │ │ └── [email protected]
  │ ├── [email protected]1.1
  │ ├── [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ ├─┬ [email protected]
  │ │ └── [email protected]
  │ ├── [email protected]
  │ └─┬ [email protected]
  │   └── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├─┬ [email protected]
  │ ├── [email protected]
  │ └── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├─┬ [email protected]
  │ ├── [email protected]
  │ └── [email protected]
  ├─┬ [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ ├─┬ [email protected]
  │ │ └── [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ ├─┬ [email protected]
  │ │ └── [email protected]
  │ ├─┬ [email protected]
  │ │ ├─┬ [email protected]
  │ │ │ ├── [email protected]
  │ │ │ ├── [email protected]
  │ │ │ ├── [email protected]
  │ │ │ └─┬ [email protected]
  │ │ │   └── [email protected]
  │ │ └── [email protected]
  │ ├─┬ [email protected]
  │ │ ├── [email protected]
  │ │ ├─┬ [email protected]
  │ │ │ ├── [email protected]
  │ │ │ ├── [email protected]
  │ │ │ └── [email protected]
  │ │ └─┬ [email protected]
  │ │   ├── [email protected]
  │ │   ├── [email protected]
  │ │   ├── [email protected]
  │ │   ├── [email protected]
  │ │   ├── [email protected]
  │ │   ├── [email protected]
  │ │   ├── [email protected]
  │ │   └── [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ ├─┬ [email protected]
  │ │ └── [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ ├─┬ [email protected]
  │ │ ├── [email protected]
  │ │ └── [email protected]
  │ └── [email protected]
  ├─┬ [email protected]
  │ └── [email protected]
  └─┬ [email protected]
    ├── [email protected]
    └─┬ [email protected]
      ├── [email protected]
      ├── [email protected]
      ├── [email protected]
      ├── [email protected]
      └── [email protected] 

npm WARN enoent ENOENT: no such file or directory, open ‘/root/package.json‘
npm WARN root No description
npm WARN root No repository field.
npm WARN root No README data
npm WARN root No license field.
[[email protected]_8_24_centos ~]# cd node_modules/elasticdump/bin

2、导出mapping

[[email protected]_8_24_centos bin]# ./elasticdump --input=http://10.2.3.159:9200/reconciliation  --output=http://10.2.100.24:9200/reconciliationonline --type=mapping
Mon, 11 Feb 2019 09:28:10 GMT | starting dump
Mon, 11 Feb 2019 09:28:10 GMT | got 1 objects from source elasticsearch (offset: 0)
Mon, 11 Feb 2019 09:28:16 GMT | sent 1 objects to destination elasticsearch, wrote 1
Mon, 11 Feb 2019 09:28:16 GMT | got 0 objects from source elasticsearch (offset: 1)
Mon, 11 Feb 2019 09:28:16 GMT | Total Writes: 1
Mon, 11 Feb 2019 09:28:16 GMT | dump complete
[[email protected]_8_24_centos bin]# curl  10.2.100.24:9200/_cat/indices|grep reconciliation
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 43 32578   43 14037    0     0   5694      0  0:00:05  0:00:02  0:00:03  5694yellow open reconciliationonline   KnZrsU-7TmWWBs-1-dBRrw 5 1       0  0    1.1kb    1.1kb

3、导出data

[[email protected]_8_24_centos bin]# ./elasticdump --input=http://10.2.3.159:9200/reconciliation  --output=http://10.2.100.24:9200/reconciliationonline --type=data
Mon, 11 Feb 2019 09:28:58 GMT | starting dump
Mon, 11 Feb 2019 09:28:58 GMT | got 100 objects from source elasticsearch (offset: 0)
Mon, 11 Feb 2019 09:29:19 GMT | sent 100 objects to destination elasticsearch, wrote 100
Mon, 11 Feb 2019 09:29:19 GMT | got 100 objects from source elasticsearch (offset: 100)
Mon, 11 Feb 2019 09:29:51 GMT | sent 100 objects to destination elasticsearch, wrote 100
Mon, 11 Feb 2019 09:29:51 GMT | got 100 objects from source elasticsearch (offset: 200)
Mon, 11 Feb 2019 09:30:13 GMT | sent 100 objects to destination elasticsearch, wrote 100
Mon, 11 Feb 2019 09:30:13 GMT | got 100 objects from source elasticsearch (offset: 300)
Mon, 11 Feb 2019 09:30:39 GMT | sent 100 objects to destination elasticsearch, wrote 100
Mon, 11 Feb 2019 09:30:39 GMT | got 100 objects from source elasticsearch (offset: 400)
Mon, 11 Feb 2019 09:31:53 GMT | sent 100 objects to destination elasticsearch, wrote 100
Mon, 11 Feb 2019 09:31:53 GMT | got 100 objects from source elasticsearch (offset: 500)
Mon, 11 Feb 2019 09:32:28 GMT | sent 100 objects to destination elasticsearch, wrote 100
Mon, 11 Feb 2019 09:32:28 GMT | got 100 objects from source elasticsearch (offset: 600)
Mon, 11 Feb 2019 09:33:24 GMT | sent 100 objects to destination elasticsearch, wrote 100
Mon, 11 Feb 2019 09:33:24 GMT | got 100 objects from source elasticsearch (offset: 700)
Mon, 11 Feb 2019 09:34:19 GMT | sent 100 objects to destination elasticsearch, wrote 100
Mon, 11 Feb 2019 09:34:19 GMT | got 7 objects from source elasticsearch (offset: 800)
Mon, 11 Feb 2019 09:34:47 GMT | sent 7 objects to destination elasticsearch, wrote 7
Mon, 11 Feb 2019 09:34:47 GMT | got 0 objects from source elasticsearch (offset: 807)
Mon, 11 Feb 2019 09:34:47 GMT | Total Writes: 807
Mon, 11 Feb 2019 09:34:47 GMT | dump complete
[[email protected]_8_24_centos bin]# curl  10.2.100.24:9200/_cat/indices|grep reconciliationonline
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:16 --:--:--     0yellow open reconciliationonline   KnZrsU-7TmWWBs-1-dBRrw 5 1     807  0    4.3mb    4.3mb
100 32578  100 32578    0     0   2010      0  0:00:16  0:00:16 --:--:--  7769

原文地址：http://blog.51cto.com/jerrymin/2349462

时间： 2024-08-29 21:10:10

使用elasticdump迁移数据到新es集群的相关文章

vertica从其他表迁移数据到新表（insert into 语句用法实例）

前面一篇开始学习solr的时候,做了个入门的示例http://blog.csdn.net/zjc/article/details/24414271 .虽然可以检索出内容,但总和想象的结果有差异--比如,检索"天龙"两个字,按常规理解,就应该只出来<天龙八部>才对,可是竟然也会把<倚天屠龙记>检出来.后来研究了一下,发现系统是这样处理的:无论是抽索引时还是分析检索词时,都把所有文字按单字拆开.这样,刚好<倚天屠龙记>里包含"天"和&

【源】从零自学Hadoop(17)：Hive数据导入导出，集群数据迁移下

阅读目录 序 将查询的结果写入文件系统 集群数据迁移一 集群数据迁移二 系列索引 本文版权归mephisto和博客园共有,欢迎转载,但须保留此段声明,并给出原文链接,谢谢合作. 文章是哥(mephisto)写的,SourceLink 序 上一篇,我们介绍了Hive的数据多种方式导入,这样我们的Hive就有了数据来源了,但有时候我们可能需要纯粹的导出,或者集群Hive数据的迁移(不同集群,不同版本),我们就可以通过这两章的知识来实现.   下面我们开始介绍hive的数据导出,以及集群Hive数据的

elasticsearch(es) 集群恢复触发配置（Local Gateway参数）

elasticsearch(es) 集群恢复触发配置(Local Gateway) 当你集群重启时,几个配置项影响你的分片恢复的表现. 首先,我们需要明白如果什么也没配置将会发生什么. 想象一下假设你有 10 个节点,每个节点只保存一个分片,这个分片是一个主分片或者是一个副本分片,或者说有一个有 5 个主分片/1 个副本分片的索引.有时你需要为整个集群做离线维护(比如,为了安装一个新的驱动程序), 当你重启你的集群,恰巧出现了 5 个节点已经启动,还有 5 个还没启动的场景. 假设其它 5 个节

ES集群修改index副本数报错 ：index read-only / allow delete

ES集群修改index副本数,报错 :index read-only / allow delete (api) 原因: es集群数据量增速过快,导致个别es node节点磁盘使用率在%80以上,接近%90 ,由于ES新节点的数据目录data存储空间不足,导致从master主节点接收同步数据的时候失败,此时ES集群为了保护数据,会自动把索引分片index置为只读read-only. 故障处理办法: 1:集群加节点,简单粗暴: 2:降低集群index副本数量: 3:其它:增加磁盘.删除历史数据等:

ES集群部署及调优

系统:Centos6ES版本:6.4.0服务器三台172.16.0.8172.16.0.6172.16.0.22 部署jdk解压jdk放在/data目录,/data/jdk配置环境变量,/etc/proifle里面加入如下 export JAVA_HOME=/data/jdk export PATH=$PATH:$JAVA_HOME/bin export CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar:$CLASSPATH s

ES集群性能调优链接汇总

ES集群稳定性: 1. 集群稳定性的一些问题(一定量数据后集群变得迟钝) https://elasticsearch.cn/question/84 2.ELK 性能(2) - 如何在大业务量下保持 Elasticsearch 集群的稳定 http://www.cnblogs.com/richaaaard/p/6117089.html

ES 集群关键状态指标

ES监控状态指标分三个级别: 1:集群级别:集群级别的监控主要是针对整个ES集群来说,包括集群的健康状况.集群的状态等.2:节点级别:节点级别的监控主要是针对每个ES实例的监控,其中包括每个实例的查询索引指标和物理资源使用指标.3:索引级别:索引级别的监控主要是针对每个索引来说,主要包括每个索引的性能指标. 1集群级别: 查看方法: api获取:http://ip:9200/_cluster/health?pretty 或者 Kibana的开发工具Dev Tools中执行 : 查看集群健康状态

ELasticSearch（五）ES集群原理与搭建

一.ES集群原理 查看集群健康状况:URL+ /GET _cat/health (1).ES基本概念名词 Cluster 代表一个集群,集群中有多个节点,其中有一个为主节点,这个主节点是可以通过选举产生的,主从节点是对于集群内部来说的.es的一个概念就是去中心化,字面上理解就是无中心节点,这是对于集群外部来说的,因为从外部来看es集群,在逻辑上是个整体,你与任何一个节点的通信和与整个es集群通信是等价的. Shards 代表索引分片,es可以把一个完整的索引分成多个分片,这样的好处是可以把一个大

ELK简介 es集群部署 es插件应用

Top NSD ARCHITECTURE DAY03 案例1:ES集群安装 案例2:ES集群安装配置 案例3:练习curl命令 案例4:练习插件 案例5:插入,增加,删除查询数据 案例6:安装Kibana 1 案例1:ES集群安装 1.1 问题 本案例要求: 准备1台虚拟机 部署elasticsearch第一个节点 访问9200端口查看是否安装成功 1.2 方案 1)ELK是日志分析平台,不是一款软件,而是一整套解决方案,是三个软件产品的首字母缩写,ELK分别代表: Elasticsearch: