spark 实战 1： 基于gettyimages spark docker image 创建spark 集群

1、首先下载该镜像到本地。https://hub.docker.com/r/gettyimages/spark/

~$ docker pull gettyimages/spark

2、从https://github.com/gettyimages/docker-spark/blob/master/docker-compose.yml下载编写好的支持 Spark 集群的 docker-compose.yml 文件

启动它

$ docker-compose up

$ docker-compose up

Creating spark_master_1

Creating spark_worker_1

Attaching to spark_master_1, spark_worker_1

master_1  | 16/10/29 13:25:05 INFO master.Master: Started daemon with process name: [email protected]

master_1  | 16/10/29 13:25:05 INFO util.SignalUtils: Registered signal handler for TERM

master_1  | 16/10/29 13:25:05 INFO util.SignalUtils: Registered signal handler for HUP

master_1  | 16/10/29 13:25:05 INFO util.SignalUtils: Registered signal handler for INT

master_1  | 16/10/29 13:25:06 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

master_1  | 16/10/29 13:25:06 INFO spark.SecurityManager: Changing view acls to: root

master_1  | 16/10/29 13:25:06 INFO spark.SecurityManager: Changing modify acls to: root

master_1  | 16/10/29 13:25:06 INFO spark.SecurityManager: Changing view acls groups to:

master_1  | 16/10/29 13:25:06 INFO spark.SecurityManager: Changing modify acls groups to:

master_1  | 16/10/29 13:25:06 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()

worker_1  | 16/10/29 13:25:06 INFO worker.Worker: Started daemon with process name: [email protected]

worker_1  | 16/10/29 13:25:06 INFO util.SignalUtils: Registered signal handler for TERM

worker_1  | 16/10/29 13:25:06 INFO util.SignalUtils: Registered signal handler for HUP

worker_1  | 16/10/29 13:25:06 INFO util.SignalUtils: Registered signal handler for INT

master_1  | 16/10/29 13:25:07 INFO util.Utils: Successfully started service ‘sparkMaster‘ on port 7077.

master_1  | 16/10/29 13:25:07 INFO master.Master: Starting Spark master at spark://master:7077

master_1  | 16/10/29 13:25:07 INFO master.Master: Running Spark version 2.0.1

master_1  | 16/10/29 13:25:07 INFO util.log: Logging initialized @2542ms

master_1  | 16/10/29 13:25:07 INFO server.Server: jetty-9.2.z-SNAPSHOT

master_1  | 16/10/29 13:25:07 INFO handler.ContextHandler: Started [email protected]{/app,null,AVAILABLE}

master_1  | 16/10/29 13:25:07 INFO handler.ContextHandler: Started [email protected]{/app/json,null,AVAILABLE}

master_1  | 16/10/29 13:25:07 INFO handler.ContextHandler: Started [email protected]{/,null,AVAILABLE}

master_1  | 16/10/29 13:25:07 INFO handler.ContextHandler: Started [email protected]{/json,null,AVAILABLE}

master_1  | 16/10/29 13:25:07 INFO handler.ContextHandler: Started [email protected]{/static,null,AVAILABLE}

master_1  | 16/10/29 13:25:07 INFO handler.ContextHandler: Started [email protected]{/app/kill,null,AVAILABLE}

master_1  | 16/10/29 13:25:07 INFO handler.ContextHandler: Started [email protected]{/driver/kill,null,AVAILABLE}

master_1  | 16/10/29 13:25:07 INFO server.ServerConnector: Started [email protected]{HTTP/1.1}{0.0.0.0:8080}

master_1  | 16/10/29 13:25:07 INFO server.Server: Started @2742ms

master_1  | 16/10/29 13:25:07 INFO util.Utils: Successfully started service ‘MasterUI‘ on port 8080.

master_1  | 16/10/29 13:25:07 INFO ui.MasterWebUI: Bound MasterWebUI to 0.0.0.0, and started at http://localhost:8080

master_1  | 16/10/29 13:25:07 INFO server.Server: jetty-9.2.z-SNAPSHOT

master_1  | 16/10/29 13:25:07 INFO handler.ContextHandler: Started [email protected]{/,null,AVAILABLE}

master_1  | 16/10/29 13:25:07 INFO server.ServerConnector: Started [email protected]{HTTP/1.1}{master:6066}

master_1  | 16/10/29 13:25:07 INFO server.Server: Started @2779ms

master_1  | 16/10/29 13:25:07 INFO util.Utils: Successfully started service on port 6066.

master_1  | 16/10/29 13:25:07 INFO rest.StandaloneRestServer: Started REST server for submitting applications on port 6066

worker_1  | 16/10/29 13:25:07 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

worker_1  | 16/10/29 13:25:08 INFO spark.SecurityManager: Changing view acls to: root

worker_1  | 16/10/29 13:25:08 INFO spark.SecurityManager: Changing modify acls to: root

worker_1  | 16/10/29 13:25:08 INFO spark.SecurityManager: Changing view acls groups to:

worker_1  | 16/10/29 13:25:08 INFO spark.SecurityManager: Changing modify acls groups to:

worker_1  | 16/10/29 13:25:08 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()

master_1  | 16/10/29 13:25:08 INFO handler.ContextHandler: Started [email protected]{/metrics/master/json,null,AVAILABLE}

master_1  | 16/10/29 13:25:08 INFO handler.ContextHandler: Started [email protected]{/metrics/applications/json,null,AVAILABLE}

master_1  | 16/10/29 13:25:08 INFO master.Master: I have been elected leader! New state: ALIVE

worker_1  | 16/10/29 13:25:08 INFO util.Utils: Successfully started service ‘sparkWorker‘ on port 8881.

worker_1  | 16/10/29 13:25:08 INFO worker.Worker: Starting Spark worker 172.17.0.3:8881 with 2 cores, 1024.0 MB RAM

worker_1  | 16/10/29 13:25:08 INFO worker.Worker: Running Spark version 2.0.1

worker_1  | 16/10/29 13:25:08 INFO worker.Worker: Spark home: /usr/spark-2.0.1

worker_1  | 16/10/29 13:25:08 INFO util.log: Logging initialized @2625ms

worker_1  | 16/10/29 13:25:08 INFO server.Server: jetty-9.2.z-SNAPSHOT

worker_1  | 16/10/29 13:25:08 INFO handler.ContextHandler: Started [email protected]{/logPage,null,AVAILABLE}

worker_1  | 16/10/29 13:25:08 INFO handler.ContextHandler: Started [email protected]{/logPage/json,null,AVAILABLE}

worker_1  | 16/10/29 13:25:08 INFO handler.ContextHandler: Started [email protected]{/,null,AVAILABLE}

worker_1  | 16/10/29 13:25:08 INFO handler.ContextHandler: Started [email protected]{/json,null,AVAILABLE}

worker_1  | 16/10/29 13:25:08 INFO handler.ContextHandler: Started [email protected]{/static,null,AVAILABLE}

worker_1  | 16/10/29 13:25:08 INFO handler.ContextHandler: Started [email protected]{/log,null,AVAILABLE}

worker_1  | 16/10/29 13:25:08 INFO server.ServerConnector: Started [email protected]{HTTP/1.1}{0.0.0.0:8081}

worker_1  | 16/10/29 13:25:08 INFO server.Server: Started @2749ms

worker_1  | 16/10/29 13:25:08 INFO util.Utils: Successfully started service ‘WorkerUI‘ on port 8081.

worker_1  | 16/10/29 13:25:08 INFO ui.WorkerWebUI: Bound WorkerWebUI to 0.0.0.0, and started at http://localhost:8081

worker_1  | 16/10/29 13:25:08 INFO worker.Worker: Connecting to master master:7077...

worker_1  | 16/10/29 13:25:08 INFO handler.ContextHandler: Started [email protected]{/metrics/json,null,AVAILABLE}

worker_1  | 16/10/29 13:25:08 INFO client.TransportClientFactory: Successfully created connection to master/172.17.0.2:7077 after 41 ms (0 ms spent in bootstraps)

master_1  | 16/10/29 13:25:09 INFO master.Master: Registering worker 172.17.0.3:8881 with 2 cores, 1024.0 MB RAM

worker_1  | 16/10/29 13:25:09 INFO worker.Worker: Successfully registered with master spark://master:7077

时间： 2024-10-04 16:33:59

spark 实战 1： 基于gettyimages spark docker image 创建spark 集群的相关文章

一步一步教你搭建基于docker的MongoDB复制集群环境

一步一步教你搭建基于docker的MongoDB复制集群环境 1.安装docker 2.创建MongoDB的Image 3.搭建MongoDB的集群 Docker 是一个开源的应用容器引擎,让开发者可以打包他们的应用以及依赖包到一个可移植的容器中. 1.ubuntu14.04安装docker 参考文档 http://docs.docker.com/installation/ubuntulinux/ 参考文档 http://docs.docker.com/mac/started/ pc@pc-Th

基于docker环境搭建kafka集群（三台真机之间）

环境基于docker,三台物理主机192.168.0.27.192.168.0.28.192.168.0.29,每台主机部署一个zookeeper节点,一个kafka节点,共三个zookeeper节点,三个kafka节点,容器之间的网络采用host模式 1.拉取镜像 2.启动容器 step1. 创建配置文件zoo.cfg并替换容器内的该文件,不同的镜像有可能zoo.cfg位置不同 # The number of milliseconds of each tick tickTime=2000 #

CentOS7.X环境下基于docker安装部署RabbitMQ集群

1.IP地址规划(将信息配置到/etc/hosts中)主机名 IP地址RabbitMQ01 192.168.8.131RabbitMQ02 192.168.8.132RabbitMQ03 192.168.8.133RabbitMQ04 192.168.8.1342.RabbitMQ集群安装(1)四个节点同时运行,下载RabbitMQ镜像[[email protected]~]# docker pull rabbitmq:3-management(2)四个节点分别运行,启动RabbitMQ容器 [

基于keepalived搭建MySQL的高可用集群

http://www.cnblogs.com/ivictor/p/5522383.html 基于keepalived搭建MySQL的高可用集群 MySQL的高可用方案一般有如下几种: keepalived+双主,MHA,MMM,Heartbeat+DRBD,PXC,Galera Cluster 比较常用的是keepalived+双主,MHA和PXC. 对于小公司,一般推荐使用keepalived+双主,简单. 下面来部署一下 配置环境: 角色                          

基于keepalived对HAproxy做高可用集群

一.Keepalived简介 Keepalived的作用是检测web服务器的状态,如果有一台web服务器死机,或工作出现故障,Keepalived将检测到,并将有故障的web服务器从系统中剔除,当web服务器工作正常后Keepalived自动将web服务器加入到服务器群中,这些工作全部自动完成,不需要人工干涉,需要人工做的只是修复故障的web服务器. Layer3,4&7工作在IP/TCP协议栈的IP层,TCP层,及应用层,原理分别如下: Layer3:Keepalived使用Layer3的方式

基于drbd的mariaDB 的高可用集群

Distributed Replicated Block Device(DRBD)是一个用软件实现的.无共享的.服务器之间镜像块设备内容的存储复制解决方案. 数据镜像:实时.透明.同步(所有服务器都成功后返回).异步(本地服务器成功后返回) DRBD的核心功能通过Linux的内核实现,最接近系统的IO栈,但它不能神奇地添加上层的功能比如检测到EXT3文件系统的崩溃. 在DRBD中,资源是特指某复制的存储设备的所有方面.包括资源名称.DRBD设备(/dev/drbdm,这里m是设备最小号,最大号可

利用docker镜像配置mysql集群+nextcloud集群+haproxy负载均衡

测试环境: docker xampp 9.1.1 ubuntu 16.0.4 hadoop 2.7 jdk 1.8 一.配置mysql集群 通过docker拉取mysql集群镜像创建容器,包括ndb_mgm(管理节点).ndb_mgmd01.ndbd01(数据节点1).ndbd02(数据节点2).mysqld01(sql节点1).mysqld02(sql节点2) docker run -itd --name ndb_mgmd01 --net=scg --ip 192.166.0.2 -v /ro

基于RHCS的web双机热备集群搭建

基于RHCS的web双机热备集群搭建 RHCS集群运行原理及功能介绍 1. 分布式集群管理器(CMAN) Cluster Manager,简称CMAN,是一个分布式集群管理工具,它运行在集群的各个节点上,为RHCS提供集群管理任务.CMAN用于管理集群成员.消息和通知.它通过监控每个节点的运行状态来了解节点成员之间的关系,当集群中某个节点出现故障,节点成员关系将发生改变,CMAN及时将这种改变通知底层,进而做出相应的调整. 2.锁管理(DLM) Distributed Lock Manager,

Docker 容器部署 Consul 集群

一.docker安装与启动1.1安装docker[[email protected] /]# yum -y install docker-io 1.2更改配置文件[[email protected] /]# vi /etc/sysconfig/dockerother-args列更改为:other_args="--exec-driver=lxc --selinux-enabled" 1.3启动docker服务[[email protected] /]# service docker st

手把手教你用Docker部署一个MongoDB集群

MongoDB是一个介于关系数据库和非关系数据库之间的产品,是非关系数据库当中最像关系数据库的.支持类似于面向对象的查询语言,几乎可以实现类似关系数据库单表查询的绝大部分功能,而且还支持对数据建立索引.本文介绍了如何使用Docker搭建MongoDB集群. 本文我会向大家介绍如何使用Docker部署一个MongoDB集群,具体如下: 2.6.5版本的MongoDB 有3个节点的副本集(Replica set) 身份验证 持久化数据到本地文件系统 首先要准备三个运行的Docker服务器,这意味着你