Docker 搭建Spark 依赖singularities/spark镜像

拉取镜像：

[[email protected] docker-spark-2.1.0]# docker pull singularities/spark

查看：

[[email protected] docker-spark-2.1.0]# docker image ls
REPOSITORY                      TAG                 IMAGE ID            CREATED             SIZE
docker.io/singularities/spark   latest              84222b254621        6 months ago        1.39 GB

创建docker-compose.yml文件

[[email protected] home]# mkdir singularitiesCR
[[email protected] home]# cd singularitiesCR
[[email protected] singularitiesCR]# touch docker-compose.yml

内容：

version: "2"

services:
  master:
    image: singularities/spark
    command: start-spark master
    hostname: master
    ports:
      - "6066:6066"
      - "7070:7070"
      - "8080:8080"
      - "50070:50070"
  worker:
    image: singularities/spark
    command: start-spark worker master
    environment:
      SPARK_WORKER_CORES: 1
      SPARK_WORKER_MEMORY: 2g
    links:
      - master

执行docker-compose up即可启动一个单工作节点的standlone模式下运行的spark集群

查看容器：

[[email protected] singularitiesCR]# docker-compose ps
          Name                      Command            State                                             Ports
--------------------------------------------------------------------------------------------------------------------------------------------------------
singularitiescr_master_1   start-spark master          Up      10020/tcp, 13562/tcp, 14000/tcp, 19888/tcp, 50010/tcp, 50020/tcp,
                                                               0.0.0.0:50070->50070/tcp, 50075/tcp, 50090/tcp, 50470/tcp, 50475/tcp,
                                                               0.0.0.0:6066->6066/tcp, 0.0.0.0:7070->7070/tcp, 7077/tcp, 8020/tcp,
                                                               0.0.0.0:8080->8080/tcp, 8081/tcp, 9000/tcp
singularitiescr_worker_1   start-spark worker master   Up      10020/tcp, 13562/tcp, 14000/tcp, 19888/tcp, 50010/tcp, 50020/tcp, 50070/tcp, 50075/tcp,
                                                               50090/tcp, 50470/tcp, 50475/tcp, 6066/tcp, 7077/tcp, 8020/tcp, 8080/tcp, 8081/tcp,
                                                               9000/tcp           

查看结果：

参考：

https://github.com/SingularitiesCR/spark-docker

https://blog.csdn.net/u013705066/article/details/80030732

原文地址：https://www.cnblogs.com/hongdada/p/9475406.html