Learn ZYNQ (9)

创建zybo cluster的spark集群（计算层面）：

1.每个节点都是同样的filesystem，mac地址冲突，故：

vi ./etc/profile

export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$PATH

export JAVA_HOME=/usr/lib/jdk1.7.0_55

export CLASSPATH=.:$JAVA_HOME/lib/tools.jar

export PATH=$JAVA_HOME/bin:$PATH

export HADOOP_HOME=/root/hadoop-2.4.0

ifconfig eth1 hw ether 00:0a:35:00:01:03

ifconfig eth1 192.168.1.3/24 up

 

2.生成私匙 id_rsa 与 公匙 id_rsa.pub 配置文件

ssh-keygen -t rsa

id_rsa是密钥文件，id_rsa.pub是公钥文件。

 

3.Worker节点/etc/hosts配置：

具体操作步骤：

ssh [email protected]

vi /etc/hosts

127.0.0.1 localhost zynq

192.168.1.1 spark1

192.168.1.2 spark2

192.168.1.3 spark3

192.168.1.4 spark4

192.168.1.5 spark5

192.168.1.100 sparkMaster

#::1 localhost ip6-localhost ip6-loopback

Master节点/etc/hosts配置：

 

4.分发公钥

ssh-copy-id -i ~/.ssh/id_rsa.pub [email protected]

ssh-copy-id -i ~/.ssh/id_rsa.pub [email protected]

ssh-copy-id -i ~/.ssh/id_rsa.pub [email protected]

ssh-copy-id -i ~/.ssh/id_rsa.pub [email protected]

…..

 

5.配置Master节点

Cd ~/spark-0.9.1-bin-hadoop2/conf

Vi slaves

 

6.配置java

否则运行pi计算时会出现count找不到的错误（因为pyspark找不到javaruntime）。

cd /usr/bin/

ln -s /usr/lib/jdk1.7.0_55/bin/java java

ln -s /usr/lib/jdk1.7.0_55/bin/javac javac

ln -s /usr/lib/jdk1.7.0_55/bin/jar jar

 

 

7.测试运行所有节点

SPARK_MASTER_IP=192.168.1.1 ./sbin/start-all.sh

SPARK_MASTER_IP=192.168.1.100 ./sbin/start-all.sh

成功启动所有节点：

 

8.查看工作状态：

Jps

Netstat -ntlp

 

9.开启脚本命令行

MASTER=spark://192.168.1.1:7077 ./bin/pyspark

MASTER=spark://192.168.1.100:7077 ./bin/pyspark

 

10.测试

from random import random

def sample(p):

x, y = random(), random()

return 1 if x*x + y*y < 1 else 0

count = sc.parallelize(xrange(0, 1000000)).map(sample) \

.reduce(lambda a, b: a + b)

print "Pi is roughly %f" % (4.0 * count / 1000000)

 

成功进行运算：

 

正常启动信息：

[email protected]:~/spark-0.9.1-bin-hadoop2# MASTER=spark://192.168.1.1:7077 ./bin/pyspark

Python 2.7.4 (default, Apr 19 2013, 19:49:55)

[GCC 4.7.3] on linux2

Type "help", "copyright", "credits" or "license" for more information.

log4j:WARN No appenders could be found for logger (akka.event.slf4j.Slf4jLogger).

log4j:WARN Please initialize the log4j system properly.

log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

70/01/01 00:07:48 INFO SparkEnv: Using Spark‘s default log4j profile: org/apache/spark/log4j-defaults.properties

70/01/01 00:07:48 INFO SparkEnv: Registering BlockManagerMaster

70/01/01 00:07:49 INFO DiskBlockManager: Created local directory at /tmp/spark-local-19700101000749-e1fb

70/01/01 00:07:49 INFO MemoryStore: MemoryStore started with capacity 297.0 MB.

70/01/01 00:07:49 INFO ConnectionManager: Bound socket to port 36414 with id = ConnectionManagerId(spark1,36414)

70/01/01 00:07:49 INFO BlockManagerMaster: Trying to register BlockManager

70/01/01 00:07:49 INFO BlockManagerMasterActor$BlockManagerInfo: Registering block manager spark1:36414 with 297.0 MB RAM

70/01/01 00:07:49 INFO BlockManagerMaster: Registered BlockManager

70/01/01 00:07:49 INFO HttpServer: Starting HTTP Server

70/01/01 00:07:50 INFO HttpBroadcast: Broadcast server started at http://192.168.1.1:42068

70/01/01 00:07:50 INFO SparkEnv: Registering MapOutputTracker

70/01/01 00:07:50 INFO HttpFileServer: HTTP File server directory is /tmp/spark-77996902-7ea4-4161-bc23-9f3538967c17

70/01/01 00:07:50 INFO HttpServer: Starting HTTP Server

70/01/01 00:07:51 INFO SparkUI: Started Spark Web UI at http://spark1:4040

70/01/01 00:07:52 INFO AppClient$ClientActor: Connecting to master spark://192.168.1.1:7077...

70/01/01 00:07:55 INFO SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-19700101000755-0001

70/01/01 00:07:55 INFO AppClient$ClientActor: Executor added: app-19700101000755-0001/0 on worker-19700101000249-spark2-53901 (spark2:53901) with 2 cores

70/01/01 00:07:55 INFO SparkDeploySchedulerBackend: Granted executor ID app-19700101000755-0001/0 on hostPort spark2:53901 with 2 cores, 512.0 MB RAM

70/01/01 00:07:55 INFO AppClient$ClientActor: Executor added: app-19700101000755-0001/1 on worker-19700101000306-spark5-38532 (spark5:38532) with 2 cores

70/01/01 00:07:55 INFO SparkDeploySchedulerBackend: Granted executor ID app-19700101000755-0001/1 on hostPort spark5:38532 with 2 cores, 512.0 MB RAM

70/01/01 00:07:55 INFO AppClient$ClientActor: Executor added: app-19700101000755-0001/2 on worker-19700101000255-spark3-41536 (spark3:41536) with 2 cores

70/01/01 00:07:55 INFO SparkDeploySchedulerBackend: Granted executor ID app-19700101000755-0001/2 on hostPort spark3:41536 with 2 cores, 512.0 MB RAM

70/01/01 00:07:55 INFO AppClient$ClientActor: Executor added: app-19700101000755-0001/3 on worker-19700101000254-spark4-38766 (spark4:38766) with 2 cores

70/01/01 00:07:55 INFO SparkDeploySchedulerBackend: Granted executor ID app-19700101000755-0001/3 on hostPort spark4:38766 with 2 cores, 512.0 MB RAM

70/01/01 00:07:55 INFO AppClient$ClientActor: Executor updated: app-19700101000755-0001/0 is now RUNNING

70/01/01 00:07:55 INFO AppClient$ClientActor: Executor updated: app-19700101000755-0001/3 is now RUNNING

70/01/01 00:07:55 INFO AppClient$ClientActor: Executor updated: app-19700101000755-0001/1 is now RUNNING

70/01/01 00:07:55 INFO AppClient$ClientActor: Executor updated: app-19700101000755-0001/2 is now RUNNING

70/01/01 00:07:56 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Welcome to

____ __

/ __/__ ___ _____/ /__

_\ \/ _ \/ _ `/ __/ ‘_/

/__ / .__/\_,_/_/ /_/\_\ version 0.9.1

/_/

Using Python version 2.7.4 (default, Apr 19 2013 19:49:55)

Spark context available as sc.

>>> 70/01/01 00:08:06 INFO SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://[email protected]:35842/user/Executor#1876589543] with ID 2

70/01/01 00:08:11 INFO BlockManagerMasterActor$BlockManagerInfo: Registering block manager spark3:42847 with 297.0 MB RAM

70/01/01 00:08:12 INFO SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://[email protected]:43445/user/Executor#-1199017431] with ID 1

70/01/01 00:08:13 INFO BlockManagerMasterActor$BlockManagerInfo: Registering block manager spark5:42630 with 297.0 MB RAM

70/01/01 00:08:15 INFO AppClient$ClientActor: Executor updated: app-19700101000755-0001/0 is now FAILED (Command exited with code 1)

70/01/01 00:08:15 INFO SparkDeploySchedulerBackend: Executor app-19700101000755-0001/0 removed: Command exited with code 1

70/01/01 00:08:15 INFO AppClient$ClientActor: Executor added: app-19700101000755-0001/4 on worker-19700101000249-spark2-53901 (spark2:53901) with 2 cores

70/01/01 00:08:15 INFO SparkDeploySchedulerBackend: Granted executor ID app-19700101000755-0001/4 on hostPort spark2:53901 with 2 cores, 512.0 MB RAM

70/01/01 00:08:15 INFO AppClient$ClientActor: Executor updated: app-19700101000755-0001/4 is now RUNNING

70/01/01 00:08:21 INFO SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://[email protected]:41692/user/Executor#-1994427913] with ID 3

70/01/01 00:08:26 INFO BlockManagerMasterActor$BlockManagerInfo: Registering block manager spark4:49788 with 297.0 MB RAM

70/01/01 00:08:27 INFO SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://[email protected]:44449/user/Executor#-1155287434] with ID 4

70/01/01 00:08:28 INFO BlockManagerMasterActor$BlockManagerInfo: Registering block manager spark2:38675 with 297.0 MB RAM

Learn ZYNQ (9)

时间： 2024-12-20 22:10:16

Learn ZYNQ (9)的相关文章

Learn ZYNQ(2)

AXI HP接口的DMA+GIC编程(参照博客) 参照文档:UG873,博客文档 我的Vivado+SDK工程文件打包(60+M) 我的DMA驱动程序(未完成) Vivado 接线图: 地址分配: Learn ZYNQ(2),布布扣,bubuko.com

Learn ZYNQ (3)

移植android3.3到ZedBoard follow doc:Android移植Guide1.3.pdf follow website: sudo apt-get install git gnupg flex bison gperf build-essential zip curl libc6-dev libncurses5-dev:i386 x11proto-core-dev libx11-dev:i386 libreadline6-dev:i386 libgl1-mesa-glx:i38

Learn ZYNQ (8)

在zed的PS端运行spark: (1)设置uboot为sd卡启动rootfs: "sdboot=if mmcinfo; then " \                         "run uenvboot; " \                         "echo Copying Linux from SD to RAM... && " \                         "fatlo

Learn ZYNQ(10) &ndash; zybo cluster word count

1.配置环境说明 spark:5台zybo板,192.168.1.1master,其它4台为slave hadoop:192.168.1.1(外接SanDisk ) 2.单节点hadoop测试: 如果出现内存不足情况如下: 查看当前虚拟内存容量: free -m cd /mnt mkdir swap cd swap/ 创建一个swap文件 dd if=/dev/zero of=swapfile bs=1024 count=1000000 把生成的文件转换成swap文件 mkswap swapfi

Learn ZYNQ (7)

矩阵相乘的例子 参考博客:http://blog.csdn.net/kkk584520/article/details/18812321 MatrixMultiply.c typedef int data_type; #define N 5 void MatrixMultiply(data_type AA[N*N],data_type bb[N],data_type cc[N]) { int i,j; for(i = 0;i<N;i++) { data_type sum = 0; for(j =

ZYNQ 7000平台UDP数据包(1字节或2字节)校验和Checksum错误0xFFFF解决方案（linux+vxworks6.9平台）

在赛灵思ZYNQ 7000平台,使用UDP方式发送1字节或者2字节数据时,校验和为错误值0xffff,接收机无法正常接收ZYNQ7000平台发送的数据,本人已经找到该问题的解决方案,有该问题的朋友可以通过邮箱[email protected]与我联系,联系时请详细描述你的环境,针对该咨询提供的问题解决方案会收取一定的费用,费用不会太高,现在是知识付费的年代,希望各位理解,同时如果能够解决您的问题,也是为你节约了开支.我会及时回复邮件的.具体事项可以邮件沟通[email protected].

SylixOS 基于ZYNQ的时钟频率修改详解

概述 本文档以ZYNQ7000平台为例,详细介绍如何去修改ZYNQ的时钟频率. 时钟频率修改流程 ZYNQ7000的时钟频率修改流程,如图 2.1所示.具体步骤如下: 步骤一:解除ZYNQ7000的寄存器写锁定: 步骤二:向对应寄存器写入我们需要设置的PLL倍频值和PLL配置参数: 步骤三:进行PLL的旁路模式转换和软件重启,使我们刚刚设置的PLL倍频值和PLL配置参数生效: 步骤四:重新使寄存器处于写锁定状态. 图 2.1 ZYNQ7000的时钟频率修改流程图 ZYNQ7000的ARM_PLL

利用Zynq Soc创建一个嵌入式工程

英文题目:Using the Zynq SoC Processing System,参考自ADI的ug1165文档. 利用Zynq Soc创建一个嵌入式工程,该工程总体上包括五个步骤: 步骤一.新建空白工程 步骤二.创建一个Embedded Processor工程 步骤三.Zynq7 Processing System的管理 步骤四.综合仿真.编译运行.生成二进制文件 步骤五.Exporting Hardware to SDK 步骤一.新建工程 1. 点击Vivado图标启动软件,Create

Vivado Zynq 学习

目标 --------------------------------------------------- 视图菜单 (Vivado SDK) 文件目录 文件格式 流程:工程 -> 非工程 -> 脚本 Create Hardware Design Create Hardware IP Use    Hardware IP:  ARM AXI USB DMA RAM CLK RST Create Software Design 方法 ------------------------------