AAS代码运行-第4章

[[email protected] aas]# ls
ch02  ch03  spark-1.2.1-bin-hadoop2.4  spark-1.2.1-bin-hadoop2.4.tgz
[[email protected] aas]# cd spark-1.2.1-bin-hadoop2.4
[[email protected] spark-1.2.1-bin-hadoop2.4]# cd ..
[[email protected] aas]# mkdir ch04
[[email protected] aas]# cd ch04
[[email protected] ch04]# ls
[[email protected] ch04]# wget https://archive.ics.uci.edu/ml/machine-learning-databases/covtype/covtype.data.gz
--2015-12-06 08:52:34--  https://archive.ics.uci.edu/ml/machine-learning-databases/covtype/covtype.data.gz
Resolving archive.ics.uci.edu... 128.195.10.249
Connecting to archive.ics.uci.edu|128.195.10.249|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 11240707 (11M) [application/x-gzip]
Saving to: ?.ovtype.data.gz?

100%[===============================================================================================================================================================>] 11,240,707  2.62M/s   in 4.2s    

2015-12-06 08:52:39 (2.53 MB/s) - ?.ovtype.data.gz?.saved [11240707/11240707]

[[email protected] ch04]# wget https://archive.ics.uci.edu/ml/machine-learning-databases/covtype/covtype.info
--2015-12-06 08:53:00--  https://archive.ics.uci.edu/ml/machine-learning-databases/covtype/covtype.info
Resolving archive.ics.uci.edu... 128.195.10.249
Connecting to archive.ics.uci.edu|128.195.10.249|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 14610 (14K) [text/plain]
Saving to: ?.ovtype.info?

100%[===============================================================================================================================================================>] 14,610      --.-K/s   in 0.001s  

2015-12-06 08:53:01 (15.6 MB/s) - ?.ovtype.info?.saved [14610/14610]

[[email protected] ch04]# wget https://archive.ics.uci.edu/ml/machine-learning-databases/covtype/old_covtype.info
--2015-12-06 08:53:25--  https://archive.ics.uci.edu/ml/machine-learning-databases/covtype/old_covtype.info
Resolving archive.ics.uci.edu... 128.195.10.249
Connecting to archive.ics.uci.edu|128.195.10.249|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4847 (4.7K) [text/plain]
Saving to: ?.ld_covtype.info?

100%[===============================================================================================================================================================>] 4,847       --.-K/s   in 0s      

2015-12-06 08:53:26 (12.7 MB/s) - ?.ld_covtype.info?.saved [4847/4847]

将数据放到HDFS上

[[email protected] ch04]# ls
covtype.data.gz  covtype.info  old_covtype.info
[[email protected] ch04]# gunzip -d covtype.data.gz
[[email protected] ch04]# ll
total 73432
-rw-r--r-- 1 root root 75169317 Sep  1  1998 covtype.data
-rw-r--r-- 1 root root    14610 Apr 18  2010 covtype.info
-rw-r--r-- 1 root root     4847 Sep  1  1998 old_covtype.info
[[email protected] ch04]# hdfs dfs -mkdir /user/root/covtype
[[email protected] ch04]# hdfs dfs -put * /user/root/covtype
[[email protected] ch04]# hdfs dfs -ls /user/root/covtype
Found 3 items
-rw-r--r--   3 root supergroup   75169317 2015-12-06 09:02 /user/root/covtype/covtype.data
-rw-r--r--   3 root supergroup      14610 2015-12-06 09:02 /user/root/covtype/covtype.info
-rw-r--r--   3 root supergroup       4847 2015-12-06 09:02 /user/root/covtype/old_covtype.info

启动spark-shell

[[email protected] ch04]# ../spark-1.2.1-bin-hadoop2.4/bin/spark-shell --master yarn-client
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  ‘_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.2.1
      /_/

Using Scala version 2.10.4 (OpenJDK 64-Bit Server VM, Java 1.7.0_09-icedtea)
Type in expressions to have them evaluated.
Type :help for more information.
15/12/06 09:08:05 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Spark context available as sc.

运行代码4.7节代码：

scala> import org.apache.spark.mllib.linalg._
import org.apache.spark.mllib.linalg._

scala> import org.apache.spark.mllib.regression._
import org.apache.spark.mllib.regression._

scala> 

scala> val rawData = sc.textFile("hdfs:///user/ds/covtype.data" )
rawData: org.apache.spark.rdd.RDD[String] = hdfs:///user/ds/covtype.data MappedRDD[1] at textFile at <console>:18

scala> val data = rawData.map { line =>
     | val values = line.split(‘,‘ ).map(_. toDouble)
     |   val featureVector = Vectors.dense(values. init)
     |   val label = values.last - 1
     |   LabeledPoint(label, featureVector)
     | }
data: org.apache.spark.rdd.RDD[org.apache.spark.mllib.regression.LabeledPoint] = MappedRDD[2] at map at <console>:20

scala> val Array(trainData, cvData, testData) = data. randomSplit(Array(0.8, 0.1, 0.1))
trainData: org.apache.spark.rdd.RDD[org.apache.spark.mllib.regression.LabeledPoint] = PartitionwiseSampledRDD[3] at randomSplit at <console>:22
cvData: org.apache.spark.rdd.RDD[org.apache.spark.mllib.regression.LabeledPoint] = PartitionwiseSampledRDD[4] at randomSplit at <console>:22
testData: org.apache.spark.rdd.RDD[org.apache.spark.mllib.regression.LabeledPoint] = PartitionwiseSampledRDD[5] at randomSplit at <console>:22

scala> trainData.cache()
res0: trainData.type = PartitionwiseSampledRDD[3] at randomSplit at <console>:22

scala> cvData.cache()
res1: cvData.type = PartitionwiseSampledRDD[4] at randomSplit at <console>:22

scala> testData.cache()
res2: testData.type = PartitionwiseSampledRDD[5] at randomSplit at <console>:22

scala> import org.apache.spark.mllib.evaluation._
import org.apache.spark.mllib.evaluation._



scala> import org.apache.spark.mllib.tree._
import org.apache.spark.mllib.tree._




scala> import org.apache.spark.mllib.tree.model._
import org.apache.spark.mllib.tree.model._




scala> import org.apache.spark.rdd._
import org.apache.spark.rdd._




scala> def getMetrics(model: DecisionTreeModel, data: RDD[LabeledPoint]):
     |   MulticlassMetrics = {
     |   val predictionsAndLabels = data. map(example =>(model. predict(example. features), example. label))
     |   new MulticlassMetrics(predictionsAndLabels)
     | }
getMetrics: (model: org.apache.spark.mllib.tree.model.DecisionTreeModel, data: org.apache.spark.rdd.RDD[org.apache.spark.mllib.regression.LabeledPoint])org.apache.spark.mllib.evaluation.MulticlassMetrics




scala> val model = DecisionTree.trainClassifier(trainData, 7, Map[Int, Int](), "gini" , 4, 100)
model: org.apache.spark.mllib.tree.model.DecisionTreeModel = DecisionTreeModel classifier of depth 4 with 31 nodes




scala> val metrics = getMetrics(model, cvData)
metrics: org.apache.spark.mllib.evaluation.MulticlassMetrics = [email protected]




scala> metrics. confusionMatrix
res6: org.apache.spark.mllib.linalg.Matrix =                                                                                                                                                             
15535.0  5345.0   21.0    0.0  0.0   0.0  392.0  
6669.0   20855.0  688.0   0.0  5.0   0.0  47.0   
0.0      610.0    2942.0  0.0  0.0   0.0  0.0    
0.0      0.0      274.0   0.0  0.0   0.0  0.0    
12.0     874.0    57.0    0.0  15.0  0.0  0.0    
0.0      446.0    1318.0  0.0  0.0   0.0  0.0    
1150.0   19.0     8.0     0.0  0.0   0.0  905.0




scala> metrics.precision
res7: Double = 0.6917696392665028


scala> metrics.precision
res7: Double = 0.6917696392665028


scala> (0 until 7).map(
     |   cat => (metrics.precision(cat), metrics.recall(cat))
     | ).foreach(println)
(0.6648549174013524,0.729582491898746)
(0.7408788944545099,0.7378644211718086)
(0.554257724189902,0.8282657657657657)
(0.0,0.0)
(0.75,0.015657620041753653)
(0.0,0.0)
(0.6733630952380952,0.4346781940441883)


scala> import org.apache.spark.rdd._
import org.apache.spark.rdd._


scala> def classProbabilities(data: RDD[LabeledPoint]): Array[Double] = {
     |   val countsByCategory = data.map(_.label).countByValue()
     |   val counts = countsByCategory.toArray.sortBy(_. _1).map(_. _2)
     |   counts.map(_.toDouble / counts.sum)
     | }
classProbabilities: (data: org.apache.spark.rdd.RDD[org.apache.spark.mllib.regression.LabeledPoint])Array[Double]


scala> val trainPriorProbabilities = classProbabilities(trainData)
trainPriorProbabilities: Array[Double] = Array(0.3644680841907762, 0.48778063233452534, 0.06163475731247069, 0.004682046846288574, 0.0163893156379504, 0.029860958700732, 0.035184204977256786)


scala> val cvPriorProbabilities = classProbabilities(cvData)
cvPriorProbabilities: Array[Double] = Array(0.36594084589341264, 0.4857442384037672, 0.061044563218588345, 0.004708955608641105, 0.016464158660869265, 0.03031604997679894, 0.03578118823792256)


scala> trainPriorProbabilities.zip(cvPriorProbabilities).map {
     |   case (trainProb, cvProb) => trainProb * cvProb
     | }.sum
res9: Double = 0.3765289404519721


scala> val evaluations = 
     |   for (impurity <- Array("gini" , "entropy" );
     |        depth <- Array(1, 20);
     |        bins <- Array(10, 300))
     |   yield {
     |     val model = DecisionTree. trainClassifier(trainData, 7, Map[Int, Int](), impurity, depth, bins)
     |     val predictionsAndLabels = cvData. map(example =>(model. predict(example. features), example. label))
     |     val accuracy = new MulticlassMetrics(predictionsAndLabels). precision
     |     ((impurity, depth, bins), accuracy)
     |  }
evaluations: Array[((String, Int, Int), Double)] = Array(((gini,1,10),0.6319968377816351), ((gini,1,300),0.6323577431385017), ((gini,20,10),0.889253613350061), ((gini,20,300),0.9074191829790159), ((entropy,1,10),0.4857442384037672), ((entropy,1,300),0.4857442384037672), ((entropy,20,10),0.8946500077336862), ((entropy,20,300),0.9099455204770825))


scala> evaluations.sortBy(_. _2).reverse.foreach(println)
((entropy,20,300),0.9099455204770825)
((gini,20,300),0.9074191829790159)
((entropy,20,10),0.8946500077336862)
((gini,20,10),0.889253613350061)
((gini,1,300),0.6323577431385017)
((gini,1,10),0.6319968377816351)
((entropy,1,300),0.4857442384037672)
((entropy,1,10),0.4857442384037672)


scala> val data = rawData.map { line =>
     |   val values = line.split(‘,‘ ).map(_.toDouble)
     |   val wilderness = values.slice(10, 14).indexOf(1.0).toDouble
     |   val soil = values.slice(14, 54).indexOf(1.0).toDouble
     |   val featureVector =
     |   Vectors.dense(values.slice(0, 10) :+ wilderness :+ soil)
     |   val label = values.last - 1
     |   LabeledPoint(label, featureVector)
     | }
data: org.apache.spark.rdd.RDD[org.apache.spark.mllib.regression.LabeledPoint] = MappedRDD[391] at map at <console>:47


scala>


scala> val evaluations =
     |   for (impurity <- Array("gini" , "entropy" );
     |        depth <- Array(10, 20, 30);
     |        bins <- Array(40, 300))
     |   yield {
     |     val model = DecisionTree. trainClassifier(trainData, 7, Map(10 -> 4, 11 -> 40),impurity, depth, bins)
     |     val trainAccuracy = getMetrics(model, trainData). precision
     |     val cvAccuracy = getMetrics(model, cvData). precision
     |     ((impurity, depth, bins), (trainAccuracy, cvAccuracy))
     |  }
evaluations: Array[((String, Int, Int), (Double, Double))] = Array(((gini,10,40),(0.7772542032989496,0.7730420884389984)), ((gini,10,300),(0.7849615065174265,0.7793665251688522)), ((gini,20,40),(0.9393033733975393,0.904480382215959)), ((gini,20,300),(0.9421715574260792,0.904480382215959)), ((gini,30,40),(0.9972329447406585,0.9341089934177738)), ((gini,30,300),(0.9974352022790551,0.9347964321927579)), ((entropy,10,40),(0.7768755083334409,0.7716672108890302)), ((entropy,10,300),(0.7715307452975122,0.7655318198222971)), ((entropy,20,40),(0.9487578374796128,0.9103407977726984)), ((entropy,20,300),(0.9484781196073622,0.9088971763452317)), ((entropy,30,40),(0.998582045555283,0.9374430714764467)), ((entropy,30,300),(0.9990833860493938,0.9413786584632307)))


scala> val forest = RandomForest. trainClassifier(
     |  trainData, 7, Map(10 -> 4, 11 -> 40), 20,
     |  "auto" , "entropy" , 30, 300)
forest: org.apache.spark.mllib.tree.model.RandomForestModel =                                                                                                                                            
TreeEnsembleModel classifier with 20 trees


 
				


				
时间： 2024-11-10 12:07:40 

		


		


	

	
	

		


		
AAS代码运行-第4章的相关文章


								

		

			

                AAS代码运行-第11章-2
			

		

		

									

				hdfs dfs -ls /user/littlesuccess/AdvancedAnalysisWithSparkhdfs dfs -mkdir /user/littlesuccess/AdvancedAnalysisWithSpark/ch11hdfs dfs -put fish.py /user/littlesuccess/AdvancedAnalysisWithSpark/ch11 做好上述准备工作之后,就可以运行pyspark代码了: raw_data = sc.textFile('h			

		

	

				

		

			

                AAS代码运行-第11章-1
			

		

		

									

				启动PySpark export IPYTHON=1 # PySpark也可使用IPython shell pyspark --master yarn --num-executors 3 发生如下错误: /opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/spark/bin/pyspark: line 135: exec: ipython: not found 原因是没有按照ipython,在google所有一下安装的方法,到网站			

		

	

				

		

			

                20172327 2017-2018-2 《第一行代码Android》第一章学习总结
			

		

		

									

				学号 2017-2018-2 <第一行代码Android>第一章学习总结 教材学习内容总结 - Android系统架构: 1.Linux内核层 Android系统是基于Linux内核的,这一层为Android设备的各种硬件提供了底层的驱动,如显示驱动,音频驱动,照相机驱动,蓝牙驱动,Wi-Fi驱动,电源管理等. 2.系统运行底层 - 通过一些C/C++库来为Android系统提供了主要的特性支持 库名 功能 SQLite库 提供数据库的支持 OpenGL/ES库 提供3D绘图支持 Webkit			

		

	

				

		

			

                内存的划分 &amp; 程序代码运行时内存工作流程
			

		

		

									

				内存的划分: 1,寄存器. 2,本地方法区. 3,方法区. 4,栈内存. 存储的都是局部变量. 而且变量所属的作用域一旦结束,该变量就自动释放. 5,堆内存. 存储是数组和对象(其实数组就是对象) ,凡是new建立的都在堆中. 特点: 1)每一个实体都有首地址值. 2)堆内存中的每一个变量都有默认初始化值,根据类型的不同而不同.整数是0,小数0.0或者0.0f,boolean类型是false,char类型是 '\u0000',引用数据类型是NULL 3)垃圾回收机制. 全局变量和局部变量的区别:			

		

	

				

		

			

                第十章实践——系统级I/O代码运行
			

		

		

									

				第十章实践——系统级I/O代码运行 实验代码清单如下: 1. cp1——复制一个文件到另一个文件中(两个已经存在的文件) 复制前: 执行后结果 2.			

		

	

				

		

			

                springAop 使用@Around，@After等注解时，代码运行两边的问题
			

		

		

									

				springAop使用@Around,@After等注解时,代码运行两边的问题 将@Component注解删掉就好了			

		

	

				

		

			

                监控代码运行时长 -- StopWatch用法例程
			

		

		

									

				在.net环境下,精确的测量出某段代码运行的时长,在网络通信.串口通信以及异步操作中很有意义.现在做了简单的总结.具体代码如下: (1).首先 using System.Diagnostics; (2).主要代码 Stopwatch sw = new Stopwatch(); //监听循环10000次需要的时长 // 计时开始 sw.Start(); for (int i = 0; i < 10000;i++ ) { // to do } // 计时结束 sw.Stop(); Console.W			

		

	

				

		

			

                Python代码运行助手(6)----帮助你走的更久
			

		

		

									

				Python代码运行助手可以让你在线输入Python代码,然后通过本机运行的一个Python脚本来执行代码.原理如下: 在网页输入代码: 点击Run按钮,代码被发送到本机正在运行的Python代码运行助手: Python代码运行助手将代码保存为临时文件,然后调用Python解释器执行代码: 网页显示代码执行结果: 下载 点击右键,目标另存为:learning.py 备用下载地址:learning.py 运行 在存放learning.py的目录下运行命令: C:\Users\michael\Dow			

		

	

				

		

			

                Hbase集群搭建及所有配置调优参数整理及API代码运行
			

		

		

									

				最近为了方便开发,在自己的虚拟机上搭建了三节点的Hadoop集群与Hbase集群,hadoop集群的搭建与zookeeper集群这里就不再详细说明,原来的笔记中记录过.这里将hbase配置参数进行相应整理,方便日后使用. 首先vi ~/.bash_profile将hbase的环境变量进行配置,最后source ~./bash_profile使之立即生效 1.修改hbase-env.sh 由于我使用的是外置的zookeeper,所以这里HBASE_MANAGES_ZK设置为,设置参数: # The