Spark1.1.0 Actions

Actions

The following table lists some of the common actions supported by Spark. Refer to the RDD API doc (ScalaJavaPython)
and pair RDD functions doc (ScalaJava)
for details.

Action Meaning
reduce(func) Aggregate the elements of the dataset using a function func (which takes two arguments and returns one). The function should be commutative and associative so that it can be computed correctly in parallel.
collect() Return all the elements of the dataset as an array at the driver program. This is usually useful after a filter or other operation that returns a sufficiently small subset of the data.
count() Return the number of elements in the dataset.
first() Return the first element of the dataset (similar to take(1)).
take(n) Return an array with the first n elements of the dataset. Note that this is currently not executed in parallel. Instead, the driver program computes all the elements.
takeSample(withReplacement,num, [seed]) Return an array with a random sample of num elements of the dataset, with or without replacement, optionally pre-specifying a random number generator seed.
takeOrdered(n[ordering]) Return the first n elements of the RDD using either their natural order or a custom comparator.
saveAsTextFile(path) Write the elements of the dataset as a text file (or set of text files) in a given directory in the local filesystem, HDFS or any other Hadoop-supported file system. Spark will call toString on each element to convert it to a line of text in the file.
saveAsSequenceFile(path)

(Java and Scala)

Write the elements of the dataset as a Hadoop SequenceFile in a given path in the local filesystem, HDFS or any other Hadoop-supported file system. This is available on RDDs of key-value pairs that either implement Hadoop‘s Writable interface. In Scala, it
is also available on types that are implicitly convertible to Writable (Spark includes conversions for basic types like Int, Double, String, etc).
saveAsObjectFile(path)

(Java and Scala)

Write the elements of the dataset in a simple format using Java serialization, which can then be loaded usingSparkContext.objectFile().
countByKey() Only available on RDDs of type (K, V). Returns a hashmap of (K, Int) pairs with the count of each key.
foreach(func) Run a function func on each element of the dataset. This is usually done for side effects such as updating an accumulator variable (see below) or interacting with external storage systems.
时间: 2024-10-07 06:43:01

Spark1.1.0 Actions的相关文章

Spark-1.4.0单机部署(Hadoop-2.6.0采用伪分布式)【已测】

??目前手上只有一个机器,就先拿来练下手(事先服务器上没有安装软件)尝试一下Spark的单机部署. ??几个参数: ??JDK-1.7+ ??Hadoop-2.6.0(伪分布式): ??Scala-2.10.5: ??Spark-1.4.0: ??下面是具体的配置过程 安装JDK 1.7+ [下载网址]http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html 环境变量设置(最好不要采用o

spark1.1.0集群安装配置

和分布式文件系统和NoSQL数据库相比而言,spark集群的安装配置还算是比较简单的: 安装JDK,这个几乎不用介绍了(很多软件都需要JDK嘛) wget http://download.oracle.com/otn-pub/java/jdk/7u71-b14/jdk-7u71-linux-x64.tar.gz?AuthParam=1416666050_dca8969bfc01e3d8d42d04040f76ff1 tar -zxvf jdk-7u71-linux-x64.tar.gz 安装sc

Hadoop2.6.0 + Spark1.4.0 在Ubuntu14.10环境下的伪分布式集群的搭建(实践可用)

前言,之前曾多次搭建集群,由于疏于记录,每次搭建的时候到处翻阅博客,很是费劲,在此特别记录集群的搭建过程. 0.环境:Ubuntu14.10.Hadoop2.6.0.spark-1.4.0 1.安装jdk1.7 (1)下载jdk-7u25-linux-i586.tar.gz: (2)解压jdk-7u25-linux-i586.tar.gz,并将其移动到 /opt/java/jdk/路径下面 (3)配置java环境变量: 在 /etc/profile文件中追加 #set java env expo

在Win7虚拟机下搭建Hadoop2.6.0+Spark1.4.0单机环境

Hadoop的安装和配置可以参考我之前的文章:在Win7虚拟机下搭建Hadoop2.6.0伪分布式环境. 本篇介绍如何在Hadoop2.6.0基础上搭建spark1.4.0单机环境. 1. 软件准备 scala-2.11.7.tgz spark-1.4.0-bin-hadoop2.6.tgz 都可以从官网下载. 2. scala安装和配置 scala-2.11.7.tgz解压缩即可.我解压缩到目录/home/vm/tools/scala,之后配置~/.bash_profile环境变量. #sca

java操作spark1.2.0

虽然推荐的是scala,但是还是试一下 1 package org.admln.java7OperateSpark; 2 3 import java.util.Arrays; 4 import java.util.List; 5 import java.util.regex.Pattern; 6 7 import org.apache.spark.SparkConf; 8 import org.apache.spark.api.java.JavaPairRDD; 9 import org.apa

spark1.2.0编译

现在最新版本是1.2.0 我用的maven编译,官网有现成的编译命令 http://spark.apache.org/docs/latest/building-spark.html 我的hadoop是2.2.0,所以命令为: export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m" mvn -Pyarn -Phadoop-2.2 -Dhadoop.version=2.2.0 -Dskip

Apache Spark1.1.0部署与开发环境搭建

Spark是Apache公司推出的一种基于Hadoop Distributed File System(HDFS)的并行计算架构.与MapReduce不同,Spark并不局限于编写map和reduce两个方法,其提供了更为强大的内存计算(in-memory computing)模型,使得用户可以通过编程将数据读取到集群的内存当中,并且可以方便用户快速地重复查询,非常适合用于实现机器学习算法.本文将介绍Apache Spark1.1.0的部署与开发环境搭建. 0. 准备 出于学习目的,本文将Spa

spark1.6.0集群安装

1. 概述 本文是对spark1.6.0分布式集群的安装的一个详细说明,旨在帮助相关人员按照本说明能够快速搭建并使用spark集群. 2. 安装环境 本安装说明的示例环境部署如下: IP 外网IP hostname 备注 10.47.110.38 120.27.153.137 iZ237654q6qZ Master.Slaver 10.24.35.51 114.55.56.190 iZ23pd81xqaZ Slaver 10.45.53.136 114.55.11.55 iZ23mr5ukpzZ

spark1.2.0版本SparkSQL使用parquet类型注意事项

在Spark1.2.0版本中是用parquet存储类型时注意事项: sql语句: select * from order_created_dynamic_partition_parquet; 在spark-sql中执行结果: 2014-05 [[email protected] [[email protected] 2014-05 [[email protected] [[email protected] 2014-05 [[email protected] [[email protected]