Eclipse 编写的spark版WordCount运行在spark上

1、代码编写

if (args.length != 3 ){
      println("usage is org.test.WordCount <master> <input> <output>")
      return
    }
    val sc = new SparkContext(args(0), "WordCount",
    System.getenv("SPARK_HOME"), Seq(System.getenv("SPARK_TEST_JAR")))
    val textFile = sc.textFile(args(1))
    val result = textFile.flatMap(line => line.split("\\s+"))
        .map(word => (word, 1)).reduceByKey(_ + _)
    result.saveAsTextFile(args(2))

2、导出jar包，这里我命名为WordCount.jar

3、运行

bin/spark-submit --master spark://master:7077 /home/zkpk/Desktop/WordCount.jar hdfs://master:9000/dedup_in hdfs://master:9000/dedup_out

先记录下，第三部还没执行，抽空试试