Spark的交互式脚本是一种学习API的简单途径,,也是分析数据集交互的有力工具。
Spark抽象的分布式集群空间叫做Resilient Distributed Dataset (RDD)弹性数据集。
其中,RDD有两种创建方式:
(1)、从Hadoop的文件系统输入(例如HDFS);
(2)、有其他已存在的RDD转换得到新的RDD;
下面进行简单的测试:
1. 进入SPARK_HOME/bin下运行命令:
$./spark-shell2. 利用HDFS上的一个文本文件创建一个新RDD:
scala> var textFile = sc.textFile("hdfs://localhost:50040/input/WordCount/text1");textFile: org.apache.spark.rdd.RDD[String] = MappedRDD[1] at textFile at <console>:123. RDD有两种类型的操作 ,分别是Action(返回values)和Transformations(返回一个新的RDD)
(1)Action相当于执行一个动作,会返回一个结果:
scala> textFile.count() // RDD中有多少行
输出结果2:
14/11/11 22:59:07 INFO spark.SparkContext: Job finished: count at <console>:15, took 5.654325469 sres1: Long = 2
scala> textFile.first() // RDD第一行的内容结果输出:
14/11/11 23:01:25 INFO spark.SparkContext: Job finished: first at <console>:15, took 0.049004829 sres3: String = hello world
(2)Transformation相当于一个转换,会将一个RDD转换,并返回一个新的RDD:
scala> textFile.filter(line => line.contains("hello")).count() // 有多少行含有hello结果输出:
14/11/11 23:06:33 INFO spark.SparkContext: Job finished: count at <console>:15, took 0.867975549 sres4: Long = 24. spark shell的WordCount
scala> val file = sc.textFile("hdfs://localhost:50040/input")scala> val count = file.flatMap(line => line.split(" ")).map(word => (word, 1)).reduceByKey(_+_)scala> count.collect()
输出结果:
14/11/11 23:11:46 INFO spark.SparkContext: Job finished: collect at <console>:17, took 1.624248037 sres5: Array[(String, Int)] = Array((hello,2), (world,1), (My,1), (is,1), (love,1), (I,1), (Urey,1), (hadoop,1), (name,1), (programming,1))当然基于RDD还有很多复杂的操作,后面学习过程中遇到再总结出来。
融入银河,就安谧地和明月为伴照亮长天;没入草莽,