参看文档
也可以关注我的另外一篇文章
也可以加入群316297243一起学习讨论
此文章用于学习和交流，转载请注明
我依赖的是hadoop-1.2.1版本的
加入如下依赖
<dependency><groupId>org.apache.hadoop</groupId><artifactId>hadoop-core</artifactId><version>1.2.1</version> </dependency>
加入依赖之后等待下载相关的jar包
这里不加也行
下载配置文件:
core-site.sh
hdfs-site.sh
mapred-site.sh
public class WordCount {public static class WordCountMapper extends MapReduceBase implementsMapper<Object, Text, Text, IntWritable> {private final static IntWritable one = new IntWritable(1);private Text word = new Text();@Overridepublic void map(Object key, Text value,OutputCollector<Text, IntWritable> output, Reporter reporter)throws IOException {StringTokenizer itr = new StringTokenizer(value.toString());while (itr.hasMoreTokens()) {word.set(itr.nextToken());output.collect(word, one);}}}public static class WordCountReducer extends MapReduceBase implementsReducer<Text, IntWritable, Text, IntWritable> {private IntWritable result = new IntWritable();@Overridepublic void reduce(Text key, Iterator<IntWritable> values,OutputCollector<Text, IntWritable> output, Reporter reporter)throws IOException {// TODO Auto-generated method stubint sum = 0;while (values.hasNext()) {sum += values.next().get();}result.set(sum);output.collect(key, result);}}public static void main(String[] args) throws Exception {String input = null;String output = null;JobConf conf = new JobConf(WordCount.class);conf.setJobName("WordCount");// 测试环境if (args == null || args.length < 2) {input = "hdfs://10.2.1.35:9000/usr/hdfs/input_test1";output = "hdfs://10.2.1.35:9000/usr/hdfs/test1_result";/ 加载配置文件,可以注释掉conf.addResource("classpath:/hadoop/core-site.xml");conf.addResource("classpath:/hadoop/hdfs-site.xml");conf.addResource("classpath:/hadoop/mapred-site.xml");} else {// 正式环境input = args[0];output = args[1];}conf.setOutputKeyClass(Text.class);conf.setOutputValueClass(IntWritable.class);conf.setMapperClass(WordCountMapper.class);conf.setCombinerClass(WordCountReducer.class);conf.setReducerClass(WordCountReducer.class);conf.setInputFormat(TextInputFormat.class);conf.setOutputFormat(TextOutputFormat.class);FileInputFormat.setInputPaths(conf, new Path(input));FileOutputFormat.setOutputPath(conf, new Path(output));JobClient.runJob(conf);System.exit(0);}}
注：
上述代码根据自己的实际情况编写
4）运行
出现异常
解决方法是，，修改/hadoop-1.2.1/src/core/org/apache/hadoop/fs/FileUtil.java文件
再次运行
<property><name>dfs.permissions</name><value>false</value></property>
不知道你们行不行,试了几次都不行。
运行输出即可。
本文以2.6.0为例
Maven构建项目上面已经有了，在这不赘述了。
<dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>3.8.1</version> <scope>test</scope></dependency><dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-common</artifactId> <version>2.6.0</version> <exclusions> <exclusion> <groupId>jdk.tools</groupId> <artifactId>jdk.tools</artifactId> </exclusion> </exclusions></dependency><dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-hdfs</artifactId> <version>2.6.0</version></dependency><dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-client</artifactId> <version>2.6.0</version></dependency><dependency> <groupId>jdk.tools</groupId> <artifactId>jdk.tools</artifactId> <version>1.6</version> <scope>system</scope> <systemPath>${JAVA_HOME}/lib/tools.jar</systemPath></dependency>
上述的jdk.tools如果不这样写会报错。
但是2.6.0不会出现这么多错误，只会出现一个错误。
明天的希望，让我们忘了今天的痛苦
