大饼辉的专栏

Hadoop版本有很多,,这里我选择的是cdh版本。cdh是Cloudera公司在Apache原版基础经过加工的东西。chd具体的下载地址为:

版本信息如下:Hadoop:Hadoop 2.3.0-cdh5.1.0JDK:1.7.0_79Maven:apache-maven-3.2.5(3.3.1以及更高版本必须在JDK1.7以上)Protobuf:protobuf-2.5.0Ant:1.7.11.安装MavenMaven可以自己到Maven官网上下载()。需要注意的是:如果JDK是1.6及其以下的,那Maven最多只能下载到3.2.5,如果下载3.3.1及其更高版本就必须要1.7的JDK。

2.安装ProtobufProtobuf的安装参见:,安装过程也很简单。

3.安装其他需要的组件还需要安装ant、gcc等组件,运行如下命令,会根据需要自己到网上下载安装。

yum install autoconf automake libtool ant gcc gcc-c++ make cmake openssl-devel ncurses-devel如果失败就多试几次。如果一直失败可考虑自己上网下载然后手动安装。具体流程为:Windows上网下载ftp传给Linux在Linux上解压安装。其中Ant的安装步骤可见:

4.下载Hadoop源码并解压源码下载地址为:

5.编译源码运行:

mvn package -DskipTests -Pdist,native -Dtar

如果出现错误可以加上-X显示错误详细信息:

mvn -X package -DskipTests -Pdist,native –Dtar

重编译完成后,native包一般在如下的目录下:

../hadoop-2.3.0-cdh5.1.0/src/hadoop-common-project/hadoop-common/target/native/target/usr/local/lib/

如下图:

安装过程中常见的错误:

1、maven中的jar包下载失败。

Downloading: https://repository.cloudera.com/artifactory/cloudera-repos/com/cloudera/cdh/cdh-root/5.1.0/cdh-root-5.1.0.pomDownloading: Downloading: https://repo.maven.apache.org/maven2/com/cloudera/cdh/cdh-root/5.1.0/cdh-root-5.1.0.pom[ERROR] The build could not read 1 project -> [Help 1][ERROR][ERROR] The project org.apache.hadoop:hadoop-main:2.3.0-cdh5.1.0 (/home/hbase/install/hadoop-2.3.0-cdh5.1.0/src/pom.xml) has 1 error[ERROR]Non-resolvable parent POM: Could not transfer artifact com.cloudera.cdh:cdh-root:pom:5.1.0 from/to cdh.repo (https://repository.cloudera.com/artifactory/cloudera-repos): repository.cloudera.com and 'parent.relativePath' points at wrong local POM @ line 19, column 11: Unknown host repository.cloudera.com -> [Help 2][ERROR][ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.[ERROR] Re-run Maven using the -X switch to enable full debug logging.[ERROR][ERROR] For more information about the errors and possible solutions, please read the following articles:[ERROR] [Help 1] [ERROR] [Help 2] [ERROR] Unresolveable build extension: Plugin org.apache.felix:maven-bundle-plugin:2.4.0 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.felix:maven-bundle-plugin:jar:2.4.0 @ at org.apache.maven.project.DefaultProjectBuilder.build(DefaultProjectBuilder.java:416)at org.apache.maven.DefaultMaven.collectProjects(DefaultMaven.java:672)at org.apache.maven.DefaultMaven.getProjectsForMavenReactor(DefaultMaven.java:663)at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:253)at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:155)at org.apache.maven.cli.MavenCli.execute(MavenCli.java:584)at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:216)at org.apache.maven.cli.MavenCli.main(MavenCli.java:160)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)at java.lang.reflect.Method.invoke(Method.java:597)at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)[ERROR] [ERROR] The project org.apache.hadoop:hadoop-main:2.3.0 (/usr/local/release-2.3.0/pom.xml) has 1 error原因找来找去,搞了一天,最后发现Linux集群无法上网(通过ping 不通)。最后修改了一下DNS就可以上网了。

2、编译失败

通过mvn package总是失败,可以考虑通过mvn compile来,如下:

mvn compile -Pdist,native -Dskiptests -Dtar

知识提示:Hadoop调试的开启和关闭开启:export HADOOP_ROOT_LOGGER=DEBUG,console关闭:export HADOOP_ROOT_LOGGER=INFO,console

全文完,转载请注明出处:

世上最累人的事,莫过於虚伪的过日子

大饼辉的专栏

相关文章:

你感兴趣的文章:

标签云: