大家在官网下载hadoop2.6安装完使用的时候,总是在控制台有这样一句
WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
因为官网给的32位系统编译的版本,在64位的操作系统上使用就需要自己编译源码了
下面介绍下我的编译过程:
这里是下载了maven,ant,findbugs,分别解压后,配置环境变量
这是hadoop 2.6的源码,解压即可
下面继续安装需要的软件:
1、安装gcc,gcc-c++
2、安装make,cmake
3、安装openssl-devel,ncurses-devel
4、最后开始安装protoc buf
压缩包解压后
./configure
make
make check
make install
5、编译
mvn clean install -DskipTests
mvn package -Pdist,native -DskipTests -Dtar
6、编译成功
[INFO] Apache Hadoop Main ………………………….. SUCCESS [7.430s][INFO] Apache Hadoop Project POM ……………………. SUCCESS [16.054s][INFO] Apache Hadoop Annotations ……………………. SUCCESS [4.956s][INFO] Apache Hadoop Assemblies …………………….. SUCCESS [0.536s][INFO] Apache Hadoop Project Dist POM ……………….. SUCCESS [18.575s][INFO] Apache Hadoop Maven Plugins ………………….. SUCCESS [5.347s][INFO] Apache Hadoop MiniKDC ……………………….. SUCCESS [4.062s][INFO] Apache Hadoop Auth ………………………….. SUCCESS [6.434s][INFO] Apache Hadoop Auth Examples ………………….. SUCCESS [4.823s][INFO] Apache Hadoop Common ………………………… SUCCESS [2:44.417s][INFO] Apache Hadoop NFS …………………………… SUCCESS [9.645s][INFO] Apache Hadoop KMS …………………………… SUCCESS [19.681s][INFO] Apache Hadoop Common Project …………………. SUCCESS [0.074s][INFO] Apache Hadoop HDFS ………………………….. SUCCESS [5:05.240s][INFO] Apache Hadoop HttpFS ………………………… SUCCESS [28.723s][INFO] Apache Hadoop HDFS BookKeeper Journal …………. SUCCESS [15.884s][INFO] Apache Hadoop HDFS-NFS ………………………. SUCCESS [6.125s][INFO] Apache Hadoop HDFS Project …………………… SUCCESS [0.072s][INFO] hadoop-yarn ………………………………… SUCCESS [0.068s][INFO] hadoop-yarn-api …………………………….. SUCCESS [4:41.183s][INFO] hadoop-yarn-common ………………………….. SUCCESS [43.377s][INFO] hadoop-yarn-server ………………………….. SUCCESS [0.073s][INFO] hadoop-yarn-server-common ……………………. SUCCESS [17.120s][INFO] hadoop-yarn-server-nodemanager ……………….. SUCCESS [23.088s][INFO] hadoop-yarn-server-web-proxy …………………. SUCCESS [4.509s][INFO] hadoop-yarn-server-applicationhistoryservice …… SUCCESS [9.256s][INFO] hadoop-yarn-server-resourcemanager ……………. SUCCESS [25.804s][INFO] hadoop-yarn-server-tests …………………….. SUCCESS [8.690s][INFO] hadoop-yarn-client ………………………….. SUCCESS [10.050s][INFO] hadoop-yarn-applications …………………….. SUCCESS [0.062s][INFO] hadoop-yarn-applications-distributedshell ……… SUCCESS [4.423s][INFO] hadoop-yarn-applications-unmanaged-am-launcher …. SUCCESS [2.732s][INFO] hadoop-yarn-site ……………………………. SUCCESS [0.064s][INFO] hadoop-yarn-registry ………………………… SUCCESS [8.125s][INFO] hadoop-yarn-project …………………………. SUCCESS [7.215s][INFO] hadoop-mapreduce-client ……………………… SUCCESS [0.114s][INFO] hadoop-mapreduce-client-core …………………. SUCCESS [27.948s][INFO] hadoop-mapreduce-client-common ……………….. SUCCESS [31.602s][INFO] hadoop-mapreduce-client-shuffle ………………. SUCCESS [6.624s][INFO] hadoop-mapreduce-client-app ………………….. SUCCESS [12.136s][INFO] hadoop-mapreduce-client-hs …………………… SUCCESS [12.574s][INFO] hadoop-mapreduce-client-jobclient …………….. SUCCESS [6.447s][INFO] hadoop-mapreduce-client-hs-plugins ……………. SUCCESS [2.662s][INFO] Apache Hadoop MapReduce Examples ……………… SUCCESS [8.593s][INFO] hadoop-mapreduce ……………………………. SUCCESS [6.852s][INFO] Apache Hadoop MapReduce Streaming …………….. SUCCESS [7.445s][INFO] Apache Hadoop Distributed Copy ……………….. SUCCESS [11.579s][INFO] Apache Hadoop Archives ………………………. SUCCESS [3.206s][INFO] Apache Hadoop Rumen …………………………. SUCCESS [8.972s][INFO] Apache Hadoop Gridmix ……………………….. SUCCESS [6.194s][INFO] Apache Hadoop Data Join ……………………… SUCCESS [4.075s][INFO] Apache Hadoop Ant Tasks ……………………… SUCCESS [3.546s][INFO] Apache Hadoop Extras ………………………… SUCCESS [4.469s][INFO] Apache Hadoop Pipes …………………………. SUCCESS [9.900s][INFO] Apache Hadoop OpenStack support ………………. SUCCESS [8.320s][INFO] Apache Hadoop Amazon Web Services support ……… SUCCESS [6.285s][INFO] Apache Hadoop Client ………………………… SUCCESS [10.249s][INFO] Apache Hadoop Mini-Cluster …………………… SUCCESS [0.183s][INFO] Apache Hadoop Scheduler Load Simulator ………… SUCCESS [6.699s][INFO] Apache Hadoop Tools Dist …………………….. SUCCESS [14.929s][INFO] Apache Hadoop Tools …………………………. SUCCESS [0.058s][INFO] Apache Hadoop Distribution …………………… SUCCESS [48.591s][INFO] ————————————————————————[INFO] BUILD SUCCESS[INFO] ————————————————————————[INFO] Total time: 22:18.595s[INFO] Finished at: Wed Feb 04 16:35:19 CST 2015[INFO] Final Memory: 101M/251M[INFO] ————————————————————————
7、替换native文件夹
/home/hadoop/hadoop-2.6.0-src/hadoop-dist/target/hadoop-2.6.0/lib/native
将以上编译好的文件夹内的所有内容替换掉你原来的32位的hadoop的native文件
现在你可以安心舒服的进行文件系统操作了,再也不会看见讨厌的:
WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
附录:
软件下载:
maven ant findbugs 3.0.0 protobuf 2.5.0 hadoop 2.6
不论你在什么时候开始,重要的是开始之后就不要停止