1。背景最近在学习hadoop。本文简单记录下如何通过源码编译hadoop。之所以重新编译hadoop源码，是为了匹配不同操作系统的本地库环境。2、编译源码2.1下载并解压缩源码[root@hadoop01~]#mkdir/opt/hadoop[root@hadoop01~]#cd/opt/hadoop/[root@hadoop01hadoop]#wgethttps://archive.apache.org/dist/hadoop/common/hadoop-3.3.4/hadoop-3.3.4-src.tar.gz[root@hadoop01hadoop]#tar-zxvfhadoop-3.3.4-src.tar.gz[root@hadoop01]hadoop]#rm-rvfhadoop-3.3.4-src.tar.gz2.2查看编译hadoop必须的环境[root@hadoop01hadoop]#pwd/opt/hadoop[root@hadoop01hadoop]#cdhadoop-3.3.4-src/[root@hadoop01hadoop-3.3.4-src]#catBUILDING.txtHadoop构建说明-------------------------------------------------------------------------------要求：*Unix系统*JDK1.8*Maven3.3或更高版本*ProtocolBuffers3.7.1（如果编译本机代码）*CMake3.1或更新版本（如果编译本机代码）*Zlibdevel（如果编译本机代码）*CyrusSASLdevel（如果编译本机代码）*支持thread_local存储的编译器之一：GCC4.8.1或更高版本、VisualStudio、Clang（社区版）、Clang（适用于iOS9和l的版本）ater)（如果编译本机代码）*openssldevel（如果编译本机hadoop-pipes并获得最佳HDFS加密性能）*LinuxFUSE（用户空间中的文件系统）版本2.6或更高版本（如果编译fuse_dfs）*Doxygen（如果编译libhdfspp并生成文档）*首次构建的互联网连接（获取所有Maven和Hadoop依赖项）*python（用于发布的文档）*bats（用于shell代码测试）*Node.js/bower/Ember-cli（用于YARNUIv2构建)--------------------------------------------------------------------------可以看到JDK1.8及以上，maven3.3及以上等.都是必需的2.3安装JDK，需要注意JDK的版本，参考这个文档安装maven[root@hadoop01hadoop]#wgethttps://dlcdn.apache.org/maven/maven-3/3.9.0/binaries/apache-maven-3.9.0-bin.tar.gz[root@hadoop01hadoop]#tar-zxvfapache-maven-3.9.0-bin.tar.gz-C/usr/local#编辑环境变量[root@hadoop01hadoop]#vim/etc/profile#configuremavenexportM2_HOME=/usr/local/apache-maven-3.9.0exportPATH=${M2_HOME}/bin:$PATH[root@hadoop01hadoop]#source/etc/profile#mvn-versionApacheMaven3.9.0(9b58d2bad23a66be161c4664ef21ce219c2c8584)Mavenhome:/usr/local/apache-maven-3.9.0Java版本：1.8......0_333,vendor:OracleCorporation,runtime:/usr/local/jdk8/jreDefaultlocale:zh_CN,platformencoding:UTF-8OSname:"linux",version:"5.11.12-300.el7.aarch64",arch:"aarch64",family:"unix"#配置外星人图片加速访问[root@hadoop01hadoop]#vim/usr/local/apache-maven-3.9.0/conf/settings.xmlalimavenaliyunmavenhttp://maven.aliyun.com/nexus/content/groups/public/central2.5安装编译相关依赖[root@hadoop01hadoop]#yuminstallgccgcc-c++makeautoconfautomakelibtoolcurllzo-develzlib-developensslopenssl-develncurses-develsnappysnappy-develbzip2bzip2-devellzolzo-devellzoplibXtstzlibdoxygencyrus-sasl*saslwrapper-devel*-y2.6安装cmakehadoop*的要求CMake3.1或更新版本（如果编译本机代码）cmake的版本是3.1和以上cmake是没有必要的。#卸载已有的cmake[root@hadoop01hadoop]#yumerasecmake#下载cmake（这里需要根据自己的操作系统下载）[root@hadoop01hadoop]#wgethttps://github.com/Kitware/CMake/releases/download/v3.25.2/cmake-3.25.2.tar.gz[root@hadoop01hadoop]#tar-zxvfcmake-3.25.2.tar.gz#编译安装cmake[root@hadoop01cmake-3.25.2-linux-aarch64]#cdcmake-3.25.2/&&./configure&&make&&makeinstall#查看cmake版本[root@hadoop01cmake-3.25.2]#cmake-versioncmakeversion3.25.2CMakesuitemaintainedandKitware(kitware.com/cmake)支持。2.7安装Snappy*Snappy压缩（仅用于hadoop-mapreduce-client-nativetask）#卸载已安装的snappy[root@hadoop01hadoop]#rm-rf/usr/local/lib/libsnappy*&&rm-rf/lib64/libsnappy*[root@hadoop01hadoop]#wgethttps://src.fedoraproject.org/repo/pkgs/snappy/snappy-1.1.3.tar.gz/7358c82f133dc77798e4c2062a749b73/snappy-1.1.3.tar.gz[root@hadoop01snappy]#tar-zxvfsnappy-1.1.3.tar.gz[root@hadoop01snappy]#cdsnappy-1.1.3/&&./configure&&make&&makeinstall[root@hadoop01build]#2.8安装ProtocolBuffer*ProtocolBuffers3.7.1（如果编译本机代码）安装版本3.7.1[root@hadoop01hadoop]#wgethttps://github。com/protocolbuffers/protobuf/releases/download/v3.7.1/protobuf-java-3.7.1.tar.gz[root@hadoop01hadoop]#tar-zxvfprotobuf-java-3.7.1.tar.gz#编译安装[root@hadoop01hadoop]#cdprotobuf-3.7.1/&&./autogen.sh&&./configure&&make&&makeinstall#验证安装是否成功[root@hadoop01protobuf-3.7.1]#protoc--versionlibprotoc3.7.12.9编译hadoop[root@hadoop01hadoop-3.3.4-src]#pwd/opt/hadoop/hadoop-3.3.4-src[root@hadoop01hadoop-3.3.4-src]#exportMAVEN_OPTS="-xms3072m-Xmx3072m"&&mvncleanpackage-Pdist,native-DskipTests-Dtar-Dbundle.snappy-Dsnappy.lib=/usr/local/lib-e这里的mvn命令也可以从BUILDING.txt文件中获取2.10编译后的安装包路径hadoop-3.3.4-src/hadoop-dist/target/hadoop-3.3.4.tar.gz2.11检测native[root@hadoop01hadoop]#tar-zxvfhadoop-3.3.4.tar.gz[root@hadoop01bin]#cdhadoop-3.3.4/bin[root@hadoop01bin]#./hadoopchecknative-a2023-02-1816:58:39,698INFObzip2.Bzip2Factory:成功加载并初始化native-bzip2librarysystem-native2023-02-1816:58:39,700INFOzlib.ZlibFactory:成功加载和初始化native-zliblibrary2023-02-1816:58:39,700WARNerasurecode.ErasureCodeNative:ISA-L支持不可用你的平台...在适用的地方使用内置java编解码器2023-02-1816:58:39,760INFOnativeio.NativeIO:本机代码是在没有PMDK支持的情况下构建的。本机库检查:hadoop:true/opt/hadoop/hadoop-3.3.4/lib/native/libhadoop.so.1.0.0zlib:true/lib64/libz.so.1zstd:falsebzip2:true/lib64/libbz2.so.1openssl:true/lib64/libcrypto.soISA-L:falselibhadoop是在没有ISA-L支持的情况下构建PMDK:falseThenativecodewasbuiltwithoutPMDKsupport.2023-02-1816:58:39,764INFOutil.ExitUtil:Exitingwithstatus1:ExitException可以看到上面还有很多false，但这不影响使用hadoop如果你想解决的话，你可以安装这些依赖，然后重新编译hadoop。3.参考文章1.https://www.vvave.net/archives/how-to-build-hadoop-334-native-libraries-full-kit-on-amd64.html2.https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Java+版本
