目錄一、hadoop單機模式1.1 部署1.2 HDFS創(chuàng)建目錄上傳測試文件二、完全分布式2.1 server1部署NFS共享服務(wù)2.2 修改Hadoop核心配置文件2.3 server2、server3掛載NFS共享目錄2.4 格式化2.5 HDFS集群測試創(chuàng)建目錄上傳文件三、集群節(jié)點擴容一、hadoop單機模式1.1 部署官方安裝網(wǎng)址Index of /hadoop/common官方指導(dǎo)文件Apache Hadoop 3.3.5 – Hadoop: Setting up a Single Node Cluster.下載安裝軟件包解壓縮并創(chuàng)建軟連接創(chuàng)建一個普通用戶hadoop并登入tar zxf hadoop-3.3.6.tar.gztar zxf jdk1.8.0_181-linux-x64.tar.gzln -s hadoop-3.3.6 hadoopln -s jdk1.8.0_181/ jdk配置Java環(huán)境變量編輯用戶家目錄.bashrc只對hadoop這個用戶生效hadoop?env.sh指定JAVA_HOME[hadoopserver1 hadoop]$ cd etc[hadoopserver1 etc]$ lshadoop[hadoopserver1 etc]$ cd hadoop[hadoopserver1 hadoop]$ vim hadoop-env.sh把本地xml配置文件復(fù)制到本地input文件夾[hadoopserver1 hadoop]$ cp etc/hadoop/*.xml input運行官方grep示例MR程序[hadoopserver1 hadoop]$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.6.jar grep input output dfs[a-z.][hadoopserver1 hadoop]$ cat output/*[hadoopserver1 hadoop]$ vim etc/hadoop/core-site.xmlfs.defaultFSHDFS文件系統(tǒng)默認地址客戶端訪問HDFS的入口單機寫localhost:9000[hadoopserver1 hadoop]$ vim etc/hadoop/hdfs-site.xmlhdfs?site.xml副本數(shù)配置免密對loaclhost這里就不詳細展示格式化NameNode格式化會清空HDFS全部數(shù)據(jù)已經(jīng)存數(shù)據(jù)的集群絕對不能重復(fù)執(zhí)行[hadoopserver1 hadoop]$ bin/hdfs namenode -format生成NameNode元數(shù)據(jù)目錄初始化fsimage鏡像文件建立HDFS文件系統(tǒng)。啟動HDFS[hadoopserver1 hadoop]$ sbin/start-dfs.sh9870端口瀏覽器正常訪問1.2 HDFS創(chuàng)建目錄上傳測試文件把本地文件上傳分布式文件系統(tǒng) HDFS給 MapReduce 任務(wù)做輸入數(shù)據(jù)源。[hadoopserver1 hadoop]$ bin/hdfs dfs -mkdir /user[hadoopserver1 hadoop]$ bin/hdfs dfs -mkdir /user/hadoop[hadoopserver1 hadoop]$ bin/hdfs dfs -mkdir input[hadoopserver1 hadoop]$ bin/hdfs dfs -put etc/hadoop/*.xml input先刪除本地舊outpu交mr任務(wù)讀取hdfs input目錄正則匹配dfs[a?z.]輸出到hdfs output[hadoopserver1 hadoop]$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.6.jar grep input output dfs[a-z.][hadoopserver1 hadoop]$ rm -rf output[hadoopserver1 hadoop]$ bin/hdfs dfs -get output output[hadoopserver1 hadoop]$ bin/hdfs dfs -cat output/*二、完全分布式2.1 server1部署NFS共享服務(wù)server1和server2、server3上安裝nfs-utils安裝nfs服務(wù)組件實現(xiàn)目錄網(wǎng)絡(luò)共享server2、server3可以掛載server1的/home/hadoop。yum install -y nfs-utils[rootserver1 etc]# vim exports[rootserver1 etc]# systemctl enable --now nfs[rootserver1 etc]# showmount -e查看本機已經(jīng)共享出來的目錄驗證exports配置是否生效cd hadoop/etc/hadoop2.2 修改Hadoop核心配置文件[hadoopserver1 hadoop]$ vim core-site.xmlHDFS 主節(jié)點地址完全分布式必須寫NameNode主機名 server1不能寫[localhost]。集群所有 DataNode 通過這個地址去找 NameNode。[hadoopserver1 hadoop]$ vim hdfs-site.xml當前集群有2臺DataNodeserver2、server3副本數(shù)設(shè)置2[hadoopserver1 hadoop]$ vim workers填入dataNode主機名2.3 server2、server3掛載NFS共享目錄[rootserver2 ~]# mount 192.168.159.133:/home/hadoop/ /home/hadoop/2.4 格式化停止舊服務(wù)[hadoopserver1 hadoop]$ sbin/stop-dfs.sh[hadoopserver1 hadoop]$ bin/hdfs namenode -format啟動完全分布式 HDFS 集群[hadoopserver1 hadoop]$ sbin/start-dfs.sh2.5 HDFS集群測試創(chuàng)建目錄上傳文件[hadoopserver1 hadoop]$ bin/hdfs dfs -mkdir /user[hadoopserver1 hadoop]$ bin/hdfs dfs -mkdir /user/hadoop[hadoopserver1 hadoop]$ bin/hdfs dfs -mkdir input[hadoopserver1 hadoop]$ bin/hdfs dfs -put etc/hadoop/*.xml input掛載成功后也會有server1的文件等三、集群節(jié)點擴容原有集群server1 (NameNodeResourceManager)、server2 (DataNodeNodeManager)、server3 (DataNodeNodeManager)擴容目標新增server4同時作為DataNodeHDFS 數(shù)據(jù)節(jié)點 NodeManagerYARN計算節(jié)點環(huán)境沿用NFS共享/home/hadoopserver4掛載之后直接復(fù)用jdk、hadoop整套程序不用重新解壓部署hadoop。新開一臺虛擬機server4初始化新節(jié)點 server4[rootserver4 ~]# useradd hadoop[rootserver4 ~]# passwd hadoop[rootserver4 ~]# mount 192.168.159.133:/home/hadoop/ /home/hadoop/server1修改MapReduce配置mapred?site.xml[hadoopserver1 hadoop]$ vim etc/hadoop/mapred-site.xmlserver1修改MapReduce配置yarn?site.xml[hadoopserver1 hadoop]$ vim etc/hadoop/yarn-site.xmlserver4手動后臺啟動DataNode、NodeManager[rootserver4 ~]# su - hadoop[hadoopserver4 ~]$ hdfs --daemon start datanode[hadoopserver4 ~]$ yarn --daemon start nodemanagerserver1更新workers文件啟動YARN集群查看集群節(jié)點狀態(tài)[hadoopserver1 hadoop]$ vim workers[hadoopserver1 hadoop]$ sbin/start-yarn.sh[hadoopserver1 hadoop]$ hdfs dfsadmin -report[hadoopserver1 hadoop]$ vim etc/hadoop/yarn-site.xml新加入兩行內(nèi)容server2也需要啟動[hadoopserver3 ~]$ yarn --daemon start nodemanager添加成功瀏覽器訪問192.168.159.133:8088端口也就是server1