hadoop|hadoop（五） hdfs|大数据

目录
群起集群
一、启动集群
Hadoop群起脚本
LOZ压缩配置

群起集群一、启动集群如果是第一次启动，则需要格式化。

[doudou@hadoop101 hadoop-3.1.3]$ bin/hdfs namenode -format

[doudou@hadoop101 hadoop-3.1.3]$ sbin/start-dfs.sh

[doudou@hadoop101 hadoop-3.1.3]$ xcall.sh jps

hadoop102中

[doudou@hadoop102 hadoop-3.1.3]$ sbin/start-yarn.sh

Hadoop群起脚本编写脚本

[doudou@hadoop101 bin]$ vim hdp.sh

#!/bin/bashif [ $# -lt 1 ] then echo "No Args Input..." exit ; ficase $1 in "start") echo " =================== 启动 hadoop集群 ==================="echo " --------------- 启动 hdfs ---------------" ssh hadoop102 "/opt/module/hadoop-3.1.3/sbin/start-dfs.sh" echo " --------------- 启动 yarn ---------------" ssh hadoop103 "/opt/module/hadoop-3.1.3/sbin/start-yarn.sh" echo " --------------- 启动 historyserver ---------------" ssh hadoop102 "/opt/module/hadoop-3.1.3/bin/mapred --daemon start historyserver" ; ; "stop") echo " =================== 关闭 hadoop集群 ==================="echo " --------------- 关闭 historyserver ---------------" ssh hadoop102 "/opt/module/hadoop-3.1.3/bin/mapred --daemon stop historyserver" echo " --------------- 关闭 yarn ---------------" ssh hadoop103 "/opt/module/hadoop-3.1.3/sbin/stop-yarn.sh" echo " --------------- 关闭 hdfs ---------------" ssh hadoop102 "/opt/module/hadoop-3.1.3/sbin/stop-dfs.sh" ; ; *) echo "Input Args Error..." ; ; esac

查看启动的日志

[doudou@hadoop101 bin]$ xcall.sh jps ---------- 192.168.149.129 --------- 2552 DataNode 2430 NameNode 2895 NodeManager 3183 Jps ---------- 192.168.149.132 --------- 2261 DataNode 3224 Jps 2617 NodeManager 2442 ResourceManager ---------- 192.168.149.133 --------- 2571 Jps 2254 DataNode 2350 SecondaryNameNode

关闭日志

[doudou@hadoop101 bin]$ hdp.sh stop

查看关闭后日志

[doudou@hadoop101 bin]$ xcall.sh jps ---------- 192.168.149.129 --------- 3825 Jps ---------- 192.168.149.132 --------- 2617 NodeManager 3674 Jps ---------- 192.168.149.133 --------- 2758 Jps

启动日志

[doudou@hadoop101 bin]$ hdp.sh start

查看启动后日志

[doudou@hadoop101 bin]$ xcall.sh jps ---------- 192.168.149.129 --------- 4051 NameNode 4181 DataNode 4762 Jps 4523 NodeManager 4699 JobHistoryServer ---------- 192.168.149.132 --------- 3778 DataNode 3974 ResourceManager 2617 NodeManager 4621 Jps ---------- 192.168.149.133 --------- 3040 Jps 2861 DataNode 2974 SecondaryNameNode

LOZ压缩配置移动编译好的文件

[doudou@hadoop101 software]$ mv hadoop-lzo-0.4.20.jar /opt/module/hadoop-3.1.3/share/hadoop/common/ [doudou@hadoop101 software]$ cd /opt/module/hadoop-3.1.3/share/hadoop/common/ [doudou@hadoop101 common]$ ll total 7348 -rw-r--r--. 1 doudou root 4096811 Sep 112019 hadoop-common-3.1.3.jar -rw-r--r--. 1 doudou root 2878235 Sep 112019 hadoop-common-3.1.3-tests.jar -rw-r--r--. 1 doudou root129977 Sep 112019 hadoop-kms-3.1.3.jar -rw-r--r--. 1 doudou root193831 Aug7 06:56 hadoop-lzo-0.4.20.jar -rw-r--r--. 1 doudou root201616 Sep 112019 hadoop-nfs-3.1.3.jar drwxr-xr-x. 2 doudou root4096 Sep 112019 jdiff drwxr-xr-x. 2 doudou root4096 Sep 112019 lib drwxr-xr-x. 2 doudou root89 Sep 112019 sources drwxr-xr-x. 3 doudou root20 Sep 112019 webapps

分发

[doudou@hadoop101 common]$ xsync hadoop-lzo-0.4.20.jar

core-site.xml增加配置支持LZO压缩

[doudou@hadoop101 hadoop]$ vim core-site.xml

io.compression.codecs org.apache.hadoop.io.compress.GzipCodec, org.apache.hadoop.io.compress.DefaultCodec, org.apache.hadoop.io.compress.BZip2Codec, org.apache.hadoop.io.compress.SnappyCodec, com.hadoop.compression.lzo.LzoCodec, com.hadoop.compression.lzo.LzopCodec io.compression.codec.lzo.class com.hadoop.compression.lzo.LzoCodec

分发

[doudou@hadoop101 hadoop]$ xsync core-site.xml

重启集群

[doudou@hadoop101 hadoop]$ hdp.sh start

测试

[doudou@hadoop101 hadoop-3.1.3]$ hadoop fs -mkdir /input [doudou@hadoop101 hadoop-3.1.3]$ hadoop fs -put README.txt /input [doudou@hadoop101 hadoop-3.1.3]$ hadoop fs -put README.txt /input 2022-08-13 04:02:46,435 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false [doudou@hadoop101 hadoop-3.1.3]$ hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar wordcount -Dmapreduce.output.fileoutputformat.compress=true -Dmapreduce.output.fileoutputformat.compress.codec=com.hadoop.compression.lzo.LzopCodec/input /output

文章图片

将bigtable.lzo上传

[doudou@hadoop101 software]$ hadoop fs -put bigtable.lzo/input

执行wordcount程序

[doudou@hadoop101 software]$ hadoop jar /opt/module/hadoop-3.1.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar wordcount -Dmapreduce.job.inputformat.class=com.hadoop.mapreduce.LzoTextInputFormat /input /output1

对上传的LZO文件建索引

[doudou@hadoop101 software]$ hadoop jar /opt/module/hadoop-3.1.3/share/hadoop/common/hadoop-lzo-0.4.20.jarcom.hadoop.compression.lzo.DistributedLzoIndexer

再次执行wordcount程序

[doudou@hadoop101 software]$ hadoop jar /opt/module/hadoop-3.1.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar wordcount -Dmapreduce.job.inputformat.class=com.hadoop.mapreduce.LzoTextInputFormat /input /output2

【hadoop|hadoop（五）】