博客
关于我
强烈建议你试试无所不能的chatGPT,快点击我
安装hadoop+zookeeper
阅读量:5743 次
发布时间:2019-06-18

本文共 19835 字,大约阅读时间需要 66 分钟。

安装hadoop+zookeeper ha前期工作配置好网络和主机名和关闭防火墙chkconfig iptables off //关闭防火墙1.安装好java并配置好相关变量 (/etc/profile)#javaexport JAVA_HOME=/usr/java/jdk1.8.0_65export JRE_HOME=$JAVA_HOME/jreexport PATH=$PATH:$JAVA_HOME/binexport CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar (最前面要有.)保存退出source /etc/profile2.设置好主机名和网络映射关系 (/etc/hosts) // hadoop.master为namenode // hadoop.slaver1/hadoop.slaver2/hadoop.slaver3 为datanode192.168.22.241 hadoop.master192.168.22.242 hadoop.slaver1192.168.22.243 hadoop.slaver2192.168.22.244 hadoop.slaver33.创建用户并创建密码(以root身份登陆)  1. useradd hadoop(或者其他用户名)  2. passwd hadoop (回车输入密码 两次)  3. su hadoop (使用hadoop用户登陆)  4.免密码登陆    1.安装ssh  具体百度  一般都自带有    2.创建在家目录底下创建.ssh目录(使用hadoop用户)  mkdir ~/.ssh    3.创建公钥(namenode端运行)        ssh-keygen -t rsa        一路回车        最后会在~/.ssh目录下生成id_rsa、id_rsa.pub  其中前者是密钥 后者是公钥    4.将id_rsa.pub文件拷贝到slaver节点的相同用户.ssh目录下        scp -r id_rsa.pub 用户名@主机名:目标文件(含路径)    5.在各个子节点执行cat id_rsa.pub >> ~/.ssh/authorized_keys    6.设置权限        chmod 600 authorized_keys        cd ..        chmod 700 -R .ssh    7.注意此时还不能免密码  需在master 节点运行ssh slaver 输入密码后才能免密码5.安装zookeeper(三台 master slaver1 slaver2)    1.下载安装包    2.解压安装包        tar zxvf zookeeper-3.4.7.tar.gz    3.配置环境变量        #zookeeper        export ZOOKEEPER_HOME=/opt/zookeeper-3.4.7        export PATH=$PATH::$ZOOKEEPER_HOME/bin:$ZOOKEEPER_HOME/conf        保存退出        source /etc/profile    4.修改配置文件        cp zoo_sample.cfg zoo.cfg        vim zoo.cfg        ####zoo.cfg####        tickTime=2000        initLimit=10        syncLimit=5        dataDir=/opt/zookeeper-3.4.7/tmp/zookeeper (注意创建相关目录)        clientPort=2181        server.1=hadoop.master:2888:3888        server.2=hadoop.slaver1:2888:3888        server.3=hadoop.slaver2:2888:3888                参数说明:        tickTime: zookeeper中使用的基本时间单位, 毫秒值.        dataDir: 数据目录. 可以是任意目录.        dataLogDir: log目录, 同样可以是任意目录. 如果没有设置该参数, 将使用和dataDir相同的设置.        clientPort: 监听client连接的端口号.        initLimit: zookeeper集群中的包含多台server, 其中一台为leader, 集群中其余的server为follower.        syncLimit: 该参数配置leader和follower之间发送消息, 请求和应答的最大时间长度.         server.X=A:B:C 其中X是一个数字, 表示这是第几号server. A是该server所在的IP地址. B配置该server和集群中的leader交换消息所使用的端口. C配置选举leader时所使用的端口.     5.分发到各个节点中       scp -r /opt/zookeeper-3.4.7 hadoop@主机名:/opt    6.根据dataDir配置的目录下新建myid文件, 写入一个数字, 该数字表示这是第几号server       cd /opt/zookeeper-3.4.7/tmp/zookeeper       touch myid(如果是安装上述配置,则master为1 slaver1为2 slaver3)    7.常用命令        ####启动/关闭/查看 zk#####        zkServer.sh start    //集群中每台主机执行一次        zkServer.sh stop        zkServer.sh status        ####查看/删除节点信息####        zkCli.sh        ls /        rmr /节点名称6.安装hadoop(四台机子 master slaver1 slaver2 slaver3 其中namenode有master和slaver1)    1.下载安装包    2.解压安装包    3.配置环境变量        #hadoop        export HADOOP_HOME=/opt/hadoop-2.5.2        export HADOOP_PREFIX=/opt/hadoop-2.5.2        export HADOOP_COMMON_HOME=$HADOOP_HOME        export HADOOP_MAPRED_HOME=$HADOOP_HOME        export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop        export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native        export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"        export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native                export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HADOOP_HOME/lib        export CLASSPATH=.:$CLASSPATH:$HADOOP_HOME/bin                保存退出        source /etc/profile    4.修改配置文件        1.创建相关目录            cd /opt/hadoop-2.5.2            mkdir logs            mkdir tmp        2.修改相关配置文件相关参数(core-site.xml/hadoop-env.sh/hdfs-site.xml/log4j.properties        /mapred-env.sh/mapred-site.xml/masters/slaves/yarn-env.sh/yarn-site.xml)                    ####core-site.xml####            
            
            
                
fs.defaultFS
                
hdfs://ns1:8020
            
                         
            
                
io.file.buffer.size
                
131072
            
                         
            
                
hadoop.tmp.dir
                
/opt/hadoop-2.5.2/tmp
                
A base for other temporary directories.
            
                        
            
                
ha.zookeeper.quorum
                
hadoop.master:2181,hadoop.slaver1:2181,hadoop.slaver2:2181
            
            
                        ####hadoop-env.sh####            export JAVA_HOME=/usr/java/jdk1.8.0_65            export HADOOP_CLASSPATH=.:$HADOOP_CLASSPATH:$HADOOP_HOME/bin            export CLASSPATH=.:$CLASSPATH:$HADOOP_HOME/bin                        ####hdfs-site.xml####            
            
            
dfs.namenode.http-address
            
hadoop.master:50070
            
The address and the base port where the dfs namenode web ui will listen on.
            
            
            
dfs.namenode.secondary.http-address
            
hadoop.slaver1:50070
            
            
            
dfs.namenode.checkpoint.dir
            
file://${hadoop.tmp.dir}/dfs/namesecondary
            
true
            
            
            
dfs.namenode.name.dir
            
file://${hadoop.tmp.dir}/dfs/name
            
true
            
            
            
dfs.datanode.data.dir
            
file://${hadoop.tmp.dir}/dfs/data
            
true
            
            
            
dfs.replication
            
3
            
            
            
dfs.permissions
            
false
            
            
            
dfs.permissions.enabled
            
false
            
            
            
dfs.namenode.hosts.exclude
            
/opt/hadoop-2.5.2/other/excludes
            
Names a file that contains a list of hosts that are not permitted to connect to the namenode.  The full pathname of the file must be specified.  If the value is empty, no hosts are excluded.
            
            
            
dfs.namenode.hosts
            
/opt/hadoop-2.5.2/etc/hadoop/slaves
            
            
            
dfs.blocksize
            
134217728
            
            
            
             
dfs.datanode.max.xcievers
             
4096
             
            
            
            
dfs.nameservices
            
ns1
            
            
            
dfs.ha.namenodes.ns1
            
nn1,nn2
            
            
            
dfs.namenode.rpc-address.ns1.nn1
            
hadoop.master:8020
            
            
            
dfs.namenode.rpc-address.ns1.nn2
            
hadoop.slaver1:8020
            
            
            
dfs.namenode.http-address.ns1.nn1
            
hadoop.master:50070
            
            
            
dfs.namenode.http-address.ns1.nn2
            
hadoop.slaver1:50070
            
                        
            
dfs.namenode.servicerpc-address.ns1.nn1
            
hadoop.master:53310
            
            
            
dfs.namenode.servicerpc-address.ns1.nn2
            
hadoop.slaver1:53310
            
             
            
            
dfs.journalnode.edits.dir
            
/opt/zookeeper-3.4.7/journal
            
            
            
dfs.namenode.shared.edits.dir
            
qjournal://hadoop.master:8485;hadoop.slaver1:8485;hadoop.slaver2:8485/ns1
            
            
            
            
dfs.ha.automatic-failover.enabled
            
true
            
            
            
            
dfs.client.failover.proxy.provider.ns1
            
org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
            
             
            
            
ha.zookeeper.quorum
            
hadoop.master:2181,hadoop.slaver1:2181,hadoop.slaver2:2181
            
            
            
            
dfs.ha.fencing.methods
            
            sshfence            shell(/bin/true)            
            
            
            
dfs.ha.fencing.ssh.private-key-files
            
/home/hadoop/.ssh/id_rsa
            
            
            
            
dfs.ha.fencing.ssh.connect-timeout
            
30000
            
             
                        ####log4j.properties####            hadoop.root.logger=INFO,console            hadoop.log.dir=/opt/hadoop-2.5.2/logs            hadoop.log.file=hadoop.log                        ####mapred-env.sh####            export HADOOP_JOB_HISTORYSERVER_HEAPSIZE=1000            export HADOOP_MAPRED_ROOT_LOGGER=INFO,RFA            ####mapred-site.xml####            
             
                
mapreduce.framework.name
                
yarn
            
                
                
mapreduce.application.classpath
                
                /opt/hadoop-2.5.2/etc/hadoop,                /opt/hadoop-2.5.2/share/hadoop/common/*,                /opt/hadoop-2.5.2/share/hadoop/common/lib/*,                /opt/hadoop-2.5.2/share/hadoop/hdfs/*,                /opt/hadoop-2.5.2/share/hadoop/hdfs/lib/*,                /opt/hadoop-2.5.2/share/hadoop/mapreduce/*,                /opt/hadoop-2.5.2/share/hadoop/mapreduce/lib/*,                /opt/hadoop-2.5.2/share/hadoop/yarn/*,                /opt/hadoop-2.5.2/share/hadoop/yarn/lib/*                
            
            
                
mapreduce.jobhistory.address
                
hadoop.master:10020
            
            
                
mapreduce.jobhistory.webapp.address
                
hadoop.master:19888
            
            
                    
mapreduce.jobhistory.done-dir
                    
/history/done
            
            
               
mapreduce.jobhistory.intermediate-done-dir
               
/history/done_intermediate
            
            
            ####masters####            hadoop.slaver1  //存储secondary namenode节点主机名                        ####slaves####            hadoop.slaver1            hadoop.slaver2            hadoop.slaver3                        ####yarn-env.sh####            export JAVA_HOME=/usr/java/jdk1.8.0_65                        ####yarn-site.xml####            
            
            
            
yarn.resourcemanager.address
            
hadoop.master:18040
            
            
            
yarn.resourcemanager.scheduler.address
            
hadoop.master:18030
            
            
            
yarn.resourcemanager.resource-tracker.address
            
hadoop.master:18025
            
            
            
yarn.resourcemanager.admin.address
            
hadoop.master:18041
            
            
            
yarn.resourcemanager.webapp.address
            
hadoop.master:8088
            
            
            
yarn.nodemanager.local-dirs
            
/opt/hadoop-2.5.2/other/mynode
            
            
            
yarn.nodemanager.log-dirs
            
/opt/hadoop-2.5.2/other/logs
            
            
            
yarn.nodemanager.log.retain-seconds
            
10800
            
            
            
yarn.nodemanager.remote-app-log-dir
            
/opt/hadoop-2.5.2/other/logs
            
            
            
yarn.nodemanager.remote-app-log-dir-suffix
            
logs
            
            
            
yarn.log-aggregation.retain-seconds
            
-1
            
            
            
yarn.log-aggregation.retain-check-interval-seconds
            
-1
            
            
            
yarn.nodemanager.aux-services
            
mapreduce_shuffle
            
            
            
            
yarn.resourcemanager.ha.enabled
            
true
            
            
            
yarn.resourcemanager.cluster-id
            
yrc
            
            
            
yarn.resourcemanager.ha.rm-ids
            
rm1,rm2
            
            
            
yarn.resourcemanager.hostname.rm1
            
hadoop.master
            
            
            
yarn.resourcemanager.hostname.rm2
            
hadoop.slaver1
            
            
            
yarn.resourcemanager.zk-address
            
hadoop.master:2181,hadoop.slaver1:2181,hadoop.slaver2:2181
            
            
            
yarn.nodemanager.aux-services
            
mapreduce_shuffle
            
            
    5.分发到各个节点中       scp -r /opt/hadoop-2.5.2 hadoop@hadoop.master:/opt        6.首次启动        6.1 启动zk            zkServer.sh start(zk 各个节点执行)        6.2 启动journalnode            hadoop-daemon.sh start journalnode(zk 各个节点执行)        6.3 格式化Namenode            hadoop namenode -format(namenode 节点运行  注意是hadoop  不是hdfs)        6.4 启动Namenode            hadoop-daemon.sh start namenode(namenode 节点运行)        6.5 格式化另一个Namenode            hadoop namenode -bootstrapStandby(在secondary namenode节点运行)        6.6 格式化zk             hdfs zkfc -formatZK (namenode节点执行)        6.7 将所有的服务停止            stop-all.sh            注意此时需在每个zk节点执行 zkServer.sh stop    7.正常启动        1.启动zk            zkServer.sh start(zk 各个节点执行)        2.启动所有服务            start-all.sh   //或者先执行start-dfs.sh   再执行start-yarn.sh        3.启动后台历史服务            mr-jobhistory-daemon.sh start historyserver(在namenode节点执行即可)        4.启动备份resourcemanger            yarn-daemon.sh start resourcemanager  //在备份节点运行        5.启动备份namenode            hadoop-daemon.sh start namenode  //在备份节点运行                8.验证        1.jps验证 查看相关进程        2.web验证            hdfs   主机名:50070            yarn   主机名:8088            history  主机名:19888            //以上主机名均指 namenode节点主机名 (此时namenode节点是active状态)        3.查看active状态            hdfs  web查看  有active状态和stangby状态两种            yarn  shell命令查看                  yarn rmadmin -getServiceState rm1(或者rm2)                //其中rm1/rm2为配置文件中配置的名称        4.kill当前active的namenode 看能不自己切换到standby namenode上    9.常见命令         ####启动/关闭yarn jobhistory记录####         web: //namenode:19888  //其中namenode 为集群任意节点主机名         mr-jobhistory-daemon.sh start historyserver  //集群中每台主机执行一次         mr-jobhistory-daemon.sh stop historyserver                  ####启动/关闭/查看 zk#####         zkServer.sh start    //集群中每台主机执行一次         zkServer.sh stop         zkServer.sh status                  ####启动/关闭/查看 yarn####         yarn-daemon.sh start resourcemanager         yarn-daemon.sh stop resourcemanager         yarn-daemon.sh stop nodemanager         yarn rmadmin -getServiceState rm2  //其中rm2是集群配置的别名                  web: //namenode:8088  //其中namenode是active状态的主机名                  ####启动/关闭/查看 hadoop####         hadoop-daemon.sh start namenode         hadoop-daemon.sh stop namenode         hadoop-daemon.sh stop datanode         web: //namenode:50070  //其中namenode是active状态的主机名                  ####格式化zkNode####          hdfs zkfc -formatZK //namenode节点执行   注意是hdfs  不是hadoop                  ####启动/关闭zkNode#####         hadoop-daemon.sh start zkfc         hadoop-daemon.sh stop zkfc                  ####查看/删除job####         hadoop job -list         hadoop job -kill 任务ID //注意不是applicationID                  ####初始化Journal Storage Directory####         hdfs namenode -initializeSharedEdits  //非ha转成ha时执行 如果一开始已经是ha了无需执行                  ####初始化namenode####         hadoop namenode -format  //namenode端执行                  hdfs namenode -bootstrapStandby //secend namenode端执行 执行前需保证namenode已经启动            10.常见异常        1.Journal Storage Directory /opt/zookeeper-3.4.7/journal/ns1 not formatted            原因:由于之前hadoop没部署ha,改成ha后形成错误            解决办法:                    1.将配置文件hdfs-site.xml中dfs.journalnode.edits.dir对应的目录删除                    2.hdfs namenode -initializeSharedEdits(namenode 执行)        2.datanode起来了,namenode起不来            解决办法:                1.查看配置文件相关配置项是否配置正确                2.查看环境变量是否配置正确                3.查看主机网络映射是否配置正确                4.是否二次格式化namenode  如果是,则需要将datanode 的clusterID和namespaceID改成namenode一致                    目录一般是tmp目录下                5.重启hdfs                6.如果执行上述还不行,则在hadoop服务运行状态下将tmp目录下所有文件夹删除,再格式化,重启服务        3.两个namenode起来了,但都是standby状态            解决办法:                1.是否均启动zk                2.格式化zfkc                    hdfs zkfc -formatZK                3.所有服务重启(含zk)
本文转自 chengxuyonghu 51CTO博客,原文链接:http://blog.51cto.com/6226001001/1895949,如需转载请自行联系原作者
你可能感兴趣的文章
docker 下 安装rancher 笔记
查看>>
spring两大核心对象IOC和AOP(新手理解)
查看>>
数据分析相关
查看>>
Python LDAP中的时间戳转换为Linux下时间
查看>>
微信小程序蓝牙连接小票打印机
查看>>
环境错误2
查看>>
C++_了解虚函数的概念
查看>>
全新jmeter视频已经上架
查看>>
Windows 8下如何删除无线配置文件
查看>>
oracle系列(五)高级DBA必知的Oracle的备份与恢复(全录收集)
查看>>
hp 服务器通过串口重定向功能的使用
查看>>
国外10大IT网站和博客网站
查看>>
android第十一期 - SmoothSwitchLibrary仿IOS切换Activity动画效果
查看>>
zabbix 批量web url监控
查看>>
MongoDB CookBook读书笔记之导入导出
查看>>
shell如何快速锁定所有账号
查看>>
HTML 5实现的手机摇一摇
查看>>
Linux 文件IO理解
查看>>
Ninject 2.x细说---2.绑定和作用域
查看>>
30个非常时尚的网页联系表单设计优秀示例
查看>>