环境:
OS:RadHat Linux As5
CDH:5.3.2
1.安装步骤
1.1 节点角色分配
| 节点角色 | ip | hostname |
| 主节点 | 192.168.56.111 | host01 |
| 从节点 | 192.168.56.112 | host02 |
这里只有两个节点的环境，一个做主节点，另外一个做从节点
1.2配置hosts文件
每台机器分配了ip和设定了主机名后,需要在hosts文件下添加如下项
[root@host01 /]#more /etc/hosts
# Do not removethe following line, or various programs
# that requirenetwork functionality will fail.
127.0.0.1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6
192.168.56.111 host01.com host01
192.168.56.112 host02.com host02
1.3 建立等效连接
[root@host01~]$ cd ~
[root@host01~]$ mkdir ~/.ssh
[root@host01~]$ chmod 700 ~/.ssh
[root@host01~]$ ssh-keygen -t rsa
Generatingpublic/private rsa key pair.
Enter file inwhich to save the key (/home/root/.ssh/id_rsa):
Enter passphrase(empty for no passphrase):
Enter samepassphrase again:
Youridentification has been saved in /home/root/.ssh/id_rsa.
Your public keyhas been saved in /home/root/.ssh/id_rsa.pub.
The keyfingerprint is:
84:88:f0:dc:7c:a6:d0:e8:68:65:53:97:35:02:05:fb root@host01
[root@host01~]$ ssh-keygen -t dsa
Generatingpublic/private dsa key pair.
Enter file inwhich to save the key (/home/root/.ssh/id_dsa):
Enter passphrase(empty for no passphrase):
Enter samepassphrase again:
Youridentification has been saved in /home/root/.ssh/id_dsa.
Your public keyhas been saved in /home/root/.ssh/id_dsa.pub.
The key fingerprintis:
94:80:60:37:38:e6:aa:10:22:50:1a:c2:42:56:22:0a root@host01
[root@host01.ssh]$ chmod 644 ~/.ssh /authorized_keys
[root@host02~]$ cd ~[root@host02~]$ mkdir ~/.ssh
[root@host02~]$ chmod 700 ~/.ssh
[root@host02~]$ ssh-keygen -t rsa
Generatingpublic/private rsa key pair.
Enter file inwhich to save the key (/home/root/.ssh/id_rsa):
Enter passphrase(empty for no passphrase):
Enter samepassphrase again:
Youridentification has been saved in /home/root/.ssh/id_rsa.
Your public keyhas been saved in /home/root/.ssh/id_rsa.pub.
The keyfingerprint is:
00:ee:68:42:3a:16:a3:75:6c:d4:36:49:57:a5:5c:57root@host02
[root@host02~]$ ssh-keygen -t dsa
Generatingpublic/private dsa key pair.
Enter file inwhich to save the key (/home/root/.ssh/id_dsa):
Enter passphrase(empty for no passphrase):
Enter samepassphrase again:
Youridentification has been saved in /home/root/.ssh/id_dsa.
Your public keyhas been saved in /home/root/.ssh/id_dsa.pub.
The keyfingerprint is:
21:47:cd:13:d2:17:a0:cb:c9:a0:c5:fc:39:3e:c4:bbroot@host02
[root@host01~]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
[root@host01~]$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
[root@host01~]$ ssh host02 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
The authenticityof host 'host02 (192.168.56.102)' can't be established.
RSA keyfingerprint is 5b:13:97:1a:0c:4d:36:93:7b:b5:87:2f:ac:34:26:1f.
Are you sure youwant to continue connecting (yes/no)? yes
Warning:Permanently added 'host02,192.168.56.102' (RSA) to the list of known hosts.
root@host02'spassword:
[root@host01~]$ ssh host02 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
root@host02'spassword:
[root@host01~]$ scp ~/.ssh/authorized_keys host02:~/.ssh/authorized_keys
root@host02'spassword:
authorized_keys 100% 1992 2.0KB/s 00:00
[root@host01.ssh]$ chmod 644 ~/.ssh /authorized_keys
1.4关闭防护墙
service iptablesstop
chkconfig iptablesoff
关闭SELINUX:
修改 /etc/selinux/config下的SELINUX=disabled (重启后永久生效)
在所有的机器上做同样的操作
1.5 安装jdk
目前系统的jdk是1.4的,需要安装1.6以上的版本
[hadoop@node1 ~]$java -version
java version"1.4.2"
gij (GNU libgcj)version 4.1.2 20080704 (Red Hat 4.1.2-51)
可以到oracle官网下载jdk,下载地址下载地址：http://www.oracle.com/technetwork/java/javase/index.html
我这里下载的是jdk-8u5-linux-i586.rpm
我这里下载的是rpm安装文件，可以先查看rpm包中java的安装路径
[root@node1 soft]#rpm -qpl jdk-8u5-linux-i586.rpm
/usr/java/jdk1.8.0_05/man/man1/javap.1
/usr/java/jdk1.8.0_05/man/man1/javaws.1
/usr/java/jdk1.8.0_05/man/man1/jcmd.1
/usr/java/jdk1.8.0_05/man/man1/jconsole.1
/usr/java/jdk1.8.0_05/man/man1/jdb.1
/usr/java/jdk1.8.0_05/man/man1/jdeps.1
/usr/java/jdk1.8.0_05/man/man1/jhat.1
/usr/java/jdk1.8.0_05/man/man1/jinfo.1
/usr/java/jdk1.8.0_05/man/man1/jjs.1
/usr/java/jdk1.8.0_05/man/man1/jmap.1
/usr/java/jdk1.8.0_05/man/man1/jmc.1
/usr/java/jdk1.8.0_05/man/man1/jps.1
/usr/java/jdk1.8.0_05/man/man1/jrunscript.1
/usr/java/jdk1.8.0_05/man/man1/jsadebugd.1
/usr/java/jdk1.8.0_05/man/man1/jstack.1
/usr/java/jdk1.8.0_05/man/man1/jstat.1
/usr/java/jdk1.8.0_05/man/man1/jstatd.1
/usr/java/jdk1.8.0_05/man/man1/jvisualvm.1
/usr/java/jdk1.8.0_05/man/man1/keytool.1
/usr/java/jdk1.8.0_05/man/man1/native2ascii.1
/usr/java/jdk1.8.0_05/man/man1/orbd.1
/usr/java/jdk1.8.0_05/man/man1/pack200.1
/usr/java/jdk1.8.0_05/man/man1/policytool.1
/usr/java/jdk1.8.0_05/man/man1/rmic.1
/usr/java/jdk1.8.0_05/man/man1/rmid.1
/usr/java/jdk1.8.0_05/man/man1/rmiregistry.1
/usr/java/jdk1.8.0_05/man/man1/schemagen.1
/usr/java/jdk1.8.0_05/man/man1/serialver.1
/usr/java/jdk1.8.0_05/man/man1/servertool.1
/usr/java/jdk1.8.0_05/man/man1/tnameserv.1
/usr/java/jdk1.8.0_05/man/man1/unpack200.1
/usr/java/jdk1.8.0_05/man/man1/wsgen.1
/usr/java/jdk1.8.0_05/man/man1/wsimport.1
/usr/java/jdk1.8.0_05/man/man1/xjc.1
/usr/java/jdk1.8.0_05/release
/usr/java/jdk1.8.0_05/src.zip
部分输出，可以看出直接安装rpm包的话，java会默认安装在/usr/java/目录下,为hadoop设置环境变量java_home的时候可以指定该目录,不指定的话,系统会默认去找系统默认安装的java目录。
使用如下的命令直接安装java
[root@node1 soft]#rpm -ivh jdk-8u5-linux-i586.rpm
Preparing... ########################################### [100%]
1:jdk ########################################### [100%]
Unpacking JARfiles...
rt.jar...
jsse.jar...
charsets.jar...
tools.jar...
localedata.jar...
jfxrt.jar...
plugin.jar...
javaws.jar...
deploy.jar...
安装完成后，为hadoop设定环境变量,如下:
[hadoop@node1 ~]$more .bash_profile
# .bash_profile
# Get the aliases andfunctions
if [ -f ~/.bashrc ];then
. ~/.bashrc
fi
# User specificenvironment and startup programs
export JAVA_HOME=/usr/java/jdk1.8.0_05
export JRE_HOME=/usr/java/jdk1.8.0_05/jre
export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib:$JRE_HOME/lib
exportPATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH
1.6 安装mysql
这里安装过程省略，具体可以在网上找安装文档，安装完成后确保设置mysql为自动启动.
设置开机启动:chkconfig mysqld on
启动mysql服务:service mysqld start
创建用户
grant ALL on *.*to root@'192.168.56.%' identified by 'hadoop' WITH GRANT OPTION;
grant ALL on *.*to root@'host01' identified by 'hadoop' WITH GRANT OPTION;
grant ALL on *.*to scm@'192.168.56.%' identified by 'scm' WITH GRANT OPTION;
grant ALL on *.*to scm@'host01' identified by 'scm' WITH GRANT OPTION;
flush privileges;
创建数据库
#hive
create database hiveDEFAULT CHARSET utf8 COLLATE utf8_general_ci;
#activity monitor
create databaseamon DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
#hue
create databasehue DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
1.6 安装Cloudera Manager Server和Agent
1.6.1 下载安装介质
到官网下载cloudera-manager,下载地址为http://archive-primary.cloudera.com/cm5/cm/5/
我这里下载的是cloudera-manager-el5-cm5.3.2_x86_64.tar.gz
CDH安装包地址：http://archive.cloudera.com/cdh5/parcels/5.3.2/
我这里下载的是
CDH-5.3.2-1.cdh5.3.2.p0.10-el5.parcel
CDH-5.3.2-1.cdh5.3.2.p0.10-el5.parcel.sha1
manifest.json
1.6.2 主节点解压安装
将下载下来的cloudera-manager-el5-cm5.3.2_x86_64.tar.gz上传到主节点/opt目录下，执行解压缩命令
tar –xvf cloudera-manager-el5-cm5.3.2_x86_64.tar.gz
1.6.3 Agent配置
修改/opt/cm-5.3.2/etc/cloudera-scm-agent/config.ini的server_host为主节点的主机名
如下：
[General]
# Hostname of the CM server.
server_host=host01
1.6.4 同步到其他节点
scp -r /opt/cm-5.3.2/ root@host02:/opt/
1.6.5 创建cloudera-scm用户
在每个节点上执行如下的命令：
useradd--home=/opt/cm-5.3.2/run/cloudera-scm-server/ --shell=/bin/false --comment"Cloudera SCM User" cloudera-scm
1.6.6为Cloudera Manager建立数据库
将mysql-connector-java-5.1.30.jar放到/opt/cm-5.3.2/share/cmf/lib/中。
在主节点中初始化CM5的数据库：
root@host01 cloudera]#/opt/cm-5.3.2/share/cmf/schema/scm_prepare_database.sh mysql cm -h127.0.0.1 -uroot -phadoop --scm-host 127.0.0.1 scm scm scm
1.6.7 准备Parcels,用以安装CDH5
将如下3个文件copy到主节点的/opt/cloudera/parcel-repo/目录下
CDH-5.3.2-1.cdh5.3.2.p0.10-el5.parcel
CDH-5.3.2-1.cdh5.3.2.p0.10-el5.parcel.sha1
manifest.json
然后将CDH-5.3.2-1.cdh5.3.2.p0.10-el5.parcel.sha1重命名为CDH-5.3.2-1.cdh5.3.2.p0.10-el5.parcel.sha
1.6.8 启动服务
/opt/cm-5.3.2/etc/init.d/cloudera-scm-serverstart启动服务端(主节点)
/opt/cm-5.3.2/etc/init.d/cloudera-scm-agentstart启动Agent服务(所有节点)
1.6.9 安装cdh
在IE栏中输入地址http://192.168.56.111:7180/cmf/login，第一次登陆估计要等待一段时间，最后会弹出登陆提示框：
验证：
1.使用hdfs用户登录创建一个目录
hdfs用户在安装cdh的过程自动创建的用户，不需要用户手工创建
[root@host01 /]# su - hdfs
[hdfs@host01 ~]$ hdfs dfs -mkdir -p /user/hadoop