linux下安装hadoop
环境:
OS:Rad Hat Linux As5
Hadoop: 1.2.1
1.安装步骤
1.1 配置hosts文件
每台机器分配了ip和设定了主机名后,需要在hosts文件下添加如下项
[root@node4 ~]# more /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6
#Public
192.168.56.101 node1 node1
192.168.56.102 node2 node2
192.168.56.103 node3 node3
192.168.56.104 node4 node4
1.2 节点角色分配
节点角色 | ip | hostname |
Name node | 192.168.56.101 | node1 |
Datanode1 | 192.168.56.102 | node2 |
Datanode2 | 192.168.56.103 | node3 |
Datanode3 | 192.168.56.104 | node4 |
我们这里共4个节点，其中一个namenode节点，3个datanode节点.
1.3 创建hadoop用户和组
在每个节点上创建hadoop用户和hadoop 组，并设置hadoop用户密码
[root@node1 ~]# groupadd hadoop
[root@node2 ~]# groupadd hadoop
[root@node3~]# groupadd hadoop
[root@node4 ~]# groupadd hadoop
[root@node1 ~]# useradd -g hadoop -G hadoop hadoop
[root@node1 ~]# passwd hadoop
[root@node2 ~]# useradd -g hadoop -G hadoop hadoop
[root@node2 ~]# passwd hadoop
[root@node3 ~]# useradd -g hadoop -G hadoop hadoop
[root@node3 ~]# passwd hadoop
[root@node4 ~]# useradd -g hadoop -G hadoop hadoop
[root@node4 ~]# passwd hadoop
1.4 创建名称节点和各数据节点之间的等效性
名称节点和数据节点之间通信需要无密码登陆，所以需要名称结合和其他节点之间配置ssh无密码登陆连接，该步骤在hadoop用户下执行。
1.4.1 建立名称节点1到Datanode1的等效连接
[hadoop@node1 ~]$ cd ~
[hadoop@node1 ~]$ mkdir ~/.ssh
[hadoop@node1 ~]$ chmod 700 ~/.ssh
[hadoop@node1 ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
84:88:f0:dc:7c:a6:d0:e8:68:65:53:97:35:02:05:fb hadoop@node1
[hadoop@node1 ~]$ ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_dsa.
Your public key has been saved in /home/hadoop/.ssh/id_dsa.pub.
The key fingerprint is:
94:80:60:37:38:e6:aa:10:22:50:1a:c2:42:56:22:0a hadoop@node1
[hadoop@node1 .ssh]$ chmod 644 ~/.ssh /authorized_keys
[hadoop@node2 ~]$ cd ~
[hadoop@node2 ~]$ mkdir ~/.ssh
[hadoop@node2 ~]$ chmod 700 ~/.ssh
[hadoop@node2 ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
00:ee:68:42:3a:16:a3:75:6c:d4:36:49:57:a5:5c:57 hadoop@node2
[hadoop@node2 ~]$ ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_dsa.
Your public key has been saved in /home/hadoop/.ssh/id_dsa.pub.
The key fingerprint is:
21:47:cd:13:d2:17:a0:cb:c9:a0:c5:fc:39:3e:c4:bb hadoop@node2
[hadoop@node1 ~]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
[hadoop@node1 ~]$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
[hadoop@node1 ~]$ ssh node2 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
The authenticity of host 'node2 (192.168.56.102)' can't be established.
RSA key fingerprint is 5b:13:97:1a:0c:4d:36:93:7b:b5:87:2f:ac:34:26:1f.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'node2,192.168.56.102' (RSA) to the list of known hosts.
hadoop@node2's password:
[hadoop@node1 ~]$ ssh node2 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
hadoop@node2's password:
[hadoop@node1 ~]$ scp ~/.ssh/authorized_keys node2:~/.ssh/authorized_keys
hadoop@node2's password:
authorized_keys 100% 1992 2.0KB/s 00:00
[hadoop@node1 .ssh]$ chmod 644 ~/.ssh /authorized_keys
1.4.2 建立名称节点1到Datanode2的等效连接
[hadoop@node3 ~]$ cd ~
[hadoop@node3 ~]$ mkdir ~/.ssh
[hadoop@node3 ~]$ chmod 700 ~/.ssh
[hadoop@node3 ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
43:18:2a:22:37:19:3c:42:25:56:7d:a9:55:26:62:e6 hadoop@node3
[hadoop@node3 ~]$ ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_dsa.
Your public key has been saved in /home/hadoop/.ssh/id_dsa.pub.
The key fingerprint is:
a4:b9:29:98:6c:57:3e:ea:a8:95:23:4c:c2:25:37:23 hadoop@node3
[hadoop@node1 ~]$ ssh node3 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
The authenticity of host 'node3 (192.168.56.103)' can't be established.
RSA key fingerprint is 56:dc:97:60:22:59:17:f2:04:67:9f:fd:82:4c:64:cf.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'node3,192.168.56.103' (RSA) to the list of known hosts.
hadoop@node3's password:
[hadoop@node1 ~]$ ssh node3 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
hadoop@node3's password:
[hadoop@node1 ~]$ scp ~/.ssh/authorized_keys node3:~/.ssh/authorized_keys
hadoop@node3's password:
authorized_keys 100% 2988 2.9KB/s 00:00
我这些遇到authorized_keys文件权限问题导致无法建立连接的,修改下该文件的权限,问题解决
[hadoop@node3 .ssh]$ chmod 644 ~/.ssh /authorized_keys
1.4.3 建立名称节点1到Datanode3的等效连接
[hadoop@node4 ~]$ cd ~
[hadoop@node4 ~]$ mkdir ~/.ssh
[hadoop@node4 ~]$ chmod 700 ~/.ssh
[hadoop@node4 ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
3d:27:cb:18:c4:ec:7b:5f:78:47:2a:5e:d4:c7:fa:75 hadoop@node4
[hadoop@node4 ~]$ ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_dsa.
Your public key has been saved in /home/hadoop/.ssh/id_dsa.pub.
The key fingerprint is:
09:17:df:34:85:76:ba:2a:4f:c1:9e:3b:46:68:05:47 hadoop@node4
[hadoop@node1 ~]$ ssh node4 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
The authenticity of host 'node3 (192.168.56.103)' can't be established.
RSA key fingerprint is 56:dc:97:60:22:59:17:f2:04:67:9f:fd:82:4c:64:cf.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'node3,192.168.56.103' (RSA) to the list of known hosts.
hadoop@node3's password:
[hadoop@node1 ~]$ ssh node4 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
hadoop@node3's password:
[hadoop@node1 ~]$ scp ~/.ssh/authorized_keys node4:~/.ssh/authorized_keys
hadoop@node3's password:
authorized_keys 100% 2988 2.9KB/s 00:00
1.4.4 验证等效连接
1.4.4.1 名称节点到各节点的等效性
[hadoop@node1 ~]$ ssh node1;node2;node3;node4 date
Last login: Wed Jul 2 16:31:47 2014 from node1
1.4.4.2 datanode1到名称节点的等效性
[hadoop@node2 ~]$ ssh node1 date
Thu Jul 3 09:19:51 CST 2014
1.4.4.3 datanode2到名称节点的等效性
[hadoop@node3 ~]$ ssh node1 date
Thu Jul 3 09:19:57 CST 2014
1.4.4.4 datanode3到名称节点的等效性
[hadoop@node4 ~]$ ssh node1 date
Thu Jul 3 09:20:03 CST 2014
1.5 关闭防护墙
1.6安装jdk
目前系统的jdk是1.4的,需要安装1.6以上的版本
[hadoop@node1 ~]$ java -version
java version "1.4.2"
gij (GNU libgcj) version 4.1.2 20080704 (Red Hat 4.1.2-51)
1.6.1 下载jdk
可以到oracle官网下载jdk,下载地址下载地址：http://www.oracle.com/technetwork/java/javase/index.html
我这里下载的是jdk-8u5-linux-i586.rpm
1.6.1 安装jdk
我这里下载的是rpm安装文件，可以先查看rpm包中java的安装路径
[root@node1 soft]# rpm -qpl jdk-8u5-linux-i586.rpm
/usr/java/jdk1.8.0_05/man/man1/javap.1
/usr/java/jdk1.8.0_05/man/man1/javaws.1
/usr/java/jdk1.8.0_05/man/man1/jcmd.1
/usr/java/jdk1.8.0_05/man/man1/jconsole.1
/usr/java/jdk1.8.0_05/man/man1/jdb.1
/usr/java/jdk1.8.0_05/man/man1/jdeps.1
/usr/java/jdk1.8.0_05/man/man1/jhat.1
/usr/java/jdk1.8.0_05/man/man1/jinfo.1
/usr/java/jdk1.8.0_05/man/man1/jjs.1
/usr/java/jdk1.8.0_05/man/man1/jmap.1
/usr/java/jdk1.8.0_05/man/man1/jmc.1
/usr/java/jdk1.8.0_05/man/man1/jps.1
/usr/java/jdk1.8.0_05/man/man1/jrunscript.1
/usr/java/jdk1.8.0_05/man/man1/jsadebugd.1
/usr/java/jdk1.8.0_05/man/man1/jstack.1
/usr/java/jdk1.8.0_05/man/man1/jstat.1
/usr/java/jdk1.8.0_05/man/man1/jstatd.1
/usr/java/jdk1.8.0_05/man/man1/jvisualvm.1
/usr/java/jdk1.8.0_05/man/man1/keytool.1
/usr/java/jdk1.8.0_05/man/man1/native2ascii.1
/usr/java/jdk1.8.0_05/man/man1/orbd.1
/usr/java/jdk1.8.0_05/man/man1/pack200.1
/usr/java/jdk1.8.0_05/man/man1/policytool.1
/usr/java/jdk1.8.0_05/man/man1/rmic.1
/usr/java/jdk1.8.0_05/man/man1/rmid.1
/usr/java/jdk1.8.0_05/man/man1/rmiregistry.1
/usr/java/jdk1.8.0_05/man/man1/schemagen.1
/usr/java/jdk1.8.0_05/man/man1/serialver.1
/usr/java/jdk1.8.0_05/man/man1/servertool.1
/usr/java/jdk1.8.0_05/man/man1/tnameserv.1
/usr/java/jdk1.8.0_05/man/man1/unpack200.1
/usr/java/jdk1.8.0_05/man/man1/wsgen.1
/usr/java/jdk1.8.0_05/man/man1/wsimport.1
/usr/java/jdk1.8.0_05/man/man1/xjc.1
/usr/java/jdk1.8.0_05/release
/usr/java/jdk1.8.0_05/src.zip
部分输出，可以看出直接安装rpm包的话，java会默认安装在/usr/java/目录下,为hadoop设置环境变量java_home的时候可以指定该目录,不指定的话,系统会默认去找系统默认安装的java目录。
使用如下的命令直接安装java
[root@node1 soft]# rpm -ivh jdk-8u5-linux-i586.rpm
Preparing... ########################################### [100%]
1:jdk ########################################### [100%]
Unpacking JAR files...
rt.jar...
jsse.jar...
charsets.jar...
tools.jar...
localedata.jar...
jfxrt.jar...
plugin.jar...
javaws.jar...
deploy.jar...
安装完成后，为hadoop设定环境变量,如下:
[hadoop@node1 ~]$ more .bash_profile
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
export JAVA_HOME=/usr/java/jdk1.8.0_05
export JRE_HOME=/usr/java/jdk1.8.0_05/jre
export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib:$JRE_HOME/lib
export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH
使用hadoop用户退出后重新登录，查看java是否指向新安装的数据库版本。
[hadoop@node1 ~]$ java -version
java version "1.8.0_05"
Java(TM) SE Runtime Environment (build 1.8.0_05-b13)
Java HotSpot(TM) Client VM (build 25.5-b02, mixed mode, sharing)
如上输出，说明jdk已经配置完成，安装如上的步骤在那个其他节点上实施，即每个节点都要配置jdk。
1.7安装hadoop
1.7.1 下载hadoop
下载地址：http://hadoop.apache.org/common/releases.html
我这里下载的版本是: hadoop-2.4.0.tar.gz
1.7.2 安装hadoop
这里先在名称节点上安装，安装没有问题后再在其他节点上安装，安装过程是在root用户下。
将hadoop安装包复制一份到/usr目录下,hadoop就安装在该目录下
[root@node1 soft]# cp hadoop-1.2.1.tar.gz /usr/
解压缩hadoop安装包
[root@node1 usr]#cd usr
[root@node1 usr]# tar -zxvf hadoop-1.2.1.tar.gz
目录命名为hadoop
[root@node1 usr]# mv hadoop-1.2.1 hadoop
创建hadoop临时目录
[root@node1 usr]# mkdir /usr/hadoop/tmp
将该目录属主赋予hadoop
[root@node1 usr]# chown -R hadoop:hadoop hadoop
1.7.3 设置hadoop环境变量
将如下的语句添加到hadoop用户下的.bash_profile文件中，同时将HADOOP_HOME添加到PATH变量中。
export HADOOP_HOME=/usr/hadoop
export PATH=$HADOOP_HOME/bin:$JAVA_HOME/bin:$JRE_HOME/bin:$PATH
1.7.4 配置hadoop
1.7.4.1 配置hadoop-env.sh
该文件路径：/usr/hadoop/conf
编辑该文件，在该文件最后添加JAVA_HOME
[hadoop@node1 hadoop]$ vi hadoop-env.sh
export JAVA_HOME=/usr/java/jdk1.8.0_05
1.7.4.2 配置core-site.xml文件
该文件路径：/usr/hadoop/conf
编辑core-site.xml，添加如下内容
1.7.4.3 配置hdfs-site.xml文件
该文件路径：/usr/hadoop/conf
编辑该文件，添加如下配置项
1.7.4.4 配置mapred-site.xml文件
修改Hadoop中MapReduce的配置文件，配置的是JobTracker的地址和端口.
编辑该文件添加如下内容：
1.7.4.5 配置masters文件
编辑masters文件
添加名称节点的ip地址
[hadoop@node1 conf]$ more masters
192.168.56.101
1.7.4.6 配置slaves文件(名称主机特有)
编辑该文件，将datanode节点的ip地址写到该文件
[hadoop@node1 hadoop]$ more slaves
192.168.56.102
192.168.56.103
192.168.56.104
到这里名称节点已经配置完成了，下面继续配置datanode节点。
1.7.4.7 配置datanode节点
从节点1拷贝到datanode1
scp hadoop-env.sh hadoop@192.168.56.102:/usr/hadoop/conf
scp core-site.xml hadoop@192.168.56.102:/usr/hadoop/conf /
scp hdfs-site.xml hadoop@192.168.56.102:/usr/hadoop/conf
scp mapred-site.xml hadoop@192.168.56.102:/usr/hadoop/conf
scp masters hadoop@192.168.56.102:/usr/hadoop/conf
使用同样的命名将如上文件拷贝到另外两个节点。
1.8 启动和验证
1.8.1 格式化HDFS文件系统
在名称节点上执行如下的命令格式化文件系统
[hadoop@node1 ~]$ hadoop namenode -format
只需执行一次，下次启动的时候不需要再执行。
1.8.2 启动hadoop
在每个节点上的root用户执行如下的命令关闭防火墙
[root@node1 ~]# service iptables stop
[hadoop@node1 sbin]$ cd $HADOOP_HOME/bin
[hadoop1@node1 bin]$ ./start-all.sh
Warning: $HADOOP_HOME is deprecated.
starting namenode, logging to /usr1/hadoop/libexec/../logs/hadoop-hadoop1-namenode-node1.out
192.168.56.104: starting datanode, logging to /usr1/hadoop/libexec/../logs/hadoop-hadoop1-datanode-node4.out
192.168.56.103: starting datanode, logging to /usr1/hadoop/libexec/../logs/hadoop-hadoop1-datanode-node3.out
192.168.56.102: starting datanode, logging to /usr1/hadoop/libexec/../logs/hadoop-hadoop1-datanode-node2.out
192.168.56.101: starting secondarynamenode, logging to /usr1/hadoop/libexec/../logs/hadoop-hadoop1-secondarynamenode-node1.out
starting jobtracker, logging to /usr1/hadoop/libexec/../logs/hadoop-hadoop1-jobtracker-node1.out
192.168.56.104: starting tasktracker, logging to /usr1/hadoop/libexec/../logs/hadoop-hadoop1-tasktracker-node4.out
192.168.56.102: starting tasktracker, logging to /usr1/hadoop/libexec/../logs/hadoop-hadoop1-tasktracker-node2.out
192.168.56.103: starting tasktracker, logging to /usr1/hadoop/libexec/../logs/hadoop-hadoop1-tasktracker-node3.out
1.8.3 验证
在每个节点上执行jps命令
namenode节点
[hadoop@node1 soft]$ jps
5409 SecondaryNameNode
15143 Jps
5241 NameNode
5485 JobTracker
datanode1节点
[hadoop@node2 logs]$ jps
13954 Jps
4971 DataNode
5069 TaskTracker
datanode2节点
[hadoop@node3 tmp]$ jps
13954 Jps
4971 DataNode
5069 TaskTracker
datanode3节点
[hadoop@node4 tmp]$ jps
13954 Jps
4971 DataNode
5069 TaskTracker
打开web界面
http:192.168.56.101:50030
http:192.168.56.101:50070
1.8.4 wordcount验证
在/home/hadoop/目录下创建file目录
[hadoop@node1 ~]$ mkdir /home/hadoop/file
在该目录下创建两个文件
[hadoop@node1 file]$ echo "Hello World">file1.txt
[hadoop@node1 file]$ echo "Hello Hadoop">file2.txt
[hadoop@node1 file]$ more file1.txt
Hello World
[hadoop@node1 file]$ more file2.txt
Hello Hadoop
创建hdfs目录
hadoop fs -mkdir -p /user/hadoop/input
将刚才创建的file1.txt和file2.txt上传到hdfs
[hadoop@node1 file]$ hadoop fs -put /home/hadoop/file/file*.txt input
[hadoop1@node1 soft]$ hadoop fs -ls input
Warning: $HADOOP_HOME is deprecated.
Found 4 items
-rw-r--r-- 1 hadoop1 supergroup 12 2014-07-08 11:29 /user/hadoop1/input/file1.txt
-rw-r--r-- 1 hadoop1 supergroup 13 2014-07-08 11:29 /user/hadoop1/input/file2.txt
执行wordcount程序
[hadoop1@node3 file]$ hadoop jar /usr1/hadoop/hadoop-examples-1.2.1.jar wordcount input output
Warning: $HADOOP_HOME is deprecated.
14/07/09 11:26:35 INFO input.FileInputFormat: Total input paths to process : 4
14/07/09 11:26:35 INFO util.NativeCodeLoader: Loaded the native-hadoop library
14/07/09 11:26:35 WARN snappy.LoadSnappy: Snappy native library not loaded
14/07/09 11:26:44 INFO mapred.JobClient: Running job: job_201407090921_0003
14/07/09 11:26:45 INFO mapred.JobClient: map 0% reduce 0%
14/07/09 11:27:27 INFO mapred.JobClient: map 25% reduce 0%
14/07/09 11:27:28 INFO mapred.JobClient: map 50% reduce 0%
14/07/09 11:27:41 INFO mapred.JobClient: map 50% reduce 16%
14/07/09 11:28:19 INFO mapred.JobClient: map 75% reduce 16%
14/07/09 11:28:27 INFO mapred.JobClient: map 75% reduce 25%
14/07/09 11:30:09 INFO mapred.JobClient: map 100% reduce 25%
14/07/09 11:30:59 INFO mapred.JobClient: map 100% reduce 100%
14/07/09 11:31:15 INFO mapred.JobClient: Job complete: job_201407090921_0003
14/07/09 11:31:47 INFO mapred.JobClient: Counters: 30
14/07/09 11:31:47 INFO mapred.JobClient: Map-Reduce Framework
14/07/09 11:31:47 INFO mapred.JobClient: Spilled Records=12
14/07/09 11:31:47 INFO mapred.JobClient: Map output materialized bytes=97
14/07/09 11:31:47 INFO mapred.JobClient: Reduce input records=6
14/07/09 11:31:47 INFO mapred.JobClient: Virtual memory (bytes) snapshot=1549914112
14/07/09 11:31:47 INFO mapred.JobClient: Map input records=4
14/07/09 11:31:47 INFO mapred.JobClient: SPLIT_RAW_BYTES=480
14/07/09 11:31:47 INFO mapred.JobClient: Map output bytes=61
14/07/09 11:31:47 INFO mapred.JobClient: Reduce shuffle bytes=97
14/07/09 11:31:47 INFO mapred.JobClient: Physical memory (bytes) snapshot=615403520
14/07/09 11:31:47 INFO mapred.JobClient: Reduce input groups=3
14/07/09 11:31:47 INFO mapred.JobClient: Combine output records=6
14/07/09 11:31:47 INFO mapred.JobClient: Reduce output records=3
14/07/09 11:31:47 INFO mapred.JobClient: Map output records=6
14/07/09 11:31:47 INFO mapred.JobClient: Combine input records=6
14/07/09 11:31:47 INFO mapred.JobClient: CPU time spent (ms)=2090
14/07/09 11:31:47 INFO mapred.JobClient: Total committed heap usage (bytes)=614481920
14/07/09 11:31:47 INFO mapred.JobClient: File Input Format Counters
14/07/09 11:31:47 INFO mapred.JobClient: Bytes Read=37
14/07/09 11:31:47 INFO mapred.JobClient: FileSystemCounters
14/07/09 11:31:47 INFO mapred.JobClient: HDFS_BYTES_READ=517
14/07/09 11:31:47 INFO mapred.JobClient: FILE_BYTES_WRITTEN=277804
14/07/09 11:31:47 INFO mapred.JobClient: FILE_BYTES_READ=79
14/07/09 11:31:47 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=25
14/07/09 11:31:47 INFO mapred.JobClient: Job Counters
14/07/09 11:31:47 INFO mapred.JobClient: Launched map tasks=6
14/07/09 11:31:47 INFO mapred.JobClient: Launched reduce tasks=1
14/07/09 11:31:47 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=210985
14/07/09 11:31:47 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
14/07/09 11:31:47 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=192073
14/07/09 11:31:47 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
14/07/09 11:31:47 INFO mapred.JobClient: Rack-local map tasks=4
14/07/09 11:31:47 INFO mapred.JobClient: Data-local map tasks=2
14/07/09 11:31:47 INFO mapred.JobClient: File Output Format Counters
14/07/09 11:31:47 INFO mapred.JobClient: Bytes Written=25
[hadoop1@node3 file]$ hadoop fs -cat /user/hadoop1/output/part-r-00000
Warning: $HADOOP_HOME is deprecated.
查看输出结果
[hadoop1@node3 file]$ hadoop fs -ls output
Warning: $HADOOP_HOME is deprecated.
Found 3 items
-rw-r--r-- 1 hadoop1 supergroup 0 2014-07-09 11:31 /user/hadoop1/output/_SUCCESS
drwxr-xr-x - hadoop1 supergroup 0 2014-07-09 11:26 /user/hadoop1/output/_logs
-rw-r--r-- 1 hadoop1 supergroup 25 2014-07-09 11:30 /user/hadoop1/output/part-r-00000
输出结果保留在part-r-00000文件中
[hadoop1@node3 file]$ hadoop fs -cat /user/hadoop/output/part-r-00000
Warning: $HADOOP_HOME is deprecated.
Hadoop 1
Hello 3
World 2
-- The End --