本文介绍了Hadoop:不能在core-site.xml中将默认FileSystem设置为HDFS的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我在伪分布式模式下使用Hadoop 1.0.3。我的conf / core-site.xml设置如下:
< configuration>
<属性>
<名称> fs.default.name< /名称>
< value> hdfs:// localhost:9000< / value>
< / property>
<属性>
<名称> mapred.child.tmp< / name>
<值> / home / administrator / hadoop / temp< /值>
< / property>
< / configuration>
所以我相信我的默认文件系统被设置为HDFS。
然而,当我运行下面的代码:
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);
我以为fs应该是 DistributedFileSystem
实例。然而,事实证明是 LocalFileSystem
实例。
但是,如果我运行以下代码:
Configuration conf = new Configuration();
conf.set(fs.default.name,hdfs:// localhost:9000);
FileSystem fs = FileSystem.get(conf);
然后我可以得到 DistributedFileSystem
fs。
不是我的默认FileSystem在core-site.xml中设置为HDFS吗?如果没有的话,我应该怎么设置呢?
解决方案 Eclipse环境不知道在Hadoop安装目录下的conf目录下找到conf目录core-default.xml和core-site.xml,除非将这些文件添加到Eclipse类路径中以便首先加载。由于这些不会在eclipse classpath中添加,所以默认的core-site.xml将从jar文件中加载hadoop - * - core.jar(例如: hadoop-0.20.2-core.jar for version 0.20),它具有本地系统作为默认文件系统,因此您看到 LocalFileSystem
对象而不是因此,将< HADOOP_INSTALL> / conf
目录添加到分发文件系统
。
eclipse项目类路径,转到项目属性(项目 - >属性) - > Java构建路径 - >库选项卡 - >添加外部类文件夹 - >从
< HADOOP_INSTALL>
$ b
上面的代码应该把你的`/core-site.xml'添加到你的eclipse类路径中,并且所有的设置都应该覆盖默认的。
I am using Hadoop 1.0.3 in a Pseudo-Distributed mode. And my conf/core-site.xml is set as follows:
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
<property>
<name>mapred.child.tmp</name>
<value>/home/administrator/hadoop/temp</value>
</property>
</configuration>
So I believed that my default filesystem is set to HDFS.However, when I run the following code:
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);
I thought that fs should be a DistributedFileSystem
instance. However, it turns out to be LocalFileSystem
instance.
But, if I run the following code:
Configuration conf = new Configuration();
conf.set("fs.default.name", "hdfs://localhost:9000");
FileSystem fs = FileSystem.get(conf);
Then I can get a DistributedFileSystem
fs.
Isn't my default FileSystem set to HDFS in core-site.xml? If not, how should I set that?
解决方案
Eclipse environment doesn't know where the conf directory under Hadoop install directory to find the core-default.xml and core-site.xml unless these files are added to the Eclipse classpath to load first.
Since these are not added in the eclipse classpath, the default core-site.xml will be loaded from the jar file hadoop-*-core.jar (For eg: hadoop-0.20.2-core.jar for version 0.20) which has the local system as default file system and hence you are seeing LocalFileSystem
object instead of DistributedFileSystem
.
So, to add the <HADOOP_INSTALL>/conf
directory to eclipse project classpath, goto the project properties(project -> properties) -> Java build path -> Libraries tab -> Add external class folder -> Select the conf directory from <HADOOP_INSTALL>
The above should add your `/core-site.xml' to your eclipse classpath and all your settings should override the default ones.
这篇关于Hadoop:不能在core-site.xml中将默认FileSystem设置为HDFS的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!