HDFS复制属性不反映在hfs-site.xml中定义的属性

问题描述

 < p>< p> ？xml version =1.0encoding =UTF-8？> 
<？xml-stylesheet type =text / xslhref =configuration.xsl？> 
 
<配置> 
<属性> 
< name> dfs.replication< / name> 
<值> 1< /值> 
< / property> 
 
<属性> 
<名称> dfs.namenode.name.dir< /名称> 
< value> / Users / *** / Documnent / hDir / hdfs / namenode< / value> 
< / property> 
<属性> 
< name> dfs.datanode.data.dir< / name> 
< value> / Users / *** / Documnent / hDir / hdfs / datanode< / value> 
< / property> 
 
<属性> 
<名称> dfs.permissions< /名称> 
<值> false< /值> 
< / property> 
 
< / configuration> 
  

但是当我试图从本地系统复制文件到hdfs文件系统时，我发现复制该文件的因子是3.下面是在hdfs上复制文件的代码：

  public class FileCopyWithWrite {
 
 public static void main（String [] args）{
 // TODO自动生成的方法存根
 
 String localSrc =/ Users / *** / Documents / hContent /input/docs/1400-8.txt; 
 String dst =hdfs：//localhost/books/1400-8.txt; 
尝试{
 
 InputStream in = new BufferedInputStream（new FileInputStream（localSrc））; 
配置conf = new配置（）;; 
 FileSystem fs = FileSystem.get（URI.create（dst），conf）; 
 
 OutputStream out = fs.create（new Path（dst），new Progressable（）{
 
 public void progress（）{
 // TODO自动生成方法存根
 System.out.println（。）; 
} 
}）; 
 
 IOUtils.copyBytes（in，out，4092，true）; 
 
 
} catch（Exception e）{
 e.printStackTrace（）; 
 
 
 
 $ b $ / code> 

请找此截图显示复制因子为3：

请注意，我已经开始使用伪分布模式，并且根据Hadoop The Definite Guide书中的文档更新了hdfs-site.xml。任何有关为什么发生这种情况的建议？

解决方案

删除namenode目录和datanode目录解决了我的问题。
所以我按照以下步骤操作：

1. 停止服务，即dfs，yarn和mr。




2. 删除hfs-site.xml文件中指定的namenode和datanote目录。

重新创建namenode和datanode目录。


3. 重新启动服务，即dfs，yarn和mr。

I am woking on HDFS and setting the replication factor to 1 in hfs-site.xml as follows:

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>

  <property>
    <name>dfs.namenode.name.dir</name>
    <value>/Users/***/Documnent/hDir/hdfs/namenode</value>
  </property>
  <property>
    <name>dfs.datanode.data.dir</name>
    <value>/Users/***/Documnent/hDir/hdfs/datanode</value >
  </property>

  <property>
    <name>dfs.permissions</name>
    <value>false</value>
  </property>

</configuration>

But when i tried copying a file from local system to the hdfs file system, i found that the replication factor for that file was 3. Here is the code which is copying the file on hdfs:

    public class FileCopyWithWrite {

    public static void main(String[] args) {
        // TODO Auto-generated method stub

        String localSrc = "/Users/***/Documents/hContent/input/docs/1400-8.txt";
        String dst = "hdfs://localhost/books/1400-8.txt";
        try{

            InputStream in = new BufferedInputStream(new FileInputStream(localSrc));
            Configuration conf = new Configuration();;
            FileSystem fs = FileSystem.get(URI.create(dst), conf);

            OutputStream out = fs.create(new Path(dst), new Progressable() {

                public void progress() {
                    // TODO Auto-generated method stub
                    System.out.println(".");
                }
            });

            IOUtils.copyBytes(in, out, 4092, true);


        }catch(Exception e){
            e.printStackTrace();
        }
    }

}

Please find this screenshot showing the replication factor as 3:

Note that i have started in pseudo-distributed mode and i have updated the hdfs-site.xml according to the documentation in Hadoop The Definite Guide book. Any suggestion on why this is happening?

解决方案

Deleting the namenode directory and datanode directory solved the issue for me.So I followed the following procedure:

1. Stop the services, viz dfs, yarn and mr.

2. Delete the namenode and datanote directories as specified in the hfs-site.xml file.

3. Re-create the namenode and datanode directories.

4. Restart the services, viz dfs, yarn and mr.

这篇关于HDFS复制属性不反映在hfs-site.xml中定义的属性的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！