HDFS复制因子

问题描述

当我将文件上传到HDFS时，如果将复制因子设置为1，那么文件将分割成一台计算机，或者分割将分布到整个网络中的多台计算机上？

  hadoop fs -D dfs.replication = 1 -copyFromLocal file.txt / user / ablimit 
  

解决方案

根据

这个逻辑很有意义，因为它减少了不同节点之间的网络喋喋不休。但是，这本书于2009年发布，Hadoop框架发生了很多变化。

我认为这取决于客户端是否与Hadoop相同是否节点。如果客户端是Hadoop节点，则所有分割将位于同一节点上。尽管集群中有多个节点，但这并没有提供更好的读/写吞吐量。如果客户端与Hadoop节点不同，那么随机为每个分裂选择节点，因此分裂分散在集群中的节点上。现在，这提供了更好的读/写吞吐量。

写入多个节点的一个优点是，即使其中一个节点出现故障，但至少有一些数据可以从剩余的分割中以某种方式恢复。

When I'm uploading a file to HDFS, if I set the replication factor to 1 then the file splits gonna reside on one single machine or the splits would be distributed to multiple machines across the network ?

hadoop fs -D dfs.replication=1 -copyFromLocal file.txt /user/ablimit

解决方案

According to the Hadoop : Definitive Guide

This logic makes sense as it decreases the network chatter between the different nodes. But, the book was published in 2009 and there had been a lot of changes in the Hadoop framework.

I think it depends on, whether the client is same as a Hadoop node or not. If the client is a Hadoop node then all the splits will be on the same node. This doesn't provide any better read/write throughput in-spite of having multiple nodes in the cluster. If the client is not same as the Hadoop node, then the node is chosen at random for each split, so the splits are spread across the nodes in a cluster. Now, this provides a better read/write throughput.

One advantage of writing to multiple nodes is that even if one of the node goes down, a couple of splits might be down, but at least some data can be recovered somehow from the remaining splits.

这篇关于HDFS复制因子的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！