奇怪的DNS服务器导致Hadoop和HBase故障

问题描述

我有一个网络，有一些奇怪的（据我所知）DNS服务器导致Hadoop或HBase故障。

它将我的主机名解析为我的机器的某个地址不知道（即没有这样的界面）。

如果我在/ etc / hosts中有以下条目，Hadoop会工作：

  127.0.0.1 localhost 
 127.0.1.1 myhostname 
  

如果条目127.0.1.1 myhostname没有上传文件到HDFS失败，并抱怨说它只能将文件复制到0个datanode而不是1。

$ b $但在这种情况下，HBase不起作用：从HBase shell创建一个表会导致NotAllMetaRegionsOnlineException（实际上由HMaster尝试绑定DNS服务器为myhostname返回的错误地址）。

在其他网络中，我使用以下/ etc / hosts：

  127.0.0.1 localhost 
 192.168.1.1 myhostname 
  

和bot h Hadoop和HBase工作。
问题是在第二个网络中，地址是动态的，我不能将其列入/ etc / hosts以覆盖由奇怪的DNS返回的结果。

Hadoop以伪分布式模式运行。 HBase也运行在单个节点上。

更改DNS服务器的行为不是一个选项。
在hbase / conf / regionservers中将localhost更改为127.0.0.1不会更改任何内容。有人可以建议一种方式，如何在保留互联网连接的同时覆盖其行为（实际上通过Teamviewer在客户端的机器上工作）。或者某种方式来配置HBase（或Zookeeper是管理的）不使用主机名确定要绑定的地址？

解决方案

幸运的是，我找到了这个DNS服务器问题的解决方法。

当本地主机名查询时，DNS服务器返回无效地址。
HBase默认情况下会对本地主机名进行DNS查找，以确定要绑定的位置。
由于DNS服务器返回的地址无效，HMaster无法绑定。

解决方法：
在hbase / conf / hbase- site.xml明确指定将用于master和regionserver的接口：

 < configuration> 
< property> 
< name> hbase.master.dns.interface< / name> 
< value> lo< / value> 
< / property> 
< property> 
< name> hbase.regionserver.dns.interface< / name> 
< value> lo< / value> 
< / property> 
< / configuration> 
  

在这种情况下，我指定了用于master和regionserver的loopback接口（lo） / p>

I have a network with some weird (as I understand) DNS server which causes Hadoop or HBase to malfunction.

It resolves my hostname to some address my machine doesn't know about (i.e. there is no such interface).

Hadoop does work if I have following entries in /etc/hosts:

127.0.0.1     localhost
127.0.1.1     myhostname

If entry "127.0.1.1 myhostname" is not present uploading file to HDFS fails and complains that it can replicate the file only to 0 datanodes instead of 1.

But in this case HBase does not work: creating a table from HBase shell causes NotAllMetaRegionsOnlineException (caused actually by HMaster trying to bind to wrong address returned by DNS server for myhostname).

In other network, I am using following /etc/hosts:

127.0.0.1   localhost
192.168.1.1 myhostname

And both Hadoop and HBase work.The problem is that in second network the address is dynamic and I can't list it into /etc/hosts to override result returned by weird DNS.

Hadoop is run in pseudo-distributed mode. HBase also runs on single node.

Changing behavior of DNS server is not an option. Changing "localhost" to 127.0.0.1 in hbase/conf/regionservers doesn't change anything.

Can somebody suggest a way how can I override its behavior while retaining internet connection (I actually work at client's machine through Teamviewer). Or some way to configure HBase (or Zookeeper it is managing) not to use hostname to determine address to bind?

解决方案

Luckily, I've found the workaround to this DNS server problem.

DNS server returned invalid address when queried by local hostname.HBase by default does reverse DNS lookup on local hostname to determine where to bind.Because the address returned by DNS server was invalid, HMaster wasn't able to bind.

Workaround:In hbase/conf/hbase-site.xml explicitly specify interfaces that will be used for master and regionserver:

<configuration>
  <property>
    <name>hbase.master.dns.interface</name>
    <value>lo</value>
  </property>
  <property>
    <name>hbase.regionserver.dns.interface</name>
    <value>lo</value>
  </property>
</configuration>

In this case, I specified loopback interface (lo) to be used for both master and regionserver.

这篇关于奇怪的DNS服务器导致Hadoop和HBase故障的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！