问题描述
yarn.scheduler.maximum-allocation-mb 和 yarn.nodemanager.resource.memory-mb yarn-site.xml 中，并且我看到了解释。
yarn.scheduler.maximum-allocation-mb 给出了以下定义： RM中的每个容器请求，以MB为单位。内存请求比此更高会抛出一个InvalidResourceRequestException。这是否意味着只对resourcemanager的内存请求受此值限制？
和 HOwever，我仍然无法区分这些。这些解释让我觉得它们是一样的。 更令人困惑的是，它们的默认值完全相同：8192 mb。我如何区分这些差异？谢谢。 考虑在您设置集群的场景中，每台机器具有48 GB的RAM。这些内存中的一部分应该保留给操作系统和其他已安装的应用程序。 yarn.nodemanager.resource.memory-mb 被定义为物理内存的数量，以MB为单位，可以分配给容器。这是否意味着所有的容器跨越整个集群，总结在一起？
可以为容器分配的物理内存量（以MB为单位）。这意味着YARN可以在此节点上使用的内存量，因此此属性
应该低于该机器的总内存。
<名称> yarn.nodemanager.resource.memory-MB< /名称>
<值> 40960< /值> <！ - 40 GB - >
下一步是提供关于如何分解Container中可用资源的YARN指南。您可以通过指定为容器分配的RAM的最小单位来完成此操作。
在 yarn-site.xml
< name> yarn.scheduler.minimum-allocation-mb< / name> <！ - 每个容器的RAM - >
<值> 2048< /值>
yarn.scheduler.maximum-allocation-mb： 它定义了一个容器可用的最大内存分配，单位为MB 这意味着RM只能以
yarn.scheduler.minimum-allocation-mb为增量分配内存给容器，并且不超过yarn.scheduler.maximum-allocation-mb 并且它不应该超过Node的总分配内存。
$ b $ yarn-site.xml
< name> yarn.scheduler.maximum-allocation-mb< / name> <！ - 每个容器的最大RAM数量>
<值> 8192< /值>
对于MapReduce应用程序，YARN处理每个映射或减少容器和单台机器上的任务是容器的数量。
我们希望每个节点上最多容纳20个容器，因此每个容器需要（40 GB总RAM）/（20＃容器）=每个容器最少2 GB。 yarn .scheduler.minimum-allocation-mb
同样，我们希望限制由属性控制的容器的最大内存利用率。 yarn.scheduler.maximum-allocation-mb
例如，如果一个作业要求每个地图容器2049 MB内存（ mapreduce.map.memory.mb = 2048在mapred-site.xml中设置），RM会给它一个4096 MB（ 2 * yarn.scheduler。如果你有一个巨大的MR作业需要一个9999 MB的地图容器，那么这个作业将是最小的分配mb ）容器。
遇到错误消息。
What is difference between yarn.scheduler.maximum-allocation-mb and yarn.nodemanager.resource.memory-mb?
I see both of these in yarn-site.xml and I see the explanations here.
yarn.scheduler.maximum-allocation-mb is given the following definition: The maximum allocation for every container request at the RM, in MBs. Memory requests higher than this will throw a InvalidResourceRequestException. Does this mean memory requests ONLY on the resourcemanager are limited by this value?
And yarn.nodemanager.resource.memory-mb is given definition of Amount of physical memory, in MB, that can be allocated for containers. Does this mean the total amount for all containers across the entire cluster, summed together?
HOwever, I still cannot discern between these. Those explanations make me think that they are the same.
Even more confusing, their default values are exactly the same: 8192 mb. How do I tell difference between these? Thank you.
Consider in a scenario where you are setting up a cluster where each machine having 48 GB of RAM. Some of this RAM should be reserved for Operating System and other installed applications.
yarn.nodemanager.resource.memory-mb:
Amount of physical memory, in MB, that can be allocated for containers. It means the amount of memory YARN can utilize on this node and therefore this property should be lower then the total memory of that machine.
<name>yarn.nodemanager.resource.memory-mb</name>
<value>40960</value> <!-- 40 GB -->
The next step is to provide YARN guidance on how to break up the total resources available into Containers. You do this by specifying the minimum unit of RAM to allocate for a Container.
In yarn-site.xml
<name>yarn.scheduler.minimum-allocation-mb</name> <!-- RAM-per-container ->
<value>2048</value>
yarn.scheduler.maximum-allocation-mb:
It defines the maximum memory allocation available for a container in MB
it means RM can only allocate memory to containers in increments of "yarn.scheduler.minimum-allocation-mb" and not exceed "yarn.scheduler.maximum-allocation-mb" and It should not be more then total allocated memory of the Node.
In yarn-site.xml
<name>yarn.scheduler.maximum-allocation-mb</name> <!-Max RAM-per-container->
<value>8192</value>
For MapReduce applications, YARN processes each map or reduce task in a container and on a single machine there can be number of containers.We want to allow for a maximum of 20 containers on each node, and thus need (40 GB total RAM) / (20 # of containers) = 2 GB minimum per container controlled by property yarn.scheduler.minimum-allocation-mb
Again we want to restrict maximum memory utilization for a container controlled by property "yarn.scheduler.maximum-allocation-mb"
For example, if one job is asking for 2049 MB memory per map container(mapreduce.map.memory.mb=2048 set in mapred-site.xml), RM will give it one 4096 MB(2*yarn.scheduler.minimum-allocation-mb) container.
If you have a huge MR job which asks for a 9999 MB map container, the job will be killed with the error message.
这篇关于`yarn.scheduler.maximum-allocation-mb`和`yarn.nodemanager.resource.memory-mb`之间的区别？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！