如何使用kubernetes执行程序为气流工作人员吊舱全局设置request_cpu?

本文介绍了如何使用kubernetes执行程序为气流工作人员吊舱全局设置request_cpu?的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！

问题描述

我正在尝试在Kubernetes执行器中为Airflow设置request_cpu参数，但无法找到可以执行此操作的位置.在默认气流配置中，我找到了default_cpus但是根据此答案，这里没有使用任何地方，在Kubernetes部分中也没有其他地方可以找到我的参考CPU请求.

I'm trying to set the request_cpu parameter in the Kubernetes executor for Airflow but haven't been able to find where I can do that. In the default airflow config I found default_cpus but according to this answer there is nowhere that that is used, and nowhere else in the Kubernetes section could I find a reference to the CPU request.

如何在Airflow Kubernetes执行器中设置request_cpu参数?

How can I set the request_cpu parameter in the Airflow Kubernetes executor?

理想情况下，我希望将其设置为全局默认值，而不是基于每个操作员，尽管总的来说，我认为按每个操作员/任务设置它确实更有意义

Ideally, what I would like to be able to do is set this as a global default rather than on a per-operator basis, even though in general I think that it does make more sense to set it per-operator/task

推荐答案

您可以在任务级别为KubernetesExecutor设置此executor_config，如下所示.

You can set this executor_config at task level for KubernetesExecutor as follows.

    exmaple_task = PythonOperator(
        task_id="exmaple_task",
        python_callable=print_stuff,
        executor_config={
            "KubernetesExecutor": {"request_cpu": "1",
                                   "request_memory": "128Mi",
                                   "limit_memory": "128Mi"}}
    )

您可以在executor_config中定义以下内容:

You can define the following in executor_config :

图片
image_pull_policy
request_memory
request_cpu
limit_memory
limit_cpu
limit_gpu
node_selectors
亲和力
公差
注释
音量
volume_mounts
volume_mounts

image
image_pull_policy
request_memory
request_cpu
limit_memory
limit_cpu
limit_gpu
node_selectors
affinity
tolerations
annotations
volumes
volume_mounts
volume_mounts

文档: https://airflow.apache.org/docs/1.10.9/_api/airflow/contrib/executors/kubernetes_executor/index.html#airflow.contrib.executors. kubernetes_executor.KubernetesExecutorConfig

这篇关于如何使用kubernetes执行程序为气流工作人员吊舱全局设置request_cpu?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！