Airflow 中的 KubernetesPodOperator 特权 security_context

问题描述

我在 Google 的 Cloud Composer 上运行 Airflow.我正在使用 KubernetesPodOperator 并希望通过 gcsfuse 将 google 存储桶挂载到 pod 中的目录.似乎要做到这一点，我需要按照 此处 的规定提供 k8s 特权安全上下文.似乎气流最近向 KubernetesPodOperator 添加了 security_context 参数.我在操作符中指定的安全上下文是:

I am running Airflow on Google's Cloud Composer. I am using the KubernetesPodOperator and would like to mount a google storage bucket to a directory in pod via gcsfuse. It seems like to do this I need to give k8s privileged security context as specified here. It seems like airflow recently added the security_context parameter to the KubernetesPodOperator. The security context I am specifying in the operator is :

security_context = {
  'securityContext': {
    'privileged': True,
    'capabilities':
      {'add': ['SYS_ADMIN']}
  }
}

当我尝试在气流工作器中运行 airflow test dag_id task_id date 时，pod 启动，当代码尝试通过 gcsfuse 挂载桶时，它会抛出错误 "fusermount: fuse device未找到，请先尝试modprobe fuse"".这使得 security_context 似乎不起作用(例如.).

When I try running airflow test dag_id task_id date in the airflow worker, the pod launches and when the code tries to mount the bucket via gcsfuse it throws the error "fusermount: fuse device not found, try 'modprobe fuse' first". This makes it seems as the security_context is not working (ex.).

我是否误解了运算符中的 security_context 参数和/或我的 securityContext 字典定义无效?

Am I misunderstanding what the security_context parameter in the operator and/or is my securityContext dictionary definition invalid?

推荐答案

KubernetesPodOperator 的 security_context kwarg 设置 pod 的安全上下文，而不是 pod 内的特定容器，因此它只支持PodSecurityContext.由于您指定的参数对于 Pod 的安全上下文无效，因此它们将被丢弃.

The security_context kwarg for the KubernetesPodOperator sets the security context for the pod, not a specific container within the pod, so it only supports the options outlined in PodSecurityContext. Since the parameters you are specifying aren't valid for a pod's security context, they are being discarded.

privileged 和 capabilities 属性仅作为容器 SecurityContext，这意味着您需要以某种方式将它们设置在 Pod 的 容器规范.您可以通过自己定义整个 pod 规范来做到这一点(而不是让操作员为您生成它).使用 KubernetesPodOperator，您可以设置 full_pod_spec 或 pod_template_file 以指定 Kubernetes API Python 对象或对象 YAML 的路径，然后将使用该对象生成 Pod.使用前者的示例:

The privileged and capabilities properties are only valid as part of a container's SecurityContext, meaning you'll need to somehow set them on the pod's container spec. You can do this by defining the entire pod spec yourself (as opposed to having the operator generate it for you). Using KubernetesPodOperator, you can set full_pod_spec or pod_template_file to specify a Kubernetes API Python object, or path to an object YAML, which would then be used to generate the pod. Example using the former:

from airflow.contrib.operators.kubernetes_pod_operator import KubernetesPodOperator
import kubernetes.client.models as k8s

pod = k8s.V1Pod()
pod.spec = k8s.V1PodSpec()
pod.spec.containers = [
    k8s.V1Container(
      ...,
      security_context={
          'privileged': True,
          'capabilities': {'add': ['SYS_ADMIN']}
      }
    )
]

# Equivalent to setting security_context from the operator
pod.spec.security_context = {}

t1 = KubernetesPodOperator(..., full_pod_spec=pod)

如果您想将 pod_template_file 与 Cloud Composer 一起使用，您可以将 pod YAML 上传到 GCS 并将其设置为 映射的存储路径(例如 /home/airflow/gcs/dags/my-pod.yaml 如果你把它在 DAGs 目录中.

If you want to use pod_template_file with Cloud Composer, you can upload a pod YAML to GCS and set it to one of the mapped storage paths (e.g. /home/airflow/gcs/dags/my-pod.yaml if you put it in the DAGs directory.

通读 KubernetesPodOperator 的 airflow.providers.google.cloud 版本，full_pod_spec 可能在较新版本的 operator 中损坏.但是，它应该适用于旧的 contrib 版本.

Reading through the airflow.providers.google.cloud version of KubernetesPodOperator, it's possible that full_pod_spec is broken in newer version of the operator. However, it should work with the old contrib version.

这篇关于Airflow 中的 KubernetesPodOperator 特权 security_context的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！