本文介绍了Kubernetes上的气流:Errno 13-权限被拒绝:'/opt/airflow/logs/scheduler的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！
问题描述
我正在从稳定头盔图表在Kubernetes上运行Airflow. .我正在AWS环境中运行它.无论是否安装任何外部卷来存储日志，都存在此错误.我试图将[logs]部分的配置设置为指向我创建的EFS卷. PV通过PVC进行安装，但是由于以下错误，我的容器崩溃了(调度程序和Web):
I am running Airflow on Kubernetes from the stable helm chart. I'm running this in an AWS environment. This error exists with and without mounting any external volumes for log storage. I tried to set the configuration of the [logs] section to point to an EFS volume that I created. The PV gets mounted through a PVC but my containers are crashing (scheduler and web) due to the following error:
*** executing Airflow initdb...
Unable to load the config, contains a configuration error.
Traceback (most recent call last):
File "/usr/local/lib/python3.6/logging/config.py", line 565, in configure
handler = self.configure_handler(handlers[name])
File "/usr/local/lib/python3.6/logging/config.py", line 738, in configure_handler
result = factory(**kwargs)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/log/file_processor_handler.py", line 50, in __init__
os.makedirs(self._get_log_directory())
File "/usr/local/lib/python3.6/os.py", line 220, in makedirs
mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/opt/airflow/logs/scheduler/2020-08-20'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 25, in <module>
from airflow.configuration import conf
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/__init__.py", line 47, in <module>
settings.initialize()
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/settings.py", line 374, in initialize
LOGGING_CLASS_PATH = configure_logging()
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/logging_config.py", line 68, in configure_logging
raise e
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/logging_config.py", line 63, in configure_logging
dictConfig(logging_config)
File "/usr/local/lib/python3.6/logging/config.py", line 802, in dictConfig
dictConfigClass(config).configure()
File "/usr/local/lib/python3.6/logging/config.py", line 573, in configure
'%r: %s' % (name, e))
ValueError: Unable to configure handler 'processor': [Errno 13] Permission denied: '/opt/airflow/logs/scheduler/2020-08-20'
持久体积(不是通过稳定/气流图手动创建的)
Persistent volume (created manually not from the stable/airflow chart)
apiVersion: v1
kind: PersistentVolume
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"PersistentVolume","metadata":{"annotations":{},"name":"efs-pv"},"spec":{"accessModes":["ReadWriteMany"],"capacity":{"storage":"5Gi"},"csi":{"driver":"efs.csi.aws.com","volumeHandle":"fs-e476a166"},"persistentVolumeReclaimPolicy":"Retain","storageClassName":"efs-sc","volumeMode":"Filesystem"}}
pv.kubernetes.io/bound-by-controller: "yes"
creationTimestamp: "2020-08-20T15:47:21Z"
finalizers:
- kubernetes.io/pv-protection
name: efs-pv
resourceVersion: "49476860"
selfLink: /api/v1/persistentvolumes/efs-pv
uid: 45d9f5ea-66c1-493e-a2f5-03e17f397747
spec:
accessModes:
- ReadWriteMany
capacity:
storage: 5Gi
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: efs-claim
namespace: airflow
resourceVersion: "49476857"
uid: 354103ea-f8a9-47f1-a7cf-8f449f9a2e8b
csi:
driver: efs.csi.aws.com
volumeHandle: fs-e476a166
persistentVolumeReclaimPolicy: Retain
storageClassName: efs-sc
volumeMode: Filesystem
status:
phase: Bound
日志的持久体积声明(不是通过稳定/气流图手动创建的):
Persistent Volume Claim for logs (created manually not from the stable/airflow chart):
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"efs-claim","namespace":"airflow"},"spec":{"accessModes":["ReadWriteMany"],"resources":{"requests":{"storage":"5Gi"}},"storageClassName":"efs-sc"}}
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
creationTimestamp: "2020-08-20T15:47:46Z"
finalizers:
- kubernetes.io/pvc-protection
name: efs-claim
namespace: airflow
resourceVersion: "49476866"
selfLink: /api/v1/namespaces/airflow/persistentvolumeclaims/efs-claim
uid: 354103ea-f8a9-47f1-a7cf-8f449f9a2e8b
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 5Gi
storageClassName: efs-sc
volumeMode: Filesystem
volumeName: efs-pv
status:
accessModes:
- ReadWriteMany
capacity:
storage: 5Gi
phase: Bound
下面的我的values.yaml:
airflow:
image:
repository: apache/airflow
tag: 1.10.10-python3.6
## values: Always or IfNotPresent
pullPolicy: IfNotPresent
pullSecret: ""
executor: KubernetesExecutor
fernetKey: "XXXXXXXXXHIVb8jK6lfmSAvx4mO6Arehnc="
config:
AIRFLOW__CORE__REMOTE_LOGGING: "True"
AIRFLOW__CORE__REMOTE_BASE_LOG_FOLDER: "s3://mybucket/airflow/logs"
AIRFLOW__CORE__REMOTE_LOG_CONN_ID: "MyS3Conn"
AIRFLOW__KUBERNETES__WORKER_CONTAINER_REPOSITORY: "apache/airflow"
AIRFLOW__KUBERNETES__WORKER_CONTAINER_TAG: "1.10.10-python3.6"
AIRFLOW__KUBERNETES__WORKER_CONTAINER_IMAGE_PULL_POLICY: "IfNotPresent"
AIRFLOW__KUBERNETES__WORKER_PODS_CREATION_BATCH_SIZE: "10"
AIRFLOW__KUBERNETES__LOGS_VOLUME_CLAIM: "efs-claim"
AIRFLOW__KUBERNETES__GIT_REPO: "git@github.com:org/myrepo.git"
AIRFLOW__KUBERNETES__GIT_BRANCH: "develop"
AIRFLOW__KUBERNETES__GIT_DAGS_FOLDER_MOUNT_POINT: "/opt/airflow/dags"
AIRFLOW__KUBERNETES__DAGS_VOLUME_SUBPATH: "repo/"
AIRFLOW__KUBERNETES__GIT_SSH_KEY_SECRET_NAME: "airflow-git-keys"
AIRFLOW__KUBERNETES__NAMESPACE: "airflow"
AIRFLOW__KUBERNETES__DELETE_WORKER_PODS: "True"
AIRFLOW__KUBERNETES__RUN_AS_USER: "50000"
AIRFLOW__CORE__LOAD_EXAMPLES: "False"
AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL: "60"
AIRFLOW__KUBERNETES__WORKER_SERVICE_ACCOUNT_NAME: "airflow"
podAnnotations: {}
extraEnv: []
extraConfigmapMounts: []
extraContainers: []
extraPipPackages: []
extraVolumeMounts: []
extraVolumes: []
scheduler:
resources: {}
nodeSelector: {}
affinity: {}
tolerations: []
labels: {}
podLabels: {}
annotations: {}
podAnnotations: {}
podDisruptionBudget:
enabled: true
maxUnavailable: "100%"
minAvailable: ""
connections:
- id: MyS3Conn
type: aws
extra: |
{
"aws_access_key_id": "XXXXXXXXX",
"aws_secret_access_key": "XXXXXXXX",
"region_name":"us-west-1"
}
refreshConnections: true
variables: |
{}
pools: |
{}
numRuns: -1
initdb: true
preinitdb: false
initialStartupDelay: 0
extraInitContainers: []
web:
resources: {}
replicas: 1
nodeSelector: {}
affinity: {}
tolerations: []
labels: {}
podLabels: {}
annotations: {}
podAnnotations: {}
service:
annotations: {}
sessionAffinity: "None"
sessionAffinityConfig: {}
type: ClusterIP
externalPort: 8080
loadBalancerIP: ""
loadBalancerSourceRanges: []
nodePort:
http: ""
baseUrl: "http://localhost:8080"
serializeDAGs: false
extraPipPackages: []
initialStartupDelay: 0
minReadySeconds: 5
readinessProbe:
enabled: false
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 1
successThreshold: 1
failureThreshold: 3
livenessProbe:
enabled: true
scheme: HTTP
initialDelaySeconds: 300
periodSeconds: 30
timeoutSeconds: 3
successThreshold: 1
failureThreshold: 2
secretsDir: /var/airflow/secrets
secrets: []
secretsMap:
workers:
enabled: false
resources: {}
replicas: 1
nodeSelector: {}
affinity: {}
tolerations: []
labels: {}
podLabels: {}
annotations: {}
podAnnotations: {}
autoscaling:
enabled: false
maxReplicas: 2
metrics: []
initialStartupDelay: 0
celery:
instances: 1
gracefullTermination: false
gracefullTerminationPeriod: 600
terminationPeriod: 60
secretsDir: /var/airflow/secrets
secrets: []
secretsMap:
flower:
enabled: false
resources: {}
nodeSelector: {}
affinity: {}
tolerations: []
labels: {}
podLabels: {}
annotations: {}
podAnnotations: {}
basicAuthSecret: ""
basicAuthSecretKey: ""
urlPrefix: ""
service:
annotations: {}
type: ClusterIP
externalPort: 5555
loadBalancerIP: ""
loadBalancerSourceRanges: []
nodePort:
http: ""
initialStartupDelay: 0
extraConfigmapMounts: []
logs:
path: /opt/airflow/logs
persistence:
enabled: true
existingClaim: efs-claim
subPath: ""
storageClass: efs-sc
accessMode: ReadWriteMany
size: 1Gi
dags:
path: /opt/airflow/dags
doNotPickle: false
installRequirements: false
persistence:
enabled: false
existingClaim: ""
subPath: ""
storageClass: ""
accessMode: ReadOnlyMany
size: 1Gi
git:
url: git@github.com:org/myrepo.git
ref: develop
secret: airflow-git-keys
sshKeyscan: false
privateKeyName: id_rsa
repoHost: github.com
repoPort: 22
gitSync:
enabled: true
resources: {}
image:
repository: alpine/git
tag: latest
pullPolicy: Always
refreshTime: 60
initContainer:
enabled: false
resources: {}
image:
repository: alpine/git
tag: latest
pullPolicy: Always
mountPath: "/dags"
syncSubPath: ""
ingress:
enabled: false
web:
annotations: {}
path: ""
host: ""
livenessPath: ""
tls:
enabled: false
secretName: ""
precedingPaths: []
succeedingPaths: []
flower:
annotations: {}
path: ""
host: ""
livenessPath: ""
tls:
enabled: false
secretName: ""
rbac:
create: true
serviceAccount:
create: true
name: ""
annotations: {}
extraManifests: []
postgresql:
enabled: true
postgresqlDatabase: airflow
postgresqlUsername: postgres
postgresqlPassword: airflow
existingSecret: ""
existingSecretKey: "postgresql-password"
persistence:
enabled: true
storageClass: ""
accessModes:
- ReadWriteOnce
size: 5Gi
externalDatabase:
type: postgres
host: localhost
port: 5432
database: airflow
user: airflow
passwordSecret: ""
passwordSecretKey: "postgresql-password"
redis:
enabled: false
password: airflow
existingSecret: ""
existingSecretKey: "redis-password"
cluster:
enabled: false
slaveCount: 1
master:
resources: {}
persistence:
enabled: false
storageClass: ""
accessModes:
- ReadWriteOnce
size: 8Gi
slave:
resources: {}
persistence:
enabled: false
storageClass: ""
accessModes:
- ReadWriteOnce
size: 8Gi
externalRedis:
host: localhost
port: 6379
databaseNumber: 1
passwordSecret: ""
passwordSecretKey: "redis-password"
serviceMonitor:
enabled: false
selector:
prometheus: kube-prometheus
path: /admin/metrics
interval: "30s"
prometheusRule:
enabled: false
additionalLabels: {}
groups: []
我不确定如果有人知道如何解决权限错误，该怎么办.
I'm not really sure what to do here if anyone knows how to fix the permission error.
推荐答案
您可以将extraInitContainers与调度程序一起使用以更改权限，诸如此类.
You can use extraInitContainers with scheduler to change the permission, something like this.
extraInitContainers:
- name: volume-logs
image: busybox
command: ["sh", "-c", "chown -R 50000:50000 /opt/airflow/logs/"]
volumeMounts:
- mountPath: /opt/airflow/logs/
name: logs-data
这将更改安装点的权限.
This will change permission of the mount point.
这篇关于Kubernetes上的气流:Errno 13-权限被拒绝:'/opt/airflow/logs/scheduler的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！