运行bashoperator时气流重置环境变量

问题描述

在执行一项气流任务时，我遇到了环境变量问题.

With one of my airflow task, I have an environment variable issue.

[2019-08-19 04:51:04,603] {{bash_operator.py:127}} INFO -   File "/home/ubuntu/.pyenv/versions/3.6.7/lib/python3.6/os.py", line 669, in __getitem__
[2019-08-19 04:51:04,603] {{bash_operator.py:127}} INFO -     raise KeyError(key) from None
[2019-08-19 04:51:04,603] {{bash_operator.py:127}} INFO - KeyError: 'HOME'
[2019-08-19 04:51:04,639] {{bash_operator.py:131}} INFO - Command exited with return code 1

我的任务如下:

task_name = BashOperator(
    task_id='task_name',
    bash_command="cd path/to/manage.py && export LC_ALL=C.UTF-8 && export LANG=C.UTF-8 "
    f'&& {Variable.get("python_virtualenv_path")}virtual-env-name/bin/python manage.py command_name',
    retries=1,
    pool='LightAndFast',
    dag=dag
)

对这个问题有什么想法吗?

Any ideas of this issue?

推荐答案

这是事实，气流使用BashOperator重置环境变量，至少我遇到了这个问题.在操作员的文档中，可从以下位置获得: https://airflow.apache.org/docs/stable/_modules/airflow/operators/bash_operator.html ，我找到了显式设置bash命令环境的方法，即

This is true that airflow resets environment variable, when using BashOperator, at least I faced this issue. In documentation of the operator, available at :https://airflow.apache.org/docs/stable/_modules/airflow/operators/bash_operator.html,I found the way to explicitly set the environment for the bash command i.e.

bash_task = BashOperator(
        task_id="bash_task",
        bash_command='echo "here is the message: \'$message\'"',
        env={'message': '{{ dag_run.conf["message"] if dag_run else "" }}'},
    )

因此，我将Bash命令的环境明确设置为:

Hence I explicitly set the environment for the Bash command as:

env = os.environ.copy(),

请确保先在 dag 文件中选择import os.它为我解决了这个问题.

make sure to import os, earlier on in the dag file. And it resolved the issue for me.

这篇关于运行bashoperator时气流重置环境变量的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！