原文:Debugging Airflow Dags;作者/维护方:Apache Airflow 项目贡献者 / The Apache Software Foundation。中文翻译与技术整理:未完纪。核验日期:2026-10-05。
在 IDE 里逐步调试 Airflow DAG,可以给 DAG 文件添加 dag.test() 入口,然后在一个串行的 Python 进程中检查任务执行。原指南指出,这种方式兼容受支持的数据库,包括本地 SQLite;默认情况下所有任务在一个进程里运行,错误更容易直接暴露。
这里的 test 并不代表模拟执行。dag.test() 会创建并执行真实的 DAG run。任务中的网络请求、数据写入、发信和删除操作仍可能发生。本稿只审查代码,不执行这些任务。
先满足元数据库与 bundle 前置条件
当前 Airflow 3.3.2 文档列出两个条件:元数据库已经初始化;DAG 位于 Airflow 配置的 DAG bundle 内,可以被序列化。默认 bundle 对应你的 DAG 文件目录。dag.test() 会重新解析 bundle 并序列化 DAG。
元数据库迁移对应的命令是 airflow db migrate。它会修改数据库结构,不能把生产数据库当临时测试库;先配置独立的测试元数据库,再按该版本的安装流程初始化。
如果仅在 pytest 函数内部,或在配置目录之外的文件中临时构造 DAG,Airflow 可能找不到可序列化的 bundle,进而报出:
Cannot create DagRun for DAG <dag_id> because the dag is not serialized
如果只想验证 operator 的行为,而不需要整个 DAG run,应采用 Airflow 单元测试方法,直接测试适当的 operator 或业务函数。完整 DAG 调试与纯单元测试解决的问题不同。

给 DAG 文件增加调试入口
在已经定义好 dag 的文件末尾加入:
if __name__ == "__main__":
dag.test()
这样,直接运行文件时才进入调试,而正常导入模块不会因这段入口主动执行 DAG。原指南还列出可选参数:execution_date 用来指定测试运行的日期;use_executor 则决定是否使用环境中配置的执行器。默认不使用执行器,任务在本地执行;启用执行器后,可能由该环境的执行器派发任务。
版本提示:参数名和日期语义应以实际安装版的 DAG.test 签名为准。本文保留原文列出的名称,不保证 Airflow 其他版本具有相同接口;不要在未核对环境配置时启用 use_executor,以免把调试任务送往远程工作节点。
按任务 ID 跳过外部等待或清理
如果本地调试不希望执行某些任务,例如等待上游 DAG 的 sensor 或删除中间文件的清理任务,可以通过 mark_success_pattern 传入匹配 task_id 的模式。命中的任务被直接标记为成功。
原文的图式包含两个 ExternalTaskSensor:wait_for_ingestion_dag 等待 ingest_raw_data,wait_for_dim_dag 等待 ingest_dim。两个 sensor 都在统计任务之前,统计任务后面是清理步骤:
collect_stats = PythonOperator(
task_id="extract_stats_csv",
python_callable=extract_stats_csv,
)
# 原文依赖关系(其余任务定义省略)
[sensor, sensor2] >> collect_stats >> cleanup
原文完整静态示例(包含已指出的问题)
以下保留原代码供来源核对。Path.unlink 会删除文件,末行任务 ID 和 XCom 参数另有下文标注的问题;不得把此块视为已修复或可直接执行的配置。
with DAG("example_dag", default_args=default_args) as dag:
sensor = ExternalTaskSensor(task_id="wait_for_ingestion_dag", external_dag_id="ingest_raw_data")
sensor2 = ExternalTaskSensor(task_id="wait_for_dim_dag", external_dag_id="ingest_dim")
collect_stats = PythonOperator(task_id="extract_stats_csv", python_callable=extract_stats_csv)
# ... run other tasks
cleanup = PythonOperator(task_id="cleanup", python_callable=Path.unlink, op_args=[collect_stats.output])
[sensor, sensor2] >> collect_stats >> cleanup
if __name__ == "__main__":
ingest_testing_data()
run = dag.test(mark_success_pattern="wait_for_.*|cleanup")
print(f"Intermediate csv: {run.get_task_instance('collect_stats').xcom_pull(task_id='collect_stats')}")
原例先调用项目自己的 ingest_testing_data() 准备测试数据,再以 wait_for_.*|cleanup 跳过两个等待节点和清理节点。这样无需等待真实上游流程,并保留中间 CSV 供检查。ingest_testing_data、默认参数、统计函数和完整导入都不是这段文档提供的通用实现,不能直接复制成独立脚本。
if __name__ == "__main__":
ingest_testing_data() # 项目提供;必须仅指向独立测试数据
run = dag.test(
mark_success_pattern="^(wait_for_ingestion_dag|wait_for_dim_dag|cleanup)$"
)
编辑改动:这里把原文的宽泛模式改成带起止锚点的三个精确任务 ID,减少误把新加入的等待任务标为成功的可能。模式仍需按自己的 DAG 调整;跳过 sensor 不代表上游结果真的存在,跳过清理也可能留下敏感测试数据。
原文清理节点把 Path.unlink 作为可调用对象,输入取自统计任务输出,这会删除文件。本文仅静态引用这一删除示例,不执行它。仅在确认是独立测试目录且有保留策略时,才应设计自己的清理步骤。
不要混淆 Python 变量名与 task_id
原文随后试图通过 run.get_task_instance('collect_stats') 和以 collect_stats 为目标的 XCom 调用打印 CSV 路径,但前面实际注册的 task_id 是 extract_stats_csv。Python 变量名不会自动成为任务 ID。
应把查找目标统一到 extract_stats_csv,同时核对所用 Airflow 版本中任务实例与 XCom API 的参数签名。原例还使用 xcom_pull(task_id=...);本稿没有执行验证该调用,因此不把“只替换字符串”声称为已经修复并通过测试的完整方案。若只是验证产生的 CSV,更直接的办法是在统计任务中观察输出,并以本版本正式 API 回读。
在命令行和 IDE 里设置断点
同样的 main 入口也适用于 pdb。原文给出的调用形式是:
python -m pdb <path-to-dag-file>.py
进入 pdb 后,用 b 45 在示例第 45 行设置断点,c 继续运行;到达断点后可以输入变量名查看对象。行号应按你自己的文件调整。原页展示的终端输出来自作者示例,本文没有复现该输出。
以下为源文的 pdb 演示记录,仅保留作者输出,不是本文执行结果:
[Breeze:3.10.19] root@ef2c84ad4856:/opt/airflow# python -m pdb providers/standard/src/airflow/providers/standard/example_dags/example_bash_operator.py
> /opt/airflow/providers/standard/src/airflow/providers/standard/example_dags/example_bash_operator.py(18)<module>()
-> """Example Dag demonstrating the usage of the BashOperator."""
(Pdb) b 45
Breakpoint 1 at /opt/airflow/providers/standard/src/airflow/providers/standard/example_dags/example_bash_operator.py:45
(Pdb) c
> /opt/airflow/providers/standard/src/airflow/providers/standard/example_dags/example_bash_operator.py(45)<module>()
-> task_id="run_after_loop",
(Pdb) run_this_last
<Task(EmptyOperator): run_this_last>
IDE 流程同样简单:在 DAG 文件末尾保留 main 入口,设置断点,再以运行或调试方式启动这个文件。调试器暂停进程不等于撤销已发生的外部操作,所以仍应先检查数据库连接、执行器、网络凭据和任务副作用。
来源、版本与使用说明
依据 Airflow 3.3.2 官方调试指南翻译整理,© The Apache Software Foundation,Apache License 2.0。精确匹配模式、任务 ID 差异及隔离说明为编辑标注;未安装 Airflow、初始化数据库、执行 DAG 或删除文件。静态审查不构成执行测试。
版权与许可全文
以下保留本页涉及的来源材料或示例代码的版权、许可条件与免责声明;各自适用范围依原声明。中文翻译及编辑标注:未完纪,2026-10-05。
LICENSE-Apache-2.0.txt
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.













暂无评论内容