Pyxis 与 Slurm:加载容器插件、执行任务并核验宿主挂载

Pyxis 是 Slurm Workload Manager 的 SPANK 插件。它把容器选项直接加到 srun 等作业入口,使无管理员权限的集群用户也能运行容器化任务。底层容器工具是 Enroot,Pyxis 负责把它接到 Slurm 作业流程中。pyxis 原指古代的小盒子或容器。

本文依据 NVIDIA 官方项目 v0.24.0 标签的完整 README 翻译整理,核查日期 2026-10-05。原页未列出可确认的个人作者,归属 NVIDIA Pyxis 项目及贡献者。本稿只做静态审查,未安装插件、重启 Slurm、提交作业或运行测试。

Slurm计算节点上Pyxis通过Enroot创建容器,容器的/etc/os-release来自镜像,只读挂载后的/host/os-release来自宿主。
原创技术示意图:检查路径来源,区分镜像文件与显式挂入的宿主文件;未完纪编辑整理,非运行截图。

能力与适用边界

原文列出的优势包括:在无特权容器中执行用户任务;命令行接口简单;Docker 镜像层下载缓存及跨用户共享;在 Slurm 支持的前提下,通过 PMI2 或 PMIx 运行多节点 MPI;允许用户在容器内安装软件包;配合共享文件系统;无需在整个集群统一管理 subordinate user/group ID。

无特权描述容器运行方式,不是对宿主文件、凭证、网络和集群配置的完整隔离保证。宿主挂载、导出的环境变量和 Enroot hooks 都会改变容器可见的内容。本文用系统自带的 os-release 与 grep 检查基础行为,不扩展成 GPU 训练教程。

安装:先匹配 Slurm release

v0.24.0 README 要求预先安装 Enroot,文中版本写作 3.1.0;不能从这个历史说明推断任意新版本都兼容。尤其从 Slurm 21.08 开始,Pyxis 必须针对集群实际部署的 Slurm release 编译。如果使用另一 release 的 spank.h,Slurm 会拒绝加载并报 Incompatible plugin version。

安装由管理员在维护流程下完成,须核对源码标签、Slurm 开发头文件、架构、Enroot 配置及插件加载目录。下面保留三种原文安装方式,按环境选择一种。它们会写系统目录并重启 slurmd,不是普通用户可以随意在生产节点执行的无副作用检查。

make install

sudo make install
sudo ln -s /usr/local/share/pyxis/pyxis.conf /etc/slurm/plugstack.conf.d/pyxis.conf
sudo systemctl restart slurmd

命令在已准备好的 Pyxis 源码和编译环境中执行。sudo make install 会以管理员权限执行项目安装规则,应先验证源码来源。符号链接要求父目录及加载配置与集群实际布局一致;目标已存在时先检查,不要自动删除旧配置或强行覆盖。

Debian 包

make orig
make deb
sudo dpkg -i ../nvslurm-plugin-pyxis_*_amd64.deb
sudo ln -s /usr/share/pyxis/pyxis.conf /etc/slurm/plugstack.conf.d/pyxis.conf
sudo systemctl restart slurmd

原文包名面向 amd64。通配符可能匹配多个历史构建,正式安装前应核对生成文件,选择经过校验的唯一版本;通配符不会自动判断哪个版本与集群兼容。

RPM 包

make rpm
sudo rpm -i rpm/RPMS/x86_64/nvslurm-plugin-pyxis-*.x86_64.rpm
sudo ln -s /usr/share/pyxis/pyxis.conf /etc/slurm/plugstack.conf.d/pyxis.conf
sudo systemctl restart slurmd

RPM 示例面向 x86_64,具有相同的源码信任、包文件多匹配与节点重启边界。安装后用 srun --help 核对 Pyxis 选项是否出现;如果没有,应检查加载配置和二进制兼容性,不能仅凭包管理器成功判断插件已接通。

容器参数完整说明

作为 SPANK 插件,Pyxis 直接扩充 srun 的帮助信息。下表保留 v0.24.0 README 列出的全部选项:

参数 含义
--container-image=[USER@][REGISTRY#]IMAGE[:TAG]|PATH 容器文件系统可以是 Enroot URI 形式的 Docker 镜像,或远端计算节点文件系统中的 SquashFS 路径。提交端的任意本地文件不会因写入路径就自动上传。
--container-mounts=SRC:DST[:FLAGS][,SRC:DST...] 宿主绑定挂载。多个挂载用逗号,多个标志用加号分隔,例如 ro+rprivate。
--container-workdir=PATH 容器内工作目录。
--container-name=NAME 保存和加载宿主上的命名容器。未命名容器在 Slurm 任务结束后删除;命名容器保留。同名容器已经存在时,复用已有容器并跳过镜像导入。
--container-save=PATH 把容器状态保存为远端宿主文件系统中的 SquashFS 文件,需要检查目标路径、容量与文件内是否含秘密。
--container-mount-home / --no-container-mount-home 请求绑定或不绑定用户主目录。系统级 Enroot 设置可能已经挂载主目录,不能只依据单个命令行开关推断最终挂载清单。
--container-remap-root / --no-container-remap-root 请求或禁止在容器内映射为 root。显示成容器 root 本身不会赋予宿主系统额外权限。
--container-entrypoint / --no-container-entrypoint 执行或不执行镜像 entrypoint;entrypoint 是可执行代码,必须审查来源。
--container-entrypoint-log 打印 entrypoint 输出,注意日志可能包含配置与秘密。
--container-writable / --container-readonly 请求容器文件系统可写或只读。根文件系统只读不能代替每个宿主挂载的单独权限控制。
--container-env=NAME[,NAME...] 指定从宿主保留并覆盖镜像值的环境变量,在 entrypoint 之前设置。默认情况下,导出的宿主变量在 entrypoint 执行后进入容器,但镜像已有同名值优先;本参数列出的变量以宿主值优先并提前设置。
--container-unshare=NS[,NS...] 请求新建命名空间,原文列出 net、ipc、uts。需检查网络、MPI及站点配置依赖。

命名容器复用、环境变量继承、主目录和其他绑定挂载,都可能让两次相同命令获得不同状态。无特权运行不会自动清空凭据;只向任务提供必要的目录与环境,检查站点 Enroot 配置和实际挂载。

用 grep 核对文件来源

首先让 Slurm 在计算节点直接读取系统信息:

srun grep PRETTY /etc/os-release

这里的宿主是分配到的计算节点,不一定是运行提交命令的登录节点。原文输出为 PRETTY_NAME="Ubuntu 24.04.3 LTS",只是原文机器的示例,不是本稿实测或当前集群的应有版本。

再把同一个检查放入镜像:

srun --container-image=almalinux:9 grep PRETTY /etc/os-release

原文先显示导入 almalinux:9 的进度,再输出 PRETTY_NAME="AlmaLinux 9.7 (Moss Jungle Cat)",表示读取了镜像中的文件。almalinux:9 是移动标签,后来可能指向另一份镜像;要复现,应按所用 Enroot 支持的方式固定经审核的镜像摘要或 SquashFS 工件。

最后把计算节点文件显式挂入容器,改读挂载目标:

srun --container-image=almalinux:9 \
  --container-mounts=/etc/os-release:/host/os-release:ro+rprivate \
  grep PRETTY /host/os-release

与原文的差异:原挂载未指定标志,本文为只读验收目的增加 ro+rprivate。ro 限制通过此挂载写入,rprivate 隔离挂载传播;它不证明所有路径和句柄都已隔离。修改只经过静态核查,未在实际站点执行。

原文再次得到 Ubuntu 的 PRETTY_NAME,因为 /host/os-release 来自宿主。三步核对的是路径来源,不是哪个发行版名称更正确。它们不能证明主目录未挂载、容器没有网络、凭据未继承或整个集群已经安全隔离。

批处理入口:sbatch

Pyxis 也可使批处理脚本在容器镜像中运行。下例保留 README 的历史镜像 nvcr.io#nvidia/pytorch:21.12-py3,只导入 PyTorch 并打印版本,没有训练或性能测量:

sbatch --wait -o slurm.out <<'EOF'
#!/bin/bash
#SBATCH --container-image nvcr.io\#nvidia/pytorch:21.12-py3

python -c 'import torch ; print(torch.__version__)'
EOF

cat slurm.out

与原文的差异:here-document 的分隔符从 <<EOF 改成 <<‘EOF’,避免以后在脚本中加入的 $变量或命令替换在提交端提前展开;原固定脚本并没有这类动态内容。# 在 SBATCH 指令中具有注释意义,用于注册表与镜像的分隔符时必须保留转义 \#。

原文 slurm.out 展示镜像导入消息及 1.11.0a0+b6df043。这是旧镜像的历史输出,不是推荐当前安装此版本。镜像还可能受注册表访问、许可与站点策略限制,应核查来源、已知漏洞、可用性及凭据注入方式。–wait 会等待作业结束;输出文件写入位置和已有文件须检查,避免覆盖需要保留的结果。

测试、进阶资料和问题反馈

项目通过 Bats 运行集成测试。README 给出的命令为:

salloc --overcommit bats tests
bats tests/sbatch

原文要求在 Slurm 作业分配内运行相关集成测试。部分测试假设特定 Enroot 配置,例如 PMIx/PyTorch hooks,因此不保证所有系统均通过;这些命令会使用调度资源,不是只读静态检查。本稿没有运行测试,未把帮助信息出现参数称为集成测试通过。

进阶资料是原文 Wiki 的 Home、Installation、Setup、Usage。这里保留进一步阅读链接,不声称已经全文核验了另一套 Wiki 教程。

一般问题可提交 issue;贡献前先阅读 CONTRIBUTING 再提交 pull request。原项目特别要求安全问题不要公开开 issue 或 pull request,而应负责地向 psirt<at>nvidia.com 披露。本文仅保留渠道说明,没有发送消息。

归属、许可与审核说明

原作和维护:NVIDIA Pyxis 项目及贡献者。项目采用 Apache License 2.0,同标签 LICENSE 保留 Copyright 2019-2020 NVIDIA CORPORATION,完整文本见下方原始许可全文。本文明确标注翻译整理、挂载标志、here-document 引号和安全边界等改动,不表示 NVIDIA 或 Slurm 为新增内容背书。

静态审查覆盖安装命令、全部容器参数、srun、sbatch 和测试入口,说明了特权安装、节点重启、通配符匹配、宿主挂载、环境继承、状态复用及旧镜像风险。未运行原文或修正版,没有发现硬编码秘密也不等于没有漏洞。文稿通过来源与编辑核查,不是生产部署验证报告。

原始版权与许可全文

Apache License
                           Version 2.0, January 2004
                        http://www.apache.org/licenses/

   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION

   1. Definitions.

      "License" shall mean the terms and conditions for use, reproduction,
      and distribution as defined by Sections 1 through 9 of this document.

      "Licensor" shall mean the copyright owner or entity authorized by
      the copyright owner that is granting the License.
      "Legal Entity" shall mean the union of the acting entity and all
      other entities that control, are controlled by, or are under common
      control with that entity. For the purposes of this definition,
      "control" means (i) the power, direct or indirect, to cause the
      direction or management of such entity, whether by contract or
      otherwise, or (ii) ownership of fifty percent (50%) or more of the
      outstanding shares, or (iii) beneficial ownership of such entity.
      "You" (or "Your") shall mean an individual or Legal Entity
      exercising permissions granted by this License.

      "Source" form shall mean the preferred form for making modifications,
      including but not limited to software source code, documentation
      source, and configuration files.
      "Object" form shall mean any form resulting from mechanical
      transformation or translation of a Source form, including but
      not limited to compiled object code, generated documentation,
      and conversions to other media types.

      "Work" shall mean the work of authorship, whether in Source or
      Object form, made available under the License, as indicated by a
      copyright notice that is included in or attached to the work
      (an example is provided in the Appendix below).
      "Derivative Works" shall mean any work, whether in Source or Object
      form, that is based on (or derived from) the Work and for which the
      editorial revisions, annotations, elaborations, or other modifications
      represent, as a whole, an original work of authorship. For the purposes
      of this License, Derivative Works shall not include works that remain
      separable from, or merely link (or bind by name) to the interfaces of,
      the Work and Derivative Works thereof.
      "Contribution" shall mean any work of authorship, including
      the original version of the Work and any modifications or additions
      to that Work or Derivative Works thereof, that is intentionally
      submitted to Licensor for inclusion in the Work by the copyright owner
      or by an individual or Legal Entity authorized to submit on behalf of
      the copyright owner. For the purposes of this definition, "submitted"
      means any form of electronic, verbal, or written communication sent
      to the Licensor or its representatives, including but not limited to
      communication on electronic mailing lists, source code control systems,
      and issue tracking systems that are managed by, or on behalf of, the
      Licensor for the purpose of discussing and improving the Work, but
      excluding communication that is conspicuously marked or otherwise
      designated in writing by the copyright owner as "Not a Contribution."
      "Contributor" shall mean Licensor and any individual or Legal Entity
      on behalf of whom a Contribution has been received by Licensor and
      subsequently incorporated within the Work.
   2. Grant of Copyright License. Subject to the terms and conditions of
      this License, each Contributor hereby grants to You a perpetual,
      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
      copyright license to reproduce, prepare Derivative Works of,
      publicly display, publicly perform, sublicense, and distribute the
      Work and such Derivative Works in Source or Object form.
   3. Grant of Patent License. Subject to the terms and conditions of
      this License, each Contributor hereby grants to You a perpetual,
      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
      (except as stated in this section) patent license to make, have made,
      use, offer to sell, sell, import, and otherwise transfer the Work,
      where such license applies only to those patent claims licensable
      by such Contributor that are necessarily infringed by their
      Contribution(s) alone or by combination of their Contribution(s)
      with the Work to which such Contribution(s) was submitted. If You
      institute patent litigation against any entity (including a
      cross-claim or counterclaim in a lawsuit) alleging that the Work
      or a Contribution incorporated within the Work constitutes direct
      or contributory patent infringement, then any patent licenses
      granted to You under this License for that Work shall terminate
      as of the date such litigation is filed.
   4. Redistribution. You may reproduce and distribute copies of the
      Work or Derivative Works thereof in any medium, with or without
      modifications, and in Source or Object form, provided that You
      meet the following conditions:

      (a) You must give any other recipients of the Work or
          Derivative Works a copy of this License; and

      (b) You must cause any modified files to carry prominent notices
          stating that You changed the files; and
      (c) You must retain, in the Source form of any Derivative Works
          that You distribute, all copyright, patent, trademark, and
          attribution notices from the Source form of the Work,
          excluding those notices that do not pertain to any part of
          the Derivative Works; and
      (d) If the Work includes a "NOTICE" text file as part of its
          distribution, then any Derivative Works that You distribute must
          include a readable copy of the attribution notices contained
          within such NOTICE file, excluding those notices that do not
          pertain to any part of the Derivative Works, in at least one
          of the following places: within a NOTICE text file distributed
          as part of the Derivative Works; within the Source form or
          documentation, if provided along with the Derivative Works; or,
          within a display generated by the Derivative Works, if and
          wherever such third-party notices normally appear. The contents
          of the NOTICE file are for informational purposes only and
          do not modify the License. You may add Your own attribution
          notices within Derivative Works that You distribute, alongside
          or as an addendum to the NOTICE text from the Work, provided
          that such additional attribution notices cannot be construed
          as modifying the License.
      You may add Your own copyright statement to Your modifications and
      may provide additional or different license terms and conditions
      for use, reproduction, or distribution of Your modifications, or
      for any such Derivative Works as a whole, provided Your use,
      reproduction, and distribution of the Work otherwise complies with
      the conditions stated in this License.
   5. Submission of Contributions. Unless You explicitly state otherwise,
      any Contribution intentionally submitted for inclusion in the Work
      by You to the Licensor shall be under the terms and conditions of
      this License, without any additional terms or conditions.
      Notwithstanding the above, nothing herein shall supersede or modify
      the terms of any separate license agreement you may have executed
      with Licensor regarding such Contributions.
   6. Trademarks. This License does not grant permission to use the trade
      names, trademarks, service marks, or product names of the Licensor,
      except as required for reasonable and customary use in describing the
      origin of the Work and reproducing the content of the NOTICE file.
   7. Disclaimer of Warranty. Unless required by applicable law or
      agreed to in writing, Licensor provides the Work (and each
      Contributor provides its Contributions) on an "AS IS" BASIS,
      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
      implied, including, without limitation, any warranties or conditions
      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
      PARTICULAR PURPOSE. You are solely responsible for determining the
      appropriateness of using or redistributing the Work and assume any
      risks associated with Your exercise of permissions under this License.
   8. Limitation of Liability. In no event and under no legal theory,
      whether in tort (including negligence), contract, or otherwise,
      unless required by applicable law (such as deliberate and grossly
      negligent acts) or agreed to in writing, shall any Contributor be
      liable to You for damages, including any direct, indirect, special,
      incidental, or consequential damages of any character arising as a
      result of this License or out of the use or inability to use the
      Work (including but not limited to damages for loss of goodwill,
      work stoppage, computer failure or malfunction, or any and all
      other commercial damages or losses), even if such Contributor
      has been advised of the possibility of such damages.
   9. Accepting Warranty or Additional Liability. While redistributing
      the Work or Derivative Works thereof, You may choose to offer,
      and charge a fee for, acceptance of support, warranty, indemnity,
      or other liability obligations and/or rights consistent with this
      License. However, in accepting such obligations, You may act only
      on Your own behalf and on Your sole responsibility, not on behalf
      of any other Contributor, and only if You agree to indemnify,
      defend, and hold each Contributor harmless for any liability
      incurred by, or claims asserted against, such Contributor by reason
      of your accepting any such warranty or additional liability.
   END OF TERMS AND CONDITIONS

   Copyright 2019-2020 NVIDIA CORPORATION

   Licensed under the Apache License, Version 2.0 (the "License");
   you may not use this file except in compliance with the License.
   You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0
   Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.
© 版权声明
THE END
喜欢就支持一下吧
点赞0 分享
评论 抢沙发

请登录后发表评论

    暂无评论内容