为 StarRocks 存算分离集群准备快照恢复演练

整理自 StarRocks 官方中文文档《自动化集群快照》,维护方:StarRocks 文档团队。本文固定核对 branch-3.5 文档;集群快照从 v3.4.2 起支持,在该版本文档中仍标记为 Beta,且只适用于存算分离集群。全文已于 2026-10-05 核对,下面的 SQL、Shell 和 YAML 只做静态审查,没有连接数据库或执行恢复。

要恢复一个存算分离集群,仅仅留住表数据还不够。CN 管理的数据已经放在对象存储中,但 FE 保存的目录、用户、权限以及导入任务等元数据仍在本地。自动化集群快照的作用,就是把恢复所需的元数据也保存到存储卷,使完整的集群状态有机会在故障后重新组装起来。

StarRocks 3.5 自动化快照与独立恢复演练流程:FE 元数据镜像和 CN 数据进入存储,最新快照用于恢复 Leader FE、其他 FE 与 CN;外部 Catalog 配置和本地 UDF JAR 需另行备份。
未完纪原创技术示意图,依据 StarRocks 3.5 官方文档绘制;不是运行截图。
StarRocks原文快照任务状态图:INITIALIZING、SNAPSHOTTING、UPLOADING、FINISHED、EXPIRED、DELETED;失败或Leader FE重启可转入ERROR。
原文工作流程图,来源:StarRocks 3.5 官方文档。Copyright 2021-present StarRocks, Inc. 原图保留,非本稿运行结果。图中写作 SNAPSHOTTING,并含 UPLOADING;正文3.5字段表写作 SNAPSHOTING,未列UPLOADING。这里保留来源差异,实际状态值应按安装版本接口核对。

先明确快照能保存什么

集群快照记录某一时刻的集群状态,覆盖 Catalog、数据库、表、用户及权限、导入任务等对象。外部依赖并不自动包含在内,例如 external catalog 使用的配置文件和放在本地的 UDF JAR。因此,演练材料除了快照,还应有这些依赖的独立副本、适配当前版本的部署文件,以及能够访问存储卷的授权。

在这里,“自动化”有一个很具体的含义:系统持续维护紧随集群最新状态的一份快照。新快照生成后,历史快照会被删除,最终只保留一个。3.5 分支文档明确说明,快照任务只由系统触发,不支持手动创建。它不能替代能够回到多个历史时刻的备份链,也不能因为名称中有时间戳就被理解为任意时间点恢复。

灾难恢复的前提是数据和元数据本身仍然完整。云存储的备份和跨区域复制可以参与远程或跨区域灾备,但本文不会把“存储服务有这些功能”等同于“已经获得一致且可恢复的副本”。实际演练必须同时检查副本、访问权限与外部依赖。

启用自动化快照并理解保存位置

自动化集群快照默认关闭。官方给出的启用语法如下,方括号表示可选部分,尖括号表示需要替换的值,不能原样执行:

ADMIN SET AUTOMATED CLUSTER SNAPSHOT ON
[STORAGE VOLUME <storage_volume_name>]

storage_volume_name 指定快照使用的存储卷;不填写时使用默认存储卷。创建存储卷的方法见原站 CREATE STORAGE VOLUME 文档。恢复演练开始前,应确认该卷的地址和访问权限与准备保全的数据一致。

每次 FE 完成元数据检查点并生成新的元数据镜像后,系统会自动创建快照。名称格式是 automated_cluster_snapshot_{timestamp}。元数据快照保存在以下目录,而数据快照仍位于原始数据所在的位置:

/{storage_volume_locations}/{service_id}/meta/image/automated_cluster_snapshot_timestamp

FE 配置 automated_cluster_snapshot_interval_seconds 控制自动化周期,默认 600 秒,也就是 10 分钟。这个默认值描述触发周期,不能被写成灾难发生后必然只损失 10 分钟数据的保证;还需要观察任务是否完成,以及对象存储中的数据是否齐备。

分别检查快照和快照任务

快照清单和任务执行状态应分别查询。前者可看到最新快照,以及尚未清除的快照记录:

SELECT * FROM information_schema.cluster_snapshots;
字段 含义
snapshot_name 快照名称
snapshot_type 快照类型;本篇锁定的 3.5 文档仅支持 automated
created_time 快照创建时间
fe_journal_id FE 日志 ID
starmgr_journal_id StarManager 日志 ID
properties 用于尚未可用的功能
storage_volume 保存快照的存储卷
storage_path 快照存储路径

任务视图用于判断自动化过程走到哪一步,以及失败时的具体信息:

SELECT * FROM information_schema.cluster_snapshot_jobs;
字段 含义
snapshot_name 对应快照名称
job_id 任务 ID
created_time 任务创建时间
finished_time 任务结束时间
state INITIALIZING、SNAPSHOTING、FINISHED、EXPIRED、DELETED 或 ERROR
detail_info 当前阶段的具体进度
error_message 错误信息(如果有)

注意状态值 SNAPSHOTING 的拼写应按接口原样使用。看到任务创建并不等于快照可恢复;应保存对应任务的完成状态、存储路径与日志 ID,再以恢复演练验证使用它的结果。本文没有任何真实查询输出,表格列出的是接口字段。

关闭功能会清理历史快照

ADMIN SET AUTOMATED CLUSTER SNAPSHOT OFF

以上语句并非无损暂停按钮。原文明确说明,禁用自动化集群快照会自动清除历史快照。不要把它安排成“先停止写入快照,再研究恢复”的通用准备步骤。确有关闭需求时,先明确将失去哪些恢复材料,并按自己的备份方案保全所需副本;本次写稿没有执行此语句。

在独立环境恢复 Leader、其他 FE 与 CN

以下保留原文的恢复顺序,并补充演练保护说明。应先准备同版本的恢复节点、完整的快照和数据副本、存储权限及外部配置。恢复环境的地址、目录和存储路径必须经过人工核对,避免把示例中的目录清理动作误用到仍承担业务的原集群。

  1. 必要时调整存储卷与恢复配置。如果保存快照的存储位置已经改变,必须把原存储路径下的所有文件复制到新路径,并修改 Leader FE 的 fe/conf/cluster_snapshot.yaml。仅移动元数据镜像而遗漏原始数据文件,不满足原文的恢复前提。模板见下一节。

  2. 启动 Leader FE。使用集群快照模式:

    ./fe/bin/start_fe.sh --cluster_snapshot --daemon
  3. 处理其他 FE。原文要求先清理其 meta 目录,再使用 Leader 的地址和 edit-log 端口启动。本文不提供可直接粘贴的删除命令;先停止待恢复节点、保存原目录的可恢复副本,并再次确认它属于本次独立演练节点。确认完成后才进入原文的启动步骤:

    ./fe/bin/start_fe.sh --helper <leader_ip>:<leader_edit_log_port> --daemon

    按照角色把节点加入集群:

    -- 添加 Follower 节点:
    ALTER SYSTEM ADD FOLLOWER "<follower_host>:<follower_edit_log_port>";
    
    -- 添加 Observer 节点:
    ALTER SYSTEM ADD OBSERVER "<observer_host>:<observer_edit_log_port>";
  4. 处理 CN。原文要求先清理 CN 的 storage_root_path 目录,再启动 CN。这里同样必须确认目标是独立恢复节点,先保全原有文件,不能把路径清理套用到任意运行中的目录。

    ./be/bin/start_cn.sh --daemon

    随后将 CN 加入集群:

    ALTER SYSTEM ADD COMPUTE NODE "<cn_host>:<cn_heartbeat_service_port>";

如果第一步修改了 cluster_snapshot.yaml,新集群会按其中的信息重新配置节点和存储卷。完成启动后,应核对 Leader、Follower、Observer 与 CN 的实际地址及角色,确认它们属于计划中的恢复拓扑,并检查目录、权限、任务及关键业务查询。这里的核对是演练验收要求,不是本文已经完成的测试;节点成功启动也不能替代业务数据的完整性检查。

完整恢复配置模板及敏感字段

下列 YAML 保留 3.5 分支原文的结构、示例值和注释。IP、URI、访问键均是原文占位示例,不是可用凭证。它不是一份可以直接投入生产的配置:

# 要下载以进行恢复的集群快照的信息。
cluster_snapshot:
    # 快照的 URI。
    # 示例 1: s3://defaultbucket/test/f7265e80-631c-44d3-a8ac-cf7cdc7adec811019/meta/image/automated_cluster_snapshot_1704038400000
    # 示例 2: s3://defaultbucket/test/f7265e80-631c-44d3-a8ac-cf7cdc7adec811019/meta
    cluster_snapshot_path: <cluster_snapshot_uri>
    # 存储快照的存储卷名称。您必须在 `storage_volumes` 部分中定义它。
    # 注意:它必须与原始集群中的相同。
    storage_volume_name: my_s3_volume

# [可选] 要恢复快照的新集群的节点信息。
# 如果未指定此部分,恢复后的新集群仅具有 Leader FE 节点。
# CN 节点保留原始集群的信息。
# 注意:不要在此部分中包含 Leader FE 节点。

frontends:
    # FE 主机。
  - host: xxx.xx.xx.x1
    # FE edit_log_port。
    edit_log_port: 9010
    # FE 节点类型。有效值:`follower`(默认)和 `observer`。
    type: follower
  - host: xxx.xx.xx.x2
    edit_log_port: 9010
    type: observer

compute_nodes:
    # CN 主机。
  - host: xxx.xx.xx.x3
    # CN heartbeat_service_port。
    heartbeat_service_port: 9050
  - host: xxx.xx.xx.x4
    heartbeat_service_port: 9050

# 新集群中存储卷的信息。用于恢复克隆的快照。
# 注意:存储卷的名称必须与原始集群中的相同。
storage_volumes:
  # S3 兼容存储卷的示例。
  - name: my_s3_volume
    type: S3
    location: s3://defaultbucket/test/
    comment: my s3 volume
    properties:
      - key: aws.s3.region
        value: us-west-2
      - key: aws.s3.endpoint
        value: https://s3.us-west-2.amazonaws.com
      - key: aws.s3.access_key
        value: xxxxxxxxxx
      - key: aws.s3.secret_key
        value: yyyyyyyyyy
  # HDFS 存储卷的示例。
  - name: my_hdfs_volume
    type: HDFS
    location: hdfs://127.0.0.1:9000/sr/test/
    comment: my hdfs volume
    properties:
      - key: hadoop.security.authentication
        value: simple
      - key: username
        value: starrocks

cluster_snapshot_path 可指向具体快照目录,也可按原文例子指向 meta 路径;实际选择应与要恢复的材料对应。storage_volume_name 必须在 storage_volumes 中定义,而且存储卷名称必须与原集群一致。不要把 Leader FE 填进 frontends 列表。

节点信息段是可选的。原文提醒,未提供该部分时,新集群只有 Leader FE 的 FE 信息,CN 会保留原集群的信息。因此,演练中不能在省略节点段之后假定所有旧地址都已被替换。

静态审查发现的配置风险:S3 示例把 aws.s3.access_key 和 aws.s3.secret_key 放在 YAML 中。原文仅包含占位符,没有发现真实秘密;但若填入长期密钥,该文件、备份、工单和版本库都会成为泄露面。应限制配置文件访问权限,并根据环境按官方 AWS 认证说明选择合适的认证方式。不要把环境变量写成模板值后就假定 StarRocks 会自动插值;本篇没有核实这种行为。

HDFS 示例使用 hadoop.security.authentication: simple,这是示例配置,不提供强身份认证保障;不能把其中的 username: starrocks 当作安全认证机制。对于需要认证的 HDFS 环境,应使用经该环境和版本验证的认证配置,而不是直接复制这里的 simple 模板。

版本与编辑说明

本次已完整读取原始 3.5 网页,并与 StarRocks 官方仓库 branch-3.5 的中文文档源文件交叉核对。当前默认文档已指向较新版本,快照类型和部分恢复说明发生变化;本文没有把新版本中的 manual 类型写进 3.5 的接口表。

与原文相比,本稿重排了停用操作的位置,增加了独立恢复环境、目录保全、凭证与 HDFS 认证风险、节点验收及“未实测”的说明;技术语法与完整 YAML 示例保持可追溯。本文未执行 SQL、启动脚本、目录清理或数据复制,也未证明任何具体集群可以恢复。静态检查没有发现问题的部分,同样不等于不存在漏洞。

来源:StarRocks 3.5 集群快照文档。Copyright 2021-present StarRocks, Inc. All rights reserved. 官方仓库的 Apache License 2.0 文本与仓库 NOTICE 全文保留于本文末尾。本文为获授权的中文整理与安全批注版,原始内容归 StarRocks 及相关贡献者,原创示意图由未完纪绘制。

版权与许可全文

以下保留本页涉及的来源材料或示例代码的版权、许可条件与免责声明;各自适用范围依原声明。中文翻译及编辑标注:未完纪,2026-10-05。

LICENSE-StarRocks.txt

Copyright 2021-present StarRocks, Inc. All rights reserved.

                                 Apache License
                           Version 2.0, January 2004
                        http://www.apache.org/licenses/

   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION

   1. Definitions.

      "License" shall mean the terms and conditions for use, reproduction,
      and distribution as defined by Sections 1 through 9 of this document.

      "Licensor" shall mean the copyright owner or entity authorized by
      the copyright owner that is granting the License.

      "Legal Entity" shall mean the union of the acting entity and all
      other entities that control, are controlled by, or are under common
      control with that entity. For the purposes of this definition,
      "control" means (i) the power, direct or indirect, to cause the
      direction or management of such entity, whether by contract or
      otherwise, or (ii) ownership of fifty percent (50%) or more of the
      outstanding shares, or (iii) beneficial ownership of such entity.

      "You" (or "Your") shall mean an individual or Legal Entity
      exercising permissions granted by this License.

      "Source" form shall mean the preferred form for making modifications,
      including but not limited to software source code, documentation
      source, and configuration files.

      "Object" form shall mean any form resulting from mechanical
      transformation or translation of a Source form, including but
      not limited to compiled object code, generated documentation,
      and conversions to other media types.

      "Work" shall mean the work of authorship, whether in Source or
      Object form, made available under the License, as indicated by a
      copyright notice that is included in or attached to the work
      (an example is provided in the Appendix below).

      "Derivative Works" shall mean any work, whether in Source or Object
      form, that is based on (or derived from) the Work and for which the
      editorial revisions, annotations, elaborations, or other modifications
      represent, as a whole, an original work of authorship. For the purposes
      of this License, Derivative Works shall not include works that remain
      separable from, or merely link (or bind by name) to the interfaces of,
      the Work and Derivative Works thereof.

      "Contribution" shall mean any work of authorship, including
      the original version of the Work and any modifications or additions
      to that Work or Derivative Works thereof, that is intentionally
      submitted to Licensor for inclusion in the Work by the copyright owner
      or by an individual or Legal Entity authorized to submit on behalf of
      the copyright owner. For the purposes of this definition, "submitted"
      means any form of electronic, verbal, or written communication sent
      to the Licensor or its representatives, including but not limited to
      communication on electronic mailing lists, source code control systems,
      and issue tracking systems that are managed by, or on behalf of, the
      Licensor for the purpose of discussing and improving the Work, but
      excluding communication that is conspicuously marked or otherwise
      designated in writing by the copyright owner as "Not a Contribution."

      "Contributor" shall mean Licensor and any individual or Legal Entity
      on behalf of whom a Contribution has been received by Licensor and
      subsequently incorporated within the Work.

   2. Grant of Copyright License. Subject to the terms and conditions of
      this License, each Contributor hereby grants to You a perpetual,
      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
      copyright license to reproduce, prepare Derivative Works of,
      publicly display, publicly perform, sublicense, and distribute the
      Work and such Derivative Works in Source or Object form.

   3. Grant of Patent License. Subject to the terms and conditions of
      this License, each Contributor hereby grants to You a perpetual,
      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
      (except as stated in this section) patent license to make, have made,
      use, offer to sell, sell, import, and otherwise transfer the Work,
      where such license applies only to those patent claims licensable
      by such Contributor that are necessarily infringed by their
      Contribution(s) alone or by combination of their Contribution(s)
      with the Work to which such Contribution(s) was submitted. If You
      institute patent litigation against any entity (including a
      cross-claim or counterclaim in a lawsuit) alleging that the Work
      or a Contribution incorporated within the Work constitutes direct
      or contributory patent infringement, then any patent licenses
      granted to You under this License for that Work shall terminate
      as of the date such litigation is filed.

   4. Redistribution. You may reproduce and distribute copies of the
      Work or Derivative Works thereof in any medium, with or without
      modifications, and in Source or Object form, provided that You
      meet the following conditions:

      (a) You must give any other recipients of the Work or
          Derivative Works a copy of this License; and

      (b) You must cause any modified files to carry prominent notices
          stating that You changed the files; and

      (c) You must retain, in the Source form of any Derivative Works
          that You distribute, all copyright, patent, trademark, and
          attribution notices from the Source form of the Work,
          excluding those notices that do not pertain to any part of
          the Derivative Works; and

      (d) If the Work includes a "NOTICE" text file as part of its
          distribution, then any Derivative Works that You distribute must
          include a readable copy of the attribution notices contained
          within such NOTICE file, excluding those notices that do not
          pertain to any part of the Derivative Works, in at least one
          of the following places: within a NOTICE text file distributed
          as part of the Derivative Works; within the Source form or
          documentation, if provided along with the Derivative Works; or,
          within a display generated by the Derivative Works, if and
          wherever such third-party notices normally appear. The contents
          of the NOTICE file are for informational purposes only and
          do not modify the License. You may add Your own attribution
          notices within Derivative Works that You distribute, alongside
          or as an addendum to the NOTICE text from the Work, provided
          that such additional attribution notices cannot be construed
          as modifying the License.

      You may add Your own copyright statement to Your modifications and
      may provide additional or different license terms and conditions
      for use, reproduction, or distribution of Your modifications, or
      for any such Derivative Works as a whole, provided Your use,
      reproduction, and distribution of the Work otherwise complies with
      the conditions stated in this License.

   5. Submission of Contributions. Unless You explicitly state otherwise,
      any Contribution intentionally submitted for inclusion in the Work
      by You to the Licensor shall be under the terms and conditions of
      this License, without any additional terms or conditions.
      Notwithstanding the above, nothing herein shall supersede or modify
      the terms of any separate license agreement you may have executed
      with Licensor regarding such Contributions.

   6. Trademarks. This License does not grant permission to use the trade
      names, trademarks, service marks, or product names of the Licensor,
      except as required for reasonable and customary use in describing the
      origin of the Work and reproducing the content of the NOTICE file.

   7. Disclaimer of Warranty. Unless required by applicable law or
      agreed to in writing, Licensor provides the Work (and each
      Contributor provides its Contributions) on an "AS IS" BASIS,
      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
      implied, including, without limitation, any warranties or conditions
      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
      PARTICULAR PURPOSE. You are solely responsible for determining the
      appropriateness of using or redistributing the Work and assume any
      risks associated with Your exercise of permissions under this License.

   8. Limitation of Liability. In no event and under no legal theory,
      whether in tort (including negligence), contract, or otherwise,
      unless required by applicable law (such as deliberate and grossly
      negligent acts) or agreed to in writing, shall any Contributor be
      liable to You for damages, including any direct, indirect, special,
      incidental, or consequential damages of any character arising as a
      result of this License or out of the use or inability to use the
      Work (including but not limited to damages for loss of goodwill,
      work stoppage, computer failure or malfunction, or any and all
      other commercial damages or losses), even if such Contributor
      has been advised of the possibility of such damages.

   9. Accepting Warranty or Additional Liability. While redistributing
      the Work or Derivative Works thereof, You may choose to offer,
      and charge a fee for, acceptance of support, warranty, indemnity,
      or other liability obligations and/or rights consistent with this
      License. However, in accepting such obligations, You may act only
      on Your own behalf and on Your sole responsibility, not on behalf
      of any other Contributor, and only if You agree to indemnify,
      defend, and hold each Contributor harmless for any liability
      incurred by, or claims asserted against, such Contributor by reason
      of your accepting any such warranty or additional liability.

   END OF TERMS AND CONDITIONS

NOTICE-StarRocks.txt

StarRocks

Copyright 2021-present, StarRocks Inc.

---------------------------
apache-doris-incubating NOTICE
---------------------------
Apache Doris (incubating)
Copyright 2018-2021 The Apache Software Foundation

This product includes software developed at
The Apache Software Foundation (http://www.apache.org/).

Based on source code originally developed by
Baidu (http://www.baidu.com/).


---------------------------
apache-impala NOTICE
---------------------------
Apache Impala
Copyright 2019 The Apache Software Foundation

This product includes software developed at
The Apache Software Foundation (http://www.apache.org/).

Portions of this software were developed at
Cloudera, Inc (http://www.cloudera.com/).

This product includes software developed by the OpenSSL
Project for use in the OpenSSL Toolkit (http://www.openssl.org/)

This product includes cryptographic software written by Eric Young
(eay@cryptsoft.com).  This product includes software written by Tim
Hudson (tjh@cryptsoft.com).

This product includes software developed by the University of Chicago,
as Operator of Argonne National Laboratory.
Copyright (C) 1999 University of Chicago. All rights reserved.

---------------------------
apache-kudu NOTICE
---------------------------
Apache Kudu
Copyright 2016 The Apache Software Foundation

This product includes software developed at
The Apache Software Foundation (http://www.apache.org/).

Portions of this software were developed at
Cloudera, Inc (http://www.cloudera.com/).

This product includes software developed by the OpenSSL
Project for use in the OpenSSL Toolkit (http://www.openssl.org/)

This product includes cryptographic software written by Eric Young
(eay@cryptsoft.com).  This product includes software written by Tim
Hudson (tjh@cryptsoft.com).

© 版权声明
THE END
喜欢就支持一下吧
点赞0 分享
评论 抢沙发

请登录后发表评论

    暂无评论内容