分两步清理 Druid 过期数据

在 Druid 中,让一个分段不再参与查询,与把它从存储中永久删除,是两个不同的动作。首先通过 Coordinator 将分段标为 unused;随后提交 kill任务,才会删除对应元数据与深存储文件。只做第一步,磁盘或对象存储中的文件仍然存在。

本文依据 Apache Druid 官方《Tutorial: Deleting data》完整技术流程译编,并核对随教程的 deletion-index.json与 deletion-kill.json。核验日期为 2026 年 10 月 5 日。源地址中的 latest与代码仓库的 master都会变化,复现时应使用与本机发行版匹配的教程文件。本文只做静态审阅,没有启动 Druid、导入数据、标记分段或提交删除任务。

Druid 分段从 used 到 unused 再到永久删除:markUnused 保留深存储文件,kill 同时移除元数据与深存储,不能用 markUsed 撤回
未完纪绘制的状态示意图,不是 Druid 控制台截图;译编与示意图采用 CC BY-SA 4.0。

准备一个独立的教程数据源

原教程要求一个运行中的 Druid 实例和命令行 JSON 工具 jq。下面保留 Linux/Bash 命令形式,目标地址来自本机 quickstart:Coordinator/Overlord 使用 localhost:8081,Web 控制台通常为 http://localhost:8888。已有集群的服务位置、认证与 TLS 配置不一定相同,不应直接替换成生产地址运行。

初始数据是 2015 年 9 月 12 日的 Wikipedia 编辑样例。发行包中的 quickstart/tutorial/deletion-index.json创建名为 deletion-tutorial的数据源。已核对的配置采用 index_parallel,时间字段为 time,格式为 iso;关键粒度部分如下:

{
  "type": "uniform",
  "segmentGranularity": "hour",
  "queryGranularity": "none",
  "intervals": ["2015-09-12/2015-09-13"],
  "rollup": false
}

因此示例按小时生成分段,不做 rollup。输入来自本地目录 quickstart/tutorial/下的 wikiticker-2015-09-12-sampled.json.gz,输入格式是 JSON;配置的 appendToExisting为假,不能把它当作随意向已有同名数据源追加的命令。教程应使用没有业务数据的独立环境。

下列请求是原教程的导入步骤,实际执行会向 Druid 提交写入任务。本文未执行:

curl -X POST http://localhost:8081/druid/indexer/v1/task \
  -H "Content-Type: application/json" \
  -d @quickstart/tutorial/deletion-index.json

确认导入任务完成后,再在控制台查看数据源与分段。原文指向 unified-console.md#datasources的本地链接是文档路径瑕疵;本稿改为说明从本机控制台入口进入相应视图,不把这个旧路径继续当成可用页面。

第一步:按时间范围标为不用

原教程先处理 UTC 时间 18:00 到 20:00 的分段,范围是 2015-09-12T18:00:00.000Z/2015-09-12T20:00:00.000Z,对应 18 点和 19 点两个小时。发送的是 markUnused请求:

# 会改变分段可用状态;只用于已核对的隔离教程数据
curl -X POST \
  -H "Content-Type: application/json" \
  -d '{"interval":"2015-09-12T18:00:00.000Z/2015-09-12T20:00:00.000Z"}' \
  http://localhost:8081/druid/coordinator/v1/datasources/deletion-tutorial/markUnused

原教程说明,这两个小时的分段随后不再显示于其控制台的已使用分段视图。但此时只是状态发生变化。原文在本地深存储目录 var/druid/segments/deletion-tutorial/的列表中,仍能看到:

2015-09-12T18:00:00.000Z_2015-09-12T19:00:00.000Z
2015-09-12T19:00:00.000Z_2015-09-12T20:00:00.000Z

这些是原文给出的目录示例,不是本次机器上的检查结果。对象存储部署也不会拥有完全相同的本地目录;应到实际深存储后端核对相应对象。

也可以用精确分段 ID 选择目标

第二种方法是提交 segmentIds。在控制台的 Segments 视图打开分段元数据,identifier字段就是完整 ID。原文 23 点分段的示例为:

deletion-tutorial_2015-09-12T23:00:00.000Z_2015-09-13T00:00:00.000Z_2023-05-16T00:04:12.091Z

ID 包含版本等信息,重新导入后可能不同,不能把文档中的这个旧 ID 当成自己的分段 ID。原教程通过 API 拉取 ID 数组,再用 jq '.[-2:]'取最后两项,意图选中 22 点与 23 点:

unusedSegmentIds=$(curl -X GET \
  -H 'Content-Type:application/json' \
  http://localhost:8081/druid/coordinator/v1/datasources/deletion-tutorial/segments \
  | jq '.[-2:]')

静态审阅发现:“数组最后两项”并不表达“最后两个小时”的业务条件。真实集群可能有多个分区、多个版本或不同返回顺序,HTTP 请求失败也可能产生非预期内容。不要仅凭数组位置自动执行删除相关操作。

以下编辑修订把查询、筛选与写入分开。它只针对原教程每小时一个目标分段的前提,按明确时间前缀筛选,并要求两个小时分别恰好匹配一个 ID;不符合前提时让 jq失败。这个片段只读取 API 并生成本地待审阅 JSON,没有发出状态修改请求,也未在本次执行。

curl --fail-with-body --silent --show-error \
  http://localhost:8081/druid/coordinator/v1/datasources/deletion-tutorial/segments \
  -o segments.json

jq -e '
  if type != "array" or any(.[]; type != "string") then
    error("Expected an array of segment IDs")
  else
    [.[] | select(startswith(
      "deletion-tutorial_2015-09-12T22:00:00.000Z_2015-09-12T23:00:00.000Z_"
    ))] as $hour22
    | [.[] | select(startswith(
      "deletion-tutorial_2015-09-12T23:00:00.000Z_2015-09-13T00:00:00.000Z_"
    ))] as $hour23
    | if ($hour22 | length) == 1 and ($hour23 | length) == 1 then
        {segmentIds: ($hour22 + $hour23)}
      else
        error("Expected exactly one segment for each tutorial hour; inspect manually")
      end
  end
' segments.json > mark-unused-ids.json

与原文的差别:增加 HTTP 失败处理、JSON 类型检查、按时间条件筛选和每小时数量断言,并把结果保存在文件里供核对。它不能代替人工确认数据源、完整 ID 与备份,也不适用于不满足单分段前提的生产数据。--fail-with-body要求支持该参数的 curl 版本;旧版可采用 --fail并另行保留错误响应。若第一条命令失败,应停止,不得继续使用旧的 segments.json。

核对生成文件中的两个完整 ID 后,才是实际改变状态的请求:

# 此请求会把文件列出的分段标为 unused
curl --fail-with-body -X POST \
  -H "Content-Type: application/json" \
  --data-binary @mark-unused-ids.json \
  http://localhost:8081/druid/coordinator/v1/datasources/deletion-tutorial/markUnused

原教程的预期结果是 22 点与 23 点分段也从已使用分段视图中移出,但它们的深存储目录仍在。到这里一共选择了四个小时,尚未永久删除文件。

第二步:用 kill 任务永久移除

永久删除的危险边界在这里。kill 任务会删除落在任务范围中的 unused 分段的元数据及深存储文件,不能靠把状态改回 used 来撤销。执行前必须核对备份、精确目标和将受影响的所有 unused 分段。本次没有提交该任务。

原教程的 quickstart/tutorial/deletion-kill.json内容是:

{
  "type": "kill",
  "dataSource": "deletion-tutorial",
  "interval": "2015-09-12/2015-09-13"
}

这里覆盖一整天,并没有列出上面那四个分段 ID。因此,如果当天还有其他已标为 unused 的分段,它们也可能落入任务范围。不要把这份 JSON 误读为“只删刚才四个 ID”。

下面是原教程提交 kill 任务的命令,只作为原流程记录。它是不可逆删除操作,不是查询命令:

# 危险:永久删除指定日期内符合条件的 unused 分段
curl -X POST \
  -H "Content-Type: application/json" \
  -d @quickstart/tutorial/deletion-kill.json \
  http://localhost:8081/druid/indexer/v1/task

编辑建议:若真实意图只覆盖教程选定的两段时间,应先检查安装版本的 kill 任务规范,再分别采用 18:00/20:00和 22:00/次日00:00这样的更窄范围,核对范围内所有 unused 分段。窄化时间也不是按 ID 锁定:同一时窗中其他符合条件的版本或分区仍需审阅。本文不把这一建议包装成已经验证的替代任务。

用两层证据确认结果

按原教程,kill 任务完成后,本地深存储列表不再包含 18、19、22、23 点四个小时目录,保留下来的时段是 00 点至 17 点,以及 20、21 点。原文展示的过程可整理为:

阶段 四个目标小时的使用状态 深存储文件
初始导入完成 used 存在
markUnused 完成 unused 仍存在
kill 任务成功完成 对应元数据被删除 对应文件被删除

这些是官方教程陈述的预期现象,并非本次实测。实际核验应首先确认任务状态与服务端错误,再核对分段元数据及深存储对象。只看控制台中的活动分段减少,无法证明物理文件已经清除;只看一次请求返回,也不能证明异步任务已成功结束。

本地示例用只读目录列表 ls -l1 var/druid/segments/deletion-tutorial/观察文件变化。对于其他深存储后端,应使用对应的只读查询方式,并区分分段清理与对象存储自己的版本、回收或保留策略。本文没有核查任何实际后端。

静态审查范围

已审阅源文请求、导入配置和 kill 配置。没有发现这些片段中的真实硬编码凭据;原文 HTTP 地址、无认证请求仅适用于其本机教程环境,不能据此推断生产服务可免认证或明文管理。也没有执行任何破坏性示例。

本文明确修正了旧控制台路径说明,指出数组末尾取两项的不可靠选择方式、导入配置的非追加语义,以及 kill 的整日范围。编辑修订仍只经过静态阅读,没有经过 Bash、jq 或 Druid 运行验证;未发现更多问题不等于不存在漏洞。


原文:Tutorial: Deleting data。维护及发布方:Apache Software Foundation,原页无可确认的个人作者。Copyright © 2026 Apache Software Foundation。除另有说明外,原文采用 CC BY-SA 4.0;本中文译编、编辑补充与自绘图同样以 CC BY-SA 4.0 提供,修改日期为 2026-10-05。软件及仓库示例另见 Druid 仓库许可证。Apache、Druid 及相关标识的商标权归 Apache Software Foundation;本文不表示其对译编内容背书。

原教程完整导入配置

以下为原仓库 deletion-index.json;与前文小时粒度摘录对应,供完整核对。它会写入教程数据源,不是只读查询。修改与执行须在可丢弃环境审查。

{
  "type" : "index_parallel",
  "spec" : {
    "dataSchema" : {
      "dataSource" : "deletion-tutorial",
      "timestampSpec": {
        "column": "time",
        "format": "iso"
      },
      "dimensionsSpec" : {
        "dimensions" : [
          "channel",
          "cityName",
          "comment",
          "countryIsoCode",
          "countryName",
          "isAnonymous",
          "isMinor",
          "isNew",
          "isRobot",
          "isUnpatrolled",
          "metroCode",
          "namespace",
          "page",
          "regionIsoCode",
          "regionName",
          "user",
          { "name": "added", "type": "long" },
          { "name": "deleted", "type": "long" },
          { "name": "delta", "type": "long" }
        ]
      },
      "metricsSpec" : [],
      "granularitySpec" : {
        "type" : "uniform",
        "segmentGranularity" : "hour",
        "queryGranularity" : "none",
        "intervals" : ["2015-09-12/2015-09-13"],
        "rollup" : false
      }
    },
    "ioConfig" : {
      "type" : "index_parallel",
      "inputSource" : {
        "type" : "local",
        "baseDir" : "quickstart/tutorial/",
        "filter" : "wikiticker-2015-09-12-sampled.json.gz"
      },
      "inputFormat" : {
        "type" : "json"
      },
      "appendToExisting" : false
    },
    "tuningConfig" : {
      "type" : "index_parallel",
      "maxRowsPerSegment" : 5000000,
      "maxRowsInMemory" : 25000
    }
  }
}

仓库配置版权与许可

以下许可仅说明原仓库配置代码的许可;正文与配图保留前述 CC BY-SA 4.0 声明。引用配置的缩进及相关讲解由未完纪于2026-10-05整理;源配置字段未改变。

Apache Druid
Copyright 2018-2026 The Apache Software Foundation

This product includes software developed at
The Apache Software Foundation (http://www.apache.org/).

Apache License
                           Version 2.0, January 2004
                        http://www.apache.org/licenses/

   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION

   1. Definitions.

      "License" shall mean the terms and conditions for use, reproduction,
      and distribution as defined by Sections 1 through 9 of this document.

      "Licensor" shall mean the copyright owner or entity authorized by
      the copyright owner that is granting the License.

      "Legal Entity" shall mean the union of the acting entity and all
      other entities that control, are controlled by, or are under common
      control with that entity. For the purposes of this definition,
      "control" means (i) the power, direct or indirect, to cause the
      direction or management of such entity, whether by contract or
      otherwise, or (ii) ownership of fifty percent (50%) or more of the
      outstanding shares, or (iii) beneficial ownership of such entity.

      "You" (or "Your") shall mean an individual or Legal Entity
      exercising permissions granted by this License.

      "Source" form shall mean the preferred form for making modifications,
      including but not limited to software source code, documentation
      source, and configuration files.

      "Object" form shall mean any form resulting from mechanical
      transformation or translation of a Source form, including but
      not limited to compiled object code, generated documentation,
      and conversions to other media types.

      "Work" shall mean the work of authorship, whether in Source or
      Object form, made available under the License, as indicated by a
      copyright notice that is included in or attached to the work
      (an example is provided in the Appendix below).

      "Derivative Works" shall mean any work, whether in Source or Object
      form, that is based on (or derived from) the Work and for which the
      editorial revisions, annotations, elaborations, or other modifications
      represent, as a whole, an original work of authorship. For the purposes
      of this License, Derivative Works shall not include works that remain
      separable from, or merely link (or bind by name) to the interfaces of,
      the Work and Derivative Works thereof.

      "Contribution" shall mean any work of authorship, including
      the original version of the Work and any modifications or additions
      to that Work or Derivative Works thereof, that is intentionally
      submitted to Licensor for inclusion in the Work by the copyright owner
      or by an individual or Legal Entity authorized to submit on behalf of
      the copyright owner. For the purposes of this definition, "submitted"
      means any form of electronic, verbal, or written communication sent
      to the Licensor or its representatives, including but not limited to
      communication on electronic mailing lists, source code control systems,
      and issue tracking systems that are managed by, or on behalf of, the
      Licensor for the purpose of discussing and improving the Work, but
      excluding communication that is conspicuously marked or otherwise
      designated in writing by the copyright owner as "Not a Contribution."

      "Contributor" shall mean Licensor and any individual or Legal Entity
      on behalf of whom a Contribution has been received by Licensor and
      subsequently incorporated within the Work.

   2. Grant of Copyright License. Subject to the terms and conditions of
      this License, each Contributor hereby grants to You a perpetual,
      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
      copyright license to reproduce, prepare Derivative Works of,
      publicly display, publicly perform, sublicense, and distribute the
      Work and such Derivative Works in Source or Object form.

   3. Grant of Patent License. Subject to the terms and conditions of
      this License, each Contributor hereby grants to You a perpetual,
      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
      (except as stated in this section) patent license to make, have made,
      use, offer to sell, sell, import, and otherwise transfer the Work,
      where such license applies only to those patent claims licensable
      by such Contributor that are necessarily infringed by their
      Contribution(s) alone or by combination of their Contribution(s)
      with the Work to which such Contribution(s) was submitted. If You
      institute patent litigation against any entity (including a
      cross-claim or counterclaim in a lawsuit) alleging that the Work
      or a Contribution incorporated within the Work constitutes direct
      or contributory patent infringement, then any patent licenses
      granted to You under this License for that Work shall terminate
      as of the date such litigation is filed.

   4. Redistribution. You may reproduce and distribute copies of the
      Work or Derivative Works thereof in any medium, with or without
      modifications, and in Source or Object form, provided that You
      meet the following conditions:

      (a) You must give any other recipients of the Work or
          Derivative Works a copy of this License; and

      (b) You must cause any modified files to carry prominent notices
          stating that You changed the files; and

      (c) You must retain, in the Source form of any Derivative Works
          that You distribute, all copyright, patent, trademark, and
          attribution notices from the Source form of the Work,
          excluding those notices that do not pertain to any part of
          the Derivative Works; and

      (d) If the Work includes a "NOTICE" text file as part of its
          distribution, then any Derivative Works that You distribute must
          include a readable copy of the attribution notices contained
          within such NOTICE file, excluding those notices that do not
          pertain to any part of the Derivative Works, in at least one
          of the following places: within a NOTICE text file distributed
          as part of the Derivative Works; within the Source form or
          documentation, if provided along with the Derivative Works; or,
          within a display generated by the Derivative Works, if and
          wherever such third-party notices normally appear. The contents
          of the NOTICE file are for informational purposes only and
          do not modify the License. You may add Your own attribution
          notices within Derivative Works that You distribute, alongside
          or as an addendum to the NOTICE text from the Work, provided
          that such additional attribution notices cannot be construed
          as modifying the License.

      You may add Your own copyright statement to Your modifications and
      may provide additional or different license terms and conditions
      for use, reproduction, or distribution of Your modifications, or
      for any such Derivative Works as a whole, provided Your use,
      reproduction, and distribution of the Work otherwise complies with
      the conditions stated in this License.

   5. Submission of Contributions. Unless You explicitly state otherwise,
      any Contribution intentionally submitted for inclusion in the Work
      by You to the Licensor shall be under the terms and conditions of
      this License, without any additional terms or conditions.
      Notwithstanding the above, nothing herein shall supersede or modify
      the terms of any separate license agreement you may have executed
      with Licensor regarding such Contributions.

   6. Trademarks. This License does not grant permission to use the trade
      names, trademarks, service marks, or product names of the Licensor,
      except as required for reasonable and customary use in describing the
      origin of the Work and reproducing the content of the NOTICE file.

   7. Disclaimer of Warranty. Unless required by applicable law or
      agreed to in writing, Licensor provides the Work (and each
      Contributor provides its Contributions) on an "AS IS" BASIS,
      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
      implied, including, without limitation, any warranties or conditions
      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
      PARTICULAR PURPOSE. You are solely responsible for determining the
      appropriateness of using or redistributing the Work and assume any
      risks associated with Your exercise of permissions under this License.

   8. Limitation of Liability. In no event and under no legal theory,
      whether in tort (including negligence), contract, or otherwise,
      unless required by applicable law (such as deliberate and grossly
      negligent acts) or agreed to in writing, shall any Contributor be
      liable to You for damages, including any direct, indirect, special,
      incidental, or consequential damages of any character arising as a
      result of this License or out of the use or inability to use the
      Work (including but not limited to damages for loss of goodwill,
      work stoppage, computer failure or malfunction, or any and all
      other commercial damages or losses), even if such Contributor
      has been advised of the possibility of such damages.

   9. Accepting Warranty or Additional Liability. While redistributing
      the Work or Derivative Works thereof, You may choose to offer,
      and charge a fee for, acceptance of support, warranty, indemnity,
      or other liability obligations and/or rights consistent with this
      License. However, in accepting such obligations, You may act only
      on Your own behalf and on Your sole responsibility, not on behalf
      of any other Contributor, and only if You agree to indemnify,
      defend, and hold each Contributor harmless for any liability
      incurred by, or claims asserted against, such Contributor by reason
      of your accepting any such warranty or additional liability.

   END OF TERMS AND CONDITIONS

   APPENDIX: How to apply the Apache License to your work.

      To apply the Apache License to your work, attach the following
      boilerplate notice, with the fields enclosed by brackets "[]"
      replaced with your own identifying information. (Don't include
      the brackets!)  The text should be enclosed in the appropriate
      comment syntax for the file format. We also recommend that a
      file or class name and description of purpose be included on the
      same "printed page" as the copyright notice for easier
      identification within third-party archives.

   Copyright [yyyy] [name of copyright owner]

   Licensed under the Apache License, Version 2.0 (the "License");
   you may not use this file except in compliance with the License.
   You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

   Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.

原文图示

按时间区间标记unused后的Segments视图
按时间区间标记unused后的Segments视图。原图:Apache Software Foundation,CC BY-SA 4.0。原图来源;原文历史结果,未重新执行。
原文查看分段identifier的元数据界面
原文查看分段identifier的元数据界面。原图:Apache Software Foundation,CC BY-SA 4.0。原图来源;原文历史结果,未重新执行。
按ID标记最后两个小时unused后的Segments视图
按ID标记最后两个小时unused后的Segments视图。原图:Apache Software Foundation,CC BY-SA 4.0。原图来源;原文历史结果,未重新执行。
© 版权声明
THE END
喜欢就支持一下吧
点赞0 分享
评论 抢沙发

请登录后发表评论

    暂无评论内容