一个模型的效果，既取决于算法，也取决于特征表示、正则化强度和数据划分。只在训练数据上试几个参数，再挑训练分数最高的一组，很容易把数据中的偶然规律当成可推广的能力。Spark MLlib 提供 CrossValidator 与 TrainValidationSplit，让候选参数在留出的数据上接受同一套评估，再用胜出的参数重新训练模型。
来源与版本：本文依据 Apache Spark 官方 ML Tuning: model selection and hyperparameter tuning 完整正文、随文两份 Python 示例及样例 LIBSVM 文件译解整理。2026-10-05 读取的 latest 页面标示 Spark 4.2.0；它是会变化的文档地址，部署时应切换到实际运行版本。原页没有可确认的个人作者，维护方为 Apache Software Foundation / Spark contributors。本文未启动 Spark、未训练模型，也没有生成准确率、RMSE 或性能结果。
模型选择需要三个对象
这里的“模型选择”也称超参数调优。被调对象可以是单个 Estimator，例如逻辑回归，也可以是包括特征处理与分类器的整条 Pipeline。把 Pipeline 整体交给调参器，可以同时比较特征维数和算法参数，而不必先孤立地决定每个阶段。
| 对象 | 作用 | 本文例子 |
|---|---|---|
| Estimator | 给定数据和参数，拟合得到模型 | 文本处理 Pipeline；线性回归 |
| ParamMap 集合 | 列出待比较的参数组合 | HashingTF 维数 × 正则化强度 |
| Evaluator | 对留出数据上的模型预测计算指标 | 二分类 ROC 曲线下面积；回归误差 |
共同流程是：划分输入数据；对每组候选参数拟合模型；用验证部分的预测计算指标；选择最佳参数；最后用这组参数在输入给调参器的全部数据上重新拟合。这里的“全部数据”有明确范围：如果传入的是外层训练集，就不应把独立测试集混进去。
原文在介绍内部划分时也使用 test 一词。为避免与最终评估混淆，本文把用于选参数的内部留出部分称为验证集，把不参与选择、只在最后使用的部分称为独立测试集。
Evaluator 应匹配任务类型。官方列出回归、二分类、多分类、多标签和排序评估器；可以通过 setMetricName 选择具体指标。指标决定“最好”的含义：ROC AUC 的比较方向与 RMSE 不同，业务真正关心的误报、漏报或排序质量也未必能由一个默认分数完整表达。
交叉验证怎样使用每一折
若设三折，数据被分成 A、B、C 三部分。第一轮以 A 验证、B 和 C 训练；第二轮以 B 验证；第三轮以 C 验证。对一组参数得到三个评估值，取平均作为这组参数的比较依据。不同参数使用相同的划分逻辑，避免比较被不同验证数据的难度混淆。
找到最佳参数后，CrossValidator 还会在交给它的完整数据上再拟合一次。最终返回的模型不是随便挑出某一折训练出来的模型。这有助于在参数已经选定后利用更多训练信息。
原示例用三种 HashingTF 维数、两种正则化强度、两折交叉验证：候选组合数是 3 × 2 = 6，内部需要 6 × 2 = 12 次拟合；另有最佳参数的最终重拟合。增大参数网格或折数，成本会相乘。两折只是演示设置，原文提到三折、十折等常见选择；实际应根据数据量、验证稳定性和算力预算决定。
文本示例：一起选择特征维数和正则化
下面以官方 Python 示例为主线。Scala 与 Java 版本实现相同流程，原页还给它们设置了 parallelism=2。为避免重复三份相同算法，本文保留 Python 的完整核心步骤；三种语言的原示例仍可在来源页对照。
完整源码会创建 SparkSession 并在结束时调用 spark.stop()。如果不是在已有 spark 的交互式环境中使用，需要先准备会话：
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("CrossValidatorExample").getOrCreate()getOrCreate() 会使用当前环境的 Spark 配置；它不代表这里已经配置了某个集群。先确认运行目标和资源限制，再启动自己的实验，避免无意把演示提交到共享生产资源。
from pyspark.ml import Pipeline
from pyspark.ml.classification import LogisticRegression
from pyspark.ml.evaluation import BinaryClassificationEvaluator
from pyspark.ml.feature import HashingTF, Tokenizer
from pyspark.ml.tuning import CrossValidator, ParamGridBuilder
training = spark.createDataFrame([
(0, "a b c d e spark", 1.0),
(1, "b d", 0.0),
(2, "spark f g h", 1.0),
(3, "hadoop mapreduce", 0.0),
(4, "b spark who", 1.0),
(5, "g d a y", 0.0),
(6, "spark fly", 1.0),
(7, "was mapreduce", 0.0),
(8, "e spark program", 1.0),
(9, "a e c l", 0.0),
(10, "spark compile", 1.0),
(11, "hadoop software", 0.0)
], ["id", "text", "label"])
tokenizer = Tokenizer(inputCol="text", outputCol="words")
hashingTF = HashingTF(
inputCol=tokenizer.getOutputCol(), outputCol="features"
)
lr = LogisticRegression(maxIter=10)
pipeline = Pipeline(stages=[tokenizer, hashingTF, lr])
paramGrid = (
ParamGridBuilder()
.addGrid(hashingTF.numFeatures, [10, 100, 1000])
.addGrid(lr.regParam, [0.1, 0.01])
.build()
)
crossval = CrossValidator(
estimator=pipeline,
estimatorParamMaps=paramGrid,
evaluator=BinaryClassificationEvaluator(),
numFolds=2,
)
cvModel = crossval.fit(training)Tokenizer 把文本变成词序列，HashingTF 把它变成固定维度的特征向量，逻辑回归做分类。参数引用绑定到这里创建的 hashingTF 和 lr 对象；构造网格后如果又换了一组 Pipeline 阶段，不能假设旧参数引用会自动对应新的实例。
二分类评估器在原例中的默认指标是 areaUnderROC。输入只有十二条刻意简化的教学文本，maxIter=10 也只是示例配置：这套数据不能证明真实文本分类效果，十次迭代同样不构成收敛保证。查看收敛、类别分布和业务指标，是把示例用于真实任务前另外需要做的工作。
官方随后准备四条无标签文本并展示预测：
test = spark.createDataFrame([
(4, "spark i j k"),
(5, "l m n"),
(6, "mapreduce spark"),
(7, "apache hadoop")
], ["id", "text"])
prediction = cvModel.transform(test)
selected = prediction.select("id", "text", "probability", "prediction")
for row in selected.collect():
print(row)这四条数据没有 label，所以代码只展示预测类别和概率，没有计算测试准确率或测试 AUC。原示例复用了部分数字 ID，但模型使用的是文本特征；在真实工程中仍应让记录身份与数据划分可追踪，检查同一用户、实体或重复文本是否跨越训练与测试边界。
collect() 会把所有结果拉到 Driver。对四条演示记录没有大规模风险，但把相同写法搬到百万条结果会造成驱动端内存压力。真实作业应限制预览条数，或者写入合适的分布式输出。
一次训练—验证划分：更低成本的选择
TrainValidationSplit 不轮流验证每一折，而是只拆分一次，每组参数拟合和评估一次。相比 k 折交叉验证，它的计算成本较低；但当数据量不足、类别稀少或一次划分偶然性较强时，参数排名更容易不稳定。
trainRatio=0.75 表示把输入给调参器的数据约 75% 用于训练，约 25% 用于验证。它也会在选定参数后，用输入给调参器的全部数据重新拟合。
官方回归示例先从完整数据中做一次 90% / 10% 的外层划分，留出最后的测试集，再对训练部分做内部 80% / 20% 的验证划分：
from pyspark.ml.evaluation import RegressionEvaluator
from pyspark.ml.regression import LinearRegression
from pyspark.ml.tuning import ParamGridBuilder, TrainValidationSplit
data = spark.read.format("libsvm").load(
"data/mllib/sample_linear_regression_data.txt"
)
train, test = data.randomSplit([0.9, 0.1], seed=12345)
lr = LinearRegression(maxIter=10)
paramGrid = (
ParamGridBuilder()
.addGrid(lr.regParam, [0.1, 0.01])
.addGrid(lr.fitIntercept, [False, True])
.addGrid(lr.elasticNetParam, [0.0, 0.5, 1.0])
.build()
)
tvs = TrainValidationSplit(
estimator=lr,
estimatorParamMaps=paramGrid,
evaluator=RegressionEvaluator(),
trainRatio=0.8,
)
model = tvs.fit(train)
model.transform(test).select("features", "label", "prediction").show()样例路径相对于 Spark 源码或发行目录，不是任意目录都存在的系统路径。本次读取了随文 train_validation_split.py、cross_validator.py 和样例文件；所取 LIBSVM 文件有 501 行，没有空行。这个静态文件核对只确认材料齐全，不代表 Spark 已成功加载数据。
参数网格有 2 × 2 × 3 = 12 组：两种正则强度、是否拟合截距，以及三种 Elastic Net 混合比例。每组只做一次内部训练与验证，然后最佳参数在整个外层 train 上重拟合。若比例恰好实现，内部训练、内部验证和最终测试约占总体的 72%、18%、10%；随机划分并不保证有限数据集里记录数精确等于这些比例。
原例最终仅调用 show() 展示预测。若想得到最终回归指标，应在参数选择完成后，对此前保留的测试集调用评估器。下面是编辑新增的显式 RMSE 计算，不含任何预设结果，也未运行：
test_predictions = model.transform(test)
test_rmse = RegressionEvaluator(
labelCol="label", predictionCol="prediction", metricName="rmse"
).evaluate(test_predictions)
print(f"Test RMSE: {test_rmse}")看到测试误差之后再不断调整网格，实际上又把测试集用于选参数。若反复迭代开发，应为最终判断另保留未被开发过程利用的数据，或者采用符合业务目标的嵌套评估方案。
控制随机性、泄漏与算力
原回归例为外层 randomSplit 指定了种子，但没有显式设置内部 TrainValidationSplit 的种子；文本例也没有显式设置交叉验证种子。当前 CrossValidator API 与 TrainValidationSplit API 都支持 seed。下面是编辑建议的配置变化，不是原 Python 示例已有内容：
crossval.setSeed(12345).setParallelism(2)
tvs.setSeed(12345).setParallelism(2)这些设置应放在各自的 fit() 之前。种子有助于在相同输入和执行条件下复核实验，但并不保证不同数据顺序、分区、依赖版本和运行环境永远得到位级一致的结果；也不会自动解决类别比例失衡或时间泄漏。
真实数据的划分应服从业务关系。同一用户的多条记录可能需要保持在同一侧；时间预测应避免用未来信息训练再预测过去；类别稀少时需要检查每个验证部分是否仍能定义所用指标。当前 API 也提供 foldCol 指定折号：该列应为整数、落在 [0, numFolds)，设置后不再做随机 k 折划分。对时间顺序任务，简单分配 foldCol 仍不自动构成滚动时间验证，训练窗口必须另行设计。
任何需要“从数据学出参数”的预处理，如标准化、词汇表、特征选择或目标编码，都应留在相应训练折内拟合。提前在全部数据上拟合这些步骤，再做交叉验证，会把验证数据的信息泄漏进模型。把流程封装进 Pipeline 的价值，就在于让拟合过程和数据边界一起被管理。
parallelism=1 默认串行评估参数组合，设为 2 或更多可以并发。这里控制的是调参层并发，单个 Spark 模型训练本身还可能消耗多个执行器与任务槽。原文提出“多数集群通常不必超过十”的经验性建议，不能视作统一安全上限；内存、Shuffle、缓存和其他作业的占用决定实际可承受值。也不宜无必要地保存全部子模型，当前 API 提醒 collectSubModels=True 可能让大型模型占满 Driver 内存。
如何阅读一次调参结果
评估一轮实验，至少应保留输入数据版本、外层测试划分、内部验证方式、种子、完整参数网格、指标名称、各组合分数、最佳参数和最终测试结果。较高的验证分数只是这个网格与这次划分下的相对结果；它没有证明对未来时间段、其他客户或不同数据分布同样有效。
两折文本例和一次留出的回归例适合解释 API。前者展示如何把 Pipeline 整体当作 Estimator，后者展示如何在外层训练部分内部选参数。做业务决策时，应从数据关系和评价目标出发，再选择验证方案，而不是把示例参数当成标准配置。
代码核对与许可
本次做了全文与伴随文件的静态核验，检查了参数引用、输入路径、随机划分、拟合与预测顺序、数据收集位置，以及代码中的权限与外部调用。示例没有硬编码秘密、字符串拼接 SQL、网络凭据或删除文件命令；实际需要关注的是错误的集群目标、数据泄漏、没有控制的拟合成本和 collect() 的驱动端内存风险。没有发现某类问题，不代表组件或运行环境没有漏洞。
本文保留官方算法和候选参数；改用括号组织 Python 链式调用以便阅读，新增种子与并行度建议、RMSE 评估片段及数据边界说明，均已标出。本文未执行示例中的 spark-submit、模型拟合或数据加载。文件检查和代码静态解析不被称为运行测试。
版权与许可：Apache Spark，Copyright 2014 and onwards The Apache Software Foundation。示例源码以 Apache License 2.0 发布，本文下方保留 Apache License 2.0 与相关 NOTICE 归属声明；示例代码原许可适用。不把代码许可自动扩展为任意第三方数据的许可。中文译解、技术补充和示意图为本次编辑新增。
Apache License 2.0 与归属声明
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
Apache Spark NOTICE：与示例相关的项目归属
Apache Spark Copyright 2014 and onwards The Apache Software Foundation. This product includes software developed at The Apache Software Foundation (http://www.apache.org/).
暂无评论内容