嵌入模型通常采用多层编码器。例如，all-mpnet-base-v2 有 12 层，all-MiniLM-L6-v2 有 6 层。通常，生成嵌入必须依次经过所有层。2D Matryoshka Sentence Embeddings（2DMSE）预印本重新审视了这一做法：通过专门训练，使模型只使用部分层时也能保持较好的表现，从而用相对较小的效果损失换取更快的推理。
2DMSE 预印本后来更新并更名为 ESE: Espresso Sentence Embeddings。Sentence Transformers 的 Adaptive Layers 和 Matryoshka2d（自适应层与 Matryoshka 嵌入的结合）实现以最初的预印本为基础；项目欢迎实现新版 ESE 论文的贡献。
适用场景
2DMSE 论文指出，对采用 Adaptive Layers 和 Matryoshka Representation Learning 训练的大模型，只使用其中几层，有可能优于按常规嵌入方式训练的小模型。
实验结果
官方示例比较了自适应层模型与常规模型。作者为实验训练了两个模型：
- tomaarsen/mpnet-base-nli-adaptive-layer：以 microsoft/mpnet-base 为基础运行 adaptive_layer_nli.py 训练。
- tomaarsen/mpnet-base-nli：与前者几乎相同，同样基于 microsoft/mpnet-base，但仅使用
MultipleNegativesRankingLoss，没有再叠加AdaptiveLayerLoss。
两个模型都使用 AllNLI 训练。AllNLI 由 SNLI 和 MultiNLI 拼接而成。作者在 STSBenchmark 测试集上以多个嵌入维度进行评估。查看官方实验图：层数、STSB 表现及 CPU/GPU 加速比。
第一幅图显示，减少层数时，自适应层模型能保留更多效果。第二幅图更清楚地展示了这一点：即使只保留 1 层，仍保留约 80% 的表现。
第三幅图展示作者测试中 GPU 和 CPU 的加速比。删去一半层数，速度约提升到 2 倍，代价是 STSB 表现下降约 15%（Spearman 相关系数约从 86 降至 75）。继续减少层数时，CPU 的速度收益更大；原测试中，以约 20% 的效果损失换取 5 至 10 倍加速是可行的。这些比例描述该实验，不能直接作为其他模型、数据或硬件的性能保证。
训练
训练自适应层模型的核心很直接：除了对最后一层的嵌入应用损失函数，还对先前各层经过池化得到的嵌入应用同一损失。同时，加入 KL 散度损失，让非最终层的嵌入接近最终层。这可以理解为一种知识蒸馏：最后一层充当教师，前面的层充当学生。
例如，对 12 层的 microsoft/mpnet-base，训练目标会使每一层的输出都能生成有意义的嵌入：
from sentence_transformers import SentenceTransformer
from sentence_transformers.sentence_transformer.losses import CoSENTLoss, AdaptiveLayerLoss
# Loading in fp32 is preferred for training if your memory can handle it
model = SentenceTransformer("microsoft/mpnet-base", model_kwargs={"torch_dtype": "float32"})
base_loss = CoSENTLoss(model=model)
loss = AdaptiveLayerLoss(model=model, loss=base_loss)
接口参考：AdaptiveLayerLoss。官方文档指出，使用它并不会明显拖慢训练。
它还可以与 MatryoshkaLoss 结合，同时减少模型层数与输出向量维度。有关缩减输出维度的内容，参见 Matryoshka Embeddings。Sentence Transformers 将这两种损失的结合称为 Matryoshka2dLoss，并提供了便于训练的接口：
from sentence_transformers import SentenceTransformer
from sentence_transformers.sentence_transformer.losses import CoSENTLoss, Matryoshka2dLoss
model = SentenceTransformer("microsoft/mpnet-base", model_kwargs={"torch_dtype": "float32"})
base_loss = CoSENTLoss(model=model)
loss = Matryoshka2dLoss(model=model, loss=base_loss, matryoshka_dims=[768, 512, 256, 128, 64])
接口参考：Matryoshka2dLoss。
推理
模型经过 Adaptive Layer 损失训练后，可以截断到所需层数。这个过程需要调整模型结构；不同模型的内部组织不同，因此具体步骤也会有所变化。
先加载模型并访问底层的 transformers 模型：
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("tomaarsen/mpnet-base-nli-adaptive-layer")
# We can access the underlying model with `model.transformers_model`
print(model.transformers_model)
官方示例中的模型结构如下：
MPNetModel(
(embeddings): MPNetEmbeddings(
(word_embeddings): Embedding(30527, 768, padding_idx=1)
(position_embeddings): Embedding(514, 768, padding_idx=1)
(LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
(dropout): Dropout(p=0.1, inplace=False)
)
(encoder): MPNetEncoder(
(layer): ModuleList(
(0-11): 12 x MPNetLayer(
(attention): MPNetAttention(
(attn): MPNetSelfAttention(
(q): Linear(in_features=768, out_features=768, bias=True)
(k): Linear(in_features=768, out_features=768, bias=True)
(v): Linear(in_features=768, out_features=768, bias=True)
(o): Linear(in_features=768, out_features=768, bias=True)
(dropout): Dropout(p=0.1, inplace=False)
)
(LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
(dropout): Dropout(p=0.1, inplace=False)
)
(intermediate): MPNetIntermediate(
(dense): Linear(in_features=768, out_features=3072, bias=True)
(intermediate_act_fn): GELUActivation()
)
(output): MPNetOutput(
(dense): Linear(in_features=3072, out_features=768, bias=True)
(LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
(dropout): Dropout(p=0.1, inplace=False)
)
)
)
(relative_attention_bias): Embedding(32, 12)
)
(pooler): MPNetPooler(
(dense): Linear(in_features=768, out_features=768, bias=True)
(activation): Tanh()
)
)
实际输出随模型而异。需要找到编码器中的重复层。这个 MPNet 模型的层列表位于 model.transformers_model.encoder.layer。截取前几层即可减少前向计算：
new_num_layers = 3
model.transformers_model.encoder.layer = model.transformers_model.encoder.layer[:new_num_layers]
接着用 SentenceTransformer.encode 推理：
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("tomaarsen/mpnet-base-nli-adaptive-layer")
new_num_layers = 3
model.transformers_model.encoder.layer = model.transformers_model.encoder.layer[:new_num_layers]
embeddings = model.encode(
[
"The weather is so nice!",
"It's so sunny outside!",
"He drove to the stadium.",
]
)
# Similarity of the first sentence with the other two
similarities = model.similarity(embeddings[0], embeddings[1:])
# => tensor([[0.7761, 0.1655]])
# compared to tensor([[ 0.7547, -0.0162]]) for the full model
示例中的两个相关句子，即使只使用 3 层，其相似度也明显高于无关句子。可以在本地复制脚本，调整 new_num_layers，观察相似度如何变化。代码注释里的数值是官方示例结果。
完整代码示例
以下脚本展示如何实际应用 AdaptiveLayerLoss：
- adaptive_layer_nli.py：将 MultipleNegativesRankingLoss 与 AdaptiveLayerLoss 结合，用自然语言推断（NLI）数据训练嵌入模型，是 NLI 文档示例的改编。
- adaptive_layer_sts.py：将 CoSENTLoss 与 AdaptiveLayerLoss 结合，在 STSBenchmark 训练集上训练，是 STS 文档示例的改编。
以下脚本展示 Matryoshka2dLoss 的用法：
- 2d_matryoshka_nli.py：结合 MultipleNegativesRankingLoss 与 Matryoshka2dLoss，使用 NLI 数据训练，是 NLI 文档示例的改编。
- 2d_matryoshka_sts.py：结合 CoSENTLoss 与 Matryoshka2dLoss，在 STSBenchmark 训练集上训练，是 STS 文档示例的改编。
来源：Sentence Transformers：Adaptive Layers，Sentence Transformers 文档贡献者。Copyright 2019 Nils Reimers。文档和示例按 Apache License 2.0 提供。本中文版本翻译并调整了叙述，补充了性能适用范围说明；代码保持原样。2026-10-03 核对当前原文，代码只做静态检查，未重新训练或运行模型。
Apache License 2.0 许可全文
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
© 版权声明
文章版权归作者所有，未经允许请勿转载。
THE END
暂无评论内容