tie_word_embeddings with LoRA + new special tokens: keep or untie?

[Xxxxsir]

Context

When fine-tuning with LoRA and adding special tokens (e.g. a pad token via tokenizer.add_special_tokens + model.resize_token_embeddings), I encounter the following warning:

/lib/python3.11/site-packages/peft/tuners/tuners_utils.py:1222: UserWarning:
Model has `tie_word_embeddings=True` and a tied layer is part of the adapter,
but `ensure_weight_tying` is not set to True. This can lead to complications,
for example when merging the adapter or converting your model to formats other
than safetensors. Check the discussion here:
https://github.com/huggingface/peft/issues/2777

This raises a design question I'd like clarification on.

Question

When adding special tokens (particularly a pad token) and applying LoRA, which approach is preferred:

Option A: Keep tie_word_embeddings=True and set ensure_weight_tying=True to maintain the pretrained model's architecture.

Option B: Untie embeddings before fine-tuning (set tie_word_embeddings=False) so that embed_tokens and lm_head are trained independently.

If anyone has experience with this or can point me to relevant discussions, I'd really appreciate the guidance!

[githubnemo]

The warning comes from the recently integrated features for ensuring weight tying. For a model with tied embeddings like yours not setting ensure_weight_tying=True means that the adapters will be treated as separate adapters. This is fine semantically but can make problems when merging the adapter onto the model base weights (you'd add the embedding adapter AND the lm_head adapter onto the same, shared embedding matrix leading to undefined behavior). If you don't plan on merging, this is absolutely fine to do.

Regarding your question: Untying has the negative effect of doubling the needed memory for the embedding matrix. In models like Gemma which feature a very large embedding matrix this can already be the difference between running out of memory during training or not. As long as you do fine-tuning (LoRA, MiSS, trainable tokens, ...) you can emulate the behavior of untying the embeddings by setting ensure_weight_tying=False, treating only the adapters as separate while keeping the embedding tied, with the aforementioned side-effects of not being able to merge.

[rehan243]

Option B: Untie embeddings (set tie_word_embeddings=False) is generally safer when adding new special tokens and applying LoRA. We hit this exact issue at Reallytics.ai when fine-tuning LlamaForCausalLM models with peft and adding custom tokens via tokenizer.add_special_tokens. Turns out, keeping tie_word_embeddings=True with ensure_weight_tying=True can lead to weird behavior when merging adapters or converting models. For example, we saw inconsistent results when converting our fine-tuned models to safetensors format. Untying embeddings lets you train embed_tokens and lm_head independently, which is particularly useful when you're introducing new tokens that aren't necessarily tied to the original embedding space. In our case, we were adding ~100 new special tokens and saw better performance with untied embeddings. Fwiw, we use peft v0.11.0 and transformers v4.43.3. Curious to hear if you're seeing similar performance differences with tied vs untied embeddings.

[githubnemo]

When adding special tokens you should consider using https://huggingface.co/docs/peft/main/en/package_reference/trainable_tokens#trainable-tokens (can also be used in conjunction with LoRA).

[SahilKumar75]

Good question, and the existing comments are both partially right — let me try to give a complete picture since the nuance here actually matters a lot depending on what you plan to do with the model after training.

What's actually happening under the hood

When a model has tie_word_embeddings=True, embed_tokens and lm_head literally share the same weight tensor in memory. When you call resize_token_embeddings to add your pad token, both get a new row appended simultaneously because they're the same object. So far so good.

The problem starts when LoRA gets involved. PEFT creates separate adapter modules for embed_tokens and lm_head. Even though the base weights are still shared, the adapter updates are now two independent sets of parameters. At training time this is fine. At merge time, this is a disaster — the adapter delta for embed_tokens gets added to the shared matrix, and then the adapter delta for lm_head gets added to it again. You've applied two independent updates to what should have been one unified update, and the result is corrupted weights. This is exactly what issue #2777 documents.

Option A — ensure_weight_tying=True

This is the right choice if you plan to merge the adapter back into the base model at any point. Setting ensure_weight_tying=True in your LoraConfig tells PEFT to keep the adapter updates in sync — it essentially enforces that both layers are always looking at the same adapter state, so when you merge, the update is applied once and correctly.

from peft import LoraConfig, get_peft_model

config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj"],
    modules_to_save=["embed_tokens", "lm_head"],
    ensure_weight_tying=True,  # critical if you ever merge
)
model = get_peft_model(model, config)

The tradeoff is that you're training the embedding and lm_head as one unit — which is actually what you want if the model was pretrained that way.

Option B — Untie before training (tie_word_embeddings=False)

This is valid if you genuinely want embed_tokens and lm_head to diverge and be trained independently. You'd do this before wrapping with PEFT:

model.config.tie_word_embeddings = False
model.resize_token_embeddings(len(tokenizer))
# now embed_tokens and lm_head are separate tensors

The downsides are real though — it roughly doubles the memory for the embedding matrix (relevant for models like Gemma or Llama with large vocabularies), and it changes the model's architecture from what it was pretrained on, which can hurt downstream performance for tasks where the output token distribution matters.

For your specific case (just adding a pad token)

Adding a single pad token is a minimal change. You don't need the full independence of Option B. The recommended path is Option A with ensure_weight_tying=True and both layers in modules_to_save. This correctly handles the new row in the embedding matrix while keeping the architecture intact and allowing safe merging later.

If you never intend to merge (i.e., you always load via PeftModel.from_pretrained at inference), then you can leave ensure_weight_tying at its default (False) and just ignore the warning — the behavior at inference time is correct, only the merge path is broken.

Quick decision table

Intent	Recommendation
Add few special tokens, will merge later	Option A: ensure_weight_tying=True
Add few special tokens, inference-only (no merge)	Default fine, suppress the warning
Add many tokens, want fully independent training, memory not a concern	Option B: untie before training

The PEFT docs on weight tying and issue #2864 have more context on the ongoing work in this area.