MixedModel.delete_adapter: stale active adapter and leaked auxiliary modules

[DaoyuanLi2816]

System Info

PEFT main at d1536ce1, transformers 5.14.1, torch 2.13.0+cpu, Python 3.10.11, CPU.

Problem

MixedModel.delete_adapter has two independent defects, both caused by the same block. I'm filing them together since they are a few lines apart and a fix touches the same loop. Neither reproduces on the non-mixed PeftModel path, which I checked as a control.

for adapter_to_delete in adapter_names:
    del self.peft_config[adapter_to_delete]

    key_list = [key for key, _ in self.model.named_modules() if not any(prefix in key for prefix in PREFIXES)]
    new_adapter = None
    for key in key_list:
        _, target, _ = _get_submodules(self.model, key)
        if isinstance(target, BaseTunerLayer):
            target.delete_adapter(adapter_to_delete)
            if new_adapter is None:
                new_adapter = target.active_adapters[:]

self.active_adapter = new_adapter or []
if adapter_to_delete in adapter_names:
    _delete_auxiliary_adapter(self.model, adapter_to_delete, new_active_adapters=new_adapter)

1. Deleting the active adapter can leave the model with no active adapter while a layer still applies one

new_adapter is taken from the first BaseTunerLayer reached, via if new_adapter is None. In a mixed model the adapters typically target different layers, so the first layer does not host the surviving adapter, and its active_adapters is empty after the deletion. That empty list becomes the model-wide active adapter.

c0 = LoraConfig(target_modules=["lin0"])
model = get_peft_model(Net(), c0, "adapter0", mixed=True)
model.add_adapter("adapter1", LoHaConfig(target_modules=["lin1"]))
model.base_model.set_adapter("adapter0")
model.delete_adapter("adapter0")

Observed:

before: model.active_adapter = adapter0
  layer lin0: available=['adapter0'] active=['adapter0']
  layer lin1: available=['adapter1'] active=['adapter0']
after delete_adapter('adapter0'):
  model.active_adapter = []          <-- no active adapter
  peft_config          = ['adapter1']
  layer lin0: available=[]           active=[]
  layer lin1: available=['adapter1'] active=['adapter1']   <-- but this layer is applying adapter1
  forward: OK (2, 4)

The forward runs and applies adapter1, so nothing raises — the model just reports an active-adapter set that disagrees with what it computes. Anything reading active_adapter (state-dict selection, status reporting, a later set_adapter round-trip) sees [] while the forward is adapted.

2. Deleting several adapters in one call only cleans up the last one's auxiliary modules

_delete_auxiliary_adapter is called once, after the loop, with adapter_to_delete left over from the final iteration. Every earlier name keeps its entry in the ModulesToSaveWrapper. It is positional, not name-dependent — reversing the argument order changes which adapter leaks:

model = build(modules_to_save=["head"])   # adapter0 + adapter1
model.delete_adapter(["adapter0", "adapter1"])

Observed:

modules_to_save before : ['adapter0', 'adapter1']
peft_config after      : []
modules_to_save after  : ['adapter0']     <-- leaked

# reversed order
model.delete_adapter(["adapter1", "adapter0"])
modules_to_save after  : ['adapter1']     <-- leaked

# non-mixed PeftModel control, deleting both
modules_to_save after  : []               <-- clean

So peft_config is empty while the wrapper still holds trained modules_to_save weights for the adapters that were supposedly deleted.

Also in that line, the if adapter_to_delete in adapter_names: guard is always true, since adapter_to_delete is by construction the last element of adapter_names.

Proposed fix

Move the auxiliary deletion inside the loop so it runs per name, and resolve the new active adapter across all tuner layers rather than trusting the first one — the union of the layers' remaining active adapters, or the surviving peft_config keys, rather than new_adapter = target.active_adapters[:] on first hit. I would keep the change confined to this method and add regression tests to tests/test_mixed.py covering both cases plus the non-mixed control.

I have the reproductions above but have not written the fix yet, and won't open a PR unless this is assigned to me. Related: I mentioned defect 2 in passing on #3504; this issue supersedes that note.

AI assistance disclosure

AI assistance was used for the investigation and write-up. I ran the reproductions above against the checkout named in System Info and reviewed the results.

[BenjaminBossan]

IIUC, the fix would be to homogenize the code with what we find on BaseTuner. I'm fine with that decision.

if adapter_to_delete in adapter_names:
    _delete_auxiliary_adapter(self.model, adapter_to_delete, new_active_adapters=new_adapter)

I'm sure this is just a typo, the intent was for adapter_to_delete in adapter_names:.

[DaoyuanLi2816]

Opened #3515. You were right about the typo — _delete_auxiliary_adapter now runs inside the loop, which also matches how BaseTuner delegates it to the shared delete_adapter helper.

One thing worth flagging while homogenizing: the first-layer heuristic for the new active adapter (if new_adapter is None: new_adapter = target.active_adapters[:]) exists in that shared helper too. It is harmless there, since non-mixed models have homogeneous tuner layers, but in a mixed model the first layer does not host the survivor, so I resolved it across all layers in MixedModel only rather than touching a path every tuner uses. Happy to align them if you would rather have one implementation.