IA3: merging a grouped convolution fails when the layer is a feedforward module

[DaoyuanLi2816]

System Info

PEFT main at d1536ce1, transformers 5.14.1, torch 2.13.0+cpu, Python 3.10.11, CPU.

Problem

Split out of #3472 as you suggested. You noted there that groups != 1 is uncommon and this is not high priority, so this is filed for the record rather than as something urgent.

Merging an IA3 adapter into a grouped convolution raises a shape error when the layer is a feedforward module:

import torch
from torch import nn
from peft import IA3Config, get_peft_model

class ConvNet(nn.Module):
    def __init__(self, groups):
        super().__init__()
        self.conv = nn.Conv2d(4, 4, kernel_size=1, groups=groups)
    def forward(self, x):
        return self.conv(x)

for groups in (1, 2):
    model = get_peft_model(
        ConvNet(groups),
        IA3Config(target_modules=["conv"], feedforward_modules=["conv"], init_ia3_weights=False),
    )
    model.base_model.merge_adapter()

Observed:

groups=1: merged OK, output preserved=True, max abs diff=5.960e-08
groups=2: RuntimeError: The size of tensor a (2) must match the size of tensor b (4) at non-singleton dimension 1

Root cause

A grouped convolution stores its weight with in_channels // groups in dimension 1, while the feedforward IA3 scaling has one value per full input channel. Multiplying the two broadcasts against mismatched shapes. Non-feedforward IA3 scales the output channels instead, which is why it is unaffected: the same script with feedforward_modules=[] merges cleanly at both group counts.

The fix is to align the scaling per group in both merge and unmerge so the two stay inverse.

I have a fix and regression tests locally but am not opening a PR until this is assigned, and I am happy for it to stay in the backlog given the priority you gave it.

AI assistance disclosure

AI assistance was used for the investigation and write-up. I ran the reproduction above against the checkout named in System Info and reviewed the result.