LoRA finetuning fails

[sefinch]

Is finetuning with LoRA supported? Based on the code, it looks like it should be supported, but I cannot get it to work.

Specifically, I tried adding "--lora_enable True" to the finetuning bash script. But I get the following error:

Traceback (most recent call last):
  File "HumanOmni/humanomni/train_flash_attn.py", line 13, in <module>
    train(attn_implementation="flash_attention_2")
  File "HumanOmni/./humanomni/train_humanomni.py", line 722, in train
    model = get_peft_model(model, lora_config)
  File ".conda/envs/humanomni/lib/python3.10/site-packages/peft/mapping.py", line 149, in get_peft_model
    return MODEL_TYPE_TO_PEFT_MODEL_MAPPING[peft_config.task_type](model, peft_config, adapter_name=adapter_name)
  File ".conda/envs/humanomni/lib/python3.10/site-packages/peft/peft_model.py", line 1395, in __init__
    super().__init__(model, peft_config, adapter_name)
  File ".conda/envs/humanomni/lib/python3.10/site-packages/peft/peft_model.py", line 138, in __init__
    self.base_model = cls(model, {adapter_name: peft_config}, adapter_name)
  File ".conda/envs/humanomni/lib/python3.10/site-packages/peft/tuners/lora/model.py", line 139, in __init__
    super().__init__(model, config, adapter_name)
  File ".conda/envs/humanomni/lib/python3.10/site-packages/peft/tuners/tuners_utils.py", line 166, in __init__
    self.inject_adapter(self.model, adapter_name)
  File ".conda/envs/humanomni/lib/python3.10/site-packages/peft/tuners/tuners_utils.py", line 372, in inject_adapter
    self._create_and_replace(peft_config, adapter_name, target, target_name, parent, current_key=key)
  File ".conda/envs/humanomni/lib/python3.10/site-packages/peft/tuners/lora/model.py", line 223, in _create_and_replace
    new_module = self._create_new_module(lora_config, adapter_name, target, **kwargs)
  File ".conda/envs/humanomni/lib/python3.10/site-packages/peft/tuners/lora/model.py", line 320, in _create_new_module
    raise ValueError(
ValueError: Target module Qwen2DecoderLayer(
  (self_attn): Qwen2Attention(
    (q_proj): Linear(in_features=3584, out_features=3584, bias=True)
    (k_proj): Linear(in_features=3584, out_features=512, bias=True)
    (v_proj): Linear(in_features=3584, out_features=512, bias=True)
    (o_proj): Linear(in_features=3584, out_features=3584, bias=False)
  )
  (mlp): Qwen2MLP(
    (gate_proj): Linear(in_features=3584, out_features=18944, bias=False)
    (up_proj): Linear(in_features=3584, out_features=18944, bias=False)
    (down_proj): Linear(in_features=18944, out_features=3584, bias=False)
    (act_fn): SiLU()
  )
  (input_layernorm): Qwen2RMSNorm((0,), eps=1e-06)
  (post_attention_layernorm): Qwen2RMSNorm((0,), eps=1e-06)
) is not supported. Currently, only the following modules are supported: torch.nn.Linear, torch.nn.Embedding, torch.nn.Conv2d, transformers.pytorch_utils.Conv1D.

[Mr-Millionairelive]

Hello. Is your problem solved? If so, what is the graphics card resource used?