Feature request
I would like to propose adding Echo-LoRA (arXiv 2605.08177) as a new tuner to huggingface/peft.
Echo-LoRA improves shallow LoRA/DoRA modules by injecting boundary hidden states from deeper source layers during training. Shallow layers only see early syntactic representations — Echo-LoRA gives them access to a compact signal derived from deeper semantic states via learned projection and gating networks. Since the Echo pathway is only active during training, inference remains identical to standard LoRA with no additional cost.
Results: +5.7 points average over LoRA across 8 commonsense reasoning benchmarks on LLaMA-7B/2-7B/3-8B. Echo-DoRA improves DoRA by +2.7 points on the same suite.
Paper: https://arxiv.org/abs/2605.08177
Code: No official implementation is currently available.
Echo-LoRA builds on the existing LoRA formulation by introducing a training-time mechanism for cross-layer representation injection, making it a natural fit as another PEFT tuner.
From my investigation of the current PEFT architecture, it appears Echo-LoRA could be integrated as a new tuner without requiring framework-wide changes. The primary architectural difference from existing tuners is a training-time two-pass forward used to collect and later inject hidden representations. Similar to several existing PEFT tuners, merge support would not be provided initially.
Before starting the implementation, I'd like to confirm whether implementing the two-pass training logic inside the tuner model is consistent with PEFT's intended architecture, or whether another integration pattern would be preferred.
Your contribution
I've begun prototyping the tuner structure (starting with EchoLoraConfig and the tuner layout) and have investigated how the design fits into PEFT's existing adapter architecture, including how tuner registration, configuration, layer replacement, and the adapter lifecycle are handled within PEFT. If the proposal aligns with the project's direction, I'd be happy to implement the full tuner, add the required tests, documentation, and example usage in a pull request.
Feature request
I would like to propose adding Echo-LoRA (arXiv 2605.08177) as a new tuner to huggingface/peft.
Echo-LoRA improves shallow LoRA/DoRA modules by injecting boundary hidden states from deeper source layers during training. Shallow layers only see early syntactic representations — Echo-LoRA gives them access to a compact signal derived from deeper semantic states via learned projection and gating networks. Since the Echo pathway is only active during training, inference remains identical to standard LoRA with no additional cost.
Results: +5.7 points average over LoRA across 8 commonsense reasoning benchmarks on LLaMA-7B/2-7B/3-8B. Echo-DoRA improves DoRA by +2.7 points on the same suite.
Paper: https://arxiv.org/abs/2605.08177
Code: No official implementation is currently available.
Echo-LoRA builds on the existing LoRA formulation by introducing a training-time mechanism for cross-layer representation injection, making it a natural fit as another PEFT tuner.
From my investigation of the current PEFT architecture, it appears Echo-LoRA could be integrated as a new tuner without requiring framework-wide changes. The primary architectural difference from existing tuners is a training-time two-pass forward used to collect and later inject hidden representations. Similar to several existing PEFT tuners, merge support would not be provided initially.
Before starting the implementation, I'd like to confirm whether implementing the two-pass training logic inside the tuner model is consistent with PEFT's intended architecture, or whether another integration pattern would be preferred.
Your contribution
I've begun prototyping the tuner structure (starting with EchoLoraConfig and the tuner layout) and have investigated how the design fits into PEFT's existing adapter architecture, including how tuner registration, configuration, layer replacement, and the adapter lifecycle are handled within PEFT. If the proposal aligns with the project's direction, I'd be happy to implement the full tuner, add the required tests, documentation, and example usage in a pull request.