We are focusing on how to Control text-to-image diffusion models with Novel Conditions.
For more detailed information, please refer to our survey paper: Controllable Generation with Text-to-Image Diffusion Models: A Survey
@article{cao2024controllable,
title={Controllable Generation with Text-to-Image Diffusion Models: A Survey},
author={Pu Cao and Feng Zhou and Qing Song and Lu Yang},
journal={arXiv preprint arXiv:2403.04279},
year={2024}
}
Since the following content is generated based on our database, please provide the following information in the issue to help us fill in the database to add new papers (please do not submit a PR directly).
1. Paper title
2. arXiv ID (if any)
3. Publication status (if any)
- Generation with Specific Condition
- Generation with Multiple Conditions
- Universal Controllable Generation
Block-wise LoRA: Revisiting Fine-grained LoRA for Effective Personalization and Stylization in Text-to-Image Generation.
Likun Li, Haoqi Zeng, Changpeng Yang, Haozhe Jia, Di Xu.
arXiv 2024. [PDF]
FaceChain-SuDe: Building Derived Class to Inherit Category Attributes for One-shot Subject-Driven Generation.
Pengchong Qiao, Lei Shang, Chang Liu, Baigui Sun, Xiangyang Ji, Jie Chen.
arXiv 2024. [PDF]
RealCustom: Narrowing Real Text Word for Real-Time Open-Domain Text-to-Image Customization.
Mengqi Huang, Zhendong Mao, Mingcong Liu, Qian He, Yongdong Zhang.
arXiv 2024. [PDF]
DiffuseKronA: A Parameter Efficient Fine-tuning Method for Personalized Diffusion Models.
Shyam Marjit, Harshit Singh, Nityanand Mathur, Sayak Paul, Chia-Mu Yu, Pin-Yu Chen.
arXiv 2024. [PDF]
Direct Consistency Optimization for Compositional Text-to-Image Personalization.
Kyungmin Lee, Sangkyung Kwak, Kihyuk Sohn, Jinwoo Shin.
arXiv 2024. [PDF]
ComFusion: Personalized Subject Generation in Multiple Specific Scenes From Single Image.
Yan Hong, Jianfu Zhang.
arXiv 2024. [PDF]
Visual Concept-driven Image Generation with Text-to-Image Diffusion Model.
Tanzila Rahman, Shweta Mahajan, Hsin-Ying Lee, Jian Ren, Sergey Tulyakov, Leonid Sigal.
arXiv 2024. [PDF]
Textual Localization: Decomposing Multi-concept Images for Subject-Driven Text-to-Image Generation.
Junjie Shentu, Matthew Watson, Noura Al Moubayed.
arXiv 2024. [PDF]
DreamMatcher: Appearance Matching Self-Attention for Semantically-Consistent Text-to-Image Personalization.
Jisu Nam, Heesu Kim, DongJae Lee, Siyoon Jin, Seungryong Kim, Seunggyu Chang.
CVPR 2024. [PDF]
SeFi-IDE: Semantic-Fidelity Identity Embedding for Personalized Diffusion-Based Generation.
Yang Li, Songlin Yang, Wei Wang, Jing Dong.
arXiv 2024. [PDF]
Pick-and-Draw: Training-free Semantic Guidance for Text-to-Image Personalization.
Henglei Lv, Jiayu Xiao, Liang Li, Qingming Huang.
arXiv 2024. [PDF]
Object-Driven One-Shot Fine-tuning of Text-to-Image Diffusion with Prototypical Embedding.
Jianxiang Lu, Cong Xie, Hui Guo.
arXiv 2024. [PDF]
BootPIG: Bootstrapping Zero-shot Personalized Image Generation Capabilities in Pretrained Diffusion Models.
Senthil Purushwalkam, Akash Gokul, Shafiq Joty, Nikhil Naik.
arXiv 2024. [PDF]
PALP: Prompt Aligned Personalization of Text-to-Image Models.
Moab Arar, Andrey Voynov, Amir Hertz, Omri Avrahami, Shlomi Fruchter, Yael Pritch, Daniel Cohen-Or, Ariel Shamir.
arXiv 2024. [PDF]
Cross Initialization for Personalized Text-to-Image Generation.
Lianyu Pang, Jian Yin, Haoran Xie, Qiping Wang, Qing Li, Xudong Mao.
CVPR 2024. [PDF]
DreamTuner: Single Image is Enough for Subject-Driven Generation.
Miao Hua, Jiawei Liu, Fei Ding, Wei Liu, Jie Wu, Qian He.
arXiv 2023. [PDF]
Decoupled Textual Embeddings for Customized Image Generation.
Yufei Cai, Yuxiang Wei, Zhilong Ji, Jinfeng Bai, Hu Han, Wangmeng Zuo.
arXiv 2023. [PDF]
Compositional Inversion for Stable Diffusion Models.
Xulu Zhang, Xiao-Yong Wei, Jinlin Wu, Tianyi Zhang, Zhaoxiang Zhang, Zhen Lei, Qing Li.
AAAI 2024. [PDF]
Customization Assistant for Text-to-image Generation.
Yufan Zhou, Ruiyi Zhang, Jiuxiang Gu, Tong Sun.
CVPR 2024. [PDF]
VideoBooth: Diffusion-based Video Generation with Image Prompts.
Yuming Jiang, Tianxing Wu, Shuai Yang, Chenyang Si, Dahua Lin, Yu Qiao, Chen Change Loy, Ziwei Liu.
arXiv 2023. [PDF]
HiFi Tuner: High-Fidelity Subject-Driven Fine-Tuning for Diffusion Models.
Zhonghao Wang, Wei Wei, Yang Zhao, Zhisheng Xiao, Mark Hasegawa-Johnson, Humphrey Shi, Tingbo Hou.
arXiv 2023. [PDF]
VideoAssembler: Identity-Consistent Video Generation with Reference Entities using Diffusion Model.
Haoyu Zhao, Tianyi Lu, Jiaxi Gu, Xing Zhang, Zuxuan Wu, Hang Xu, Yu-Gang Jiang.
arXiv 2023. [PDF]
CLiC: Concept Learning in Context.
Mehdi Safaee, Aryan Mikaeili, Or Patashnik, Daniel Cohen-Or, Ali Mahdavi-Amiri.
CVPR 2024. [PDF]
CatVersion- Concatenating Embeddings for Diffusion-Based Text-to-Image Personalization.
Ruoyu Zhao, Mingrui Zhu, Shiyin Dong, Nannan Wang, Xinbo Gao.
arXiv 2023. [PDF]
Lego- Learning to Disentangle and Invert Concepts Beyond Object Appearance in Text-to-Image Diffusion Models.
Saman Motamed, Danda Pani Paudel, Luc Van Gool.
arXiv 2023. [PDF]
An Image is Worth Multiple Words- Multi-attribute Inversion for Constrained Text-to-Image Synthesis.
Aishwarya Agarwal, Srikrishna Karanam, Tripti Shukla, Balaji Vasan Srinivasan.
arXiv 2023. [PDF]
DIFFNAT- Improving Diffusion Image Quality Using Natural Image Statistics.
Aniket Roy, Maiterya Suin, Anshul Shah, Ketul Shah, Jiang Liu, Rama Chellappa.
arXiv 2023. [PDF]
A Data Perspective on Enhanced Identity Preservation for Diffusion Personalization.
Xingzhe He, Zhiwen Cao, Nicholas Kolkin, Lantao Yu, Helge Rhodin, Ratheesh Kalarot.
arXiv 2023. [PDF]
AN IMAGE IS WORTH MULTIPLE WORDS : LEARNING OBJECT LEVEL CONCEPTS USING MULTI-CONCEPT PROMPT LEARNING.
Chen Jin, Ryutaro Tanno, Amrutha Saseendran, Tom Diethe, Philip Teare.
arXiv 2023. [PDF]
Kosmos-G: Generating Images in Context with Multimodal Large Language Models.
Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, Furu Wei.
arXiv 2023. [PDF]
Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluation.
Shin-Ying Yeh, Yu-Guan Hsieh, Zhidong Gao, Bernard B W Yang, Giyeong Oh, Yanmin Gong.
ICLR 2024. [PDF]
Subject-Diffusion- Open Domain Personalized Text-to-Image Generation without Test-time Fine-tuning.
Jian Ma, Junhao Liang, Chen Chen, Haonan Lu.
ICLR2024 (3566). [PDF]
Domain-Agnostic Tuning-Encoder for Fast Personalization of Text-To-Image Models .
Moab Arar, Rinon Gal, Yuval Atzmon, Gal Chechik, Daniel Cohen-Or, Ariel Shamir, Amit H. Bermano.
SIGGRAPH ASIA 2023. [PDF]
Generate Anything Anywhere in Any Scene.
Yuheng Li, Haotian Liu, Yangming Wen, Yong Jae Lee.
arXiv 2023. [PDF]
Controlling Text-to-Image Diffusion by Orthogonal Finetuning .
Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, Bernhard Schölkopf.
NeurIPS 2023. [PDF]
COMCAT: Towards Efficient Compression and Customization of
Attention-Based Vision Models.
Jinqi Xiao, Miao Yin, Yu Gong, Xiao Zang, Jian Ren, Bo Yuan.
arXiv 2023. [PDF]
Break-A-Scene: Extracting Multiple Concepts from a Single Image.
Omri Avrahami, Kfir Aberman, Ohad Fried, Daniel Cohen-Or, Dani Lischinski.
SIGGRAPH ASIA 2023. [PDF]
ProSpect- Prompt Spectrum for Attribute-Aware Personalization of Diffusion Models.
Yuxin Zhang, Weiming Dong, Fan Tang, Nisha Huang, Haibin Huang, Chongyang Ma, Tong-Yee Lee, Oliver Deussen, Changsheng Xu.
TOG 2023. [PDF]
BLIP-Diffusion- Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing.
Dongxu Li, Junnan Li, Steven C. H. Hoi.
NeurIPS 2023. [PDF]
DisenBooth- Disentangled Parameter-Efficient Tuning for Subject-Driven Text-to-Image Generation.
Hong Chen, Yipeng Zhang, Simin Wu, Xin Wang, Xuguang Duan, Yuwei Zhou, Wenwu Zhu.
arXiv 2023. [PDF]
Key-Locked Rank One Editing for Text-to-Image Personalization.
Yoad Tewel, Rinon Gal, Gal Chechik, Yuval Atzmon.
SIGGRAPH 2023. [PDF]
Gradient-Free Textual Inversion.
Zhengcong Fei, Mingyuan Fan, Junshi Huang.
ACM MM 2023. [PDF]
Controllable Textual Inversion for Personalized Text-to-Image Generation.
Jianan Yang, Haobo Wang, Yanming Zhang, Ruixuan Xiao, Sai Wu, Gang Chen, Junbo Zhao.
arXiv 2023. [PDF]
InstantBooth- Personalized Text-to-Image Generation without Test-Time Finetuning.
Jing Shi, Wei Xiong, Zhe Lin, Hyun Joon Jung.
CVPR 2024. [PDF]
Taming Encoder for Zero Fine-tuning Image Customization with Text-to-Image Diffusion Models.
Xuhui Jia, Yang Zhao, Kelvin C. K. Chan, Yandong Li, Han Zhang, Boqing Gong, Tingbo Hou, Huisheng Wang, Yu-Chuan Su.
arXiv 2023. [PDF]
Subject-driven Text-to-Image Generation via Apprenticeship Learning.
Wenhu Chen, Hexiang Hu, Yandong Li, Nataniel Ruiz, Xuhui Jia, Ming-Wei Chang, William W. Cohen.
NeurIPS 2023. [PDF]
A Closer Look at Parameter-Efficient Tuning in Diffusion Models .
Chendong Xiang, Fan Bao, Chongxuan Li, Hang Su, Jun Zhu.
arXiv 2023. [PDF]
SVDiff- Compact Parameter Space for Diffusion Fine-Tuning.
Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, Feng Yang.
ICCV 2023. [PDF]
P+- Extended Textual Conditioning in Text-to-Image Generation.
Andrey Voynov, Qinghao Chu, Daniel Cohen-Or, Kfir Aberman.
arXiv 2023. [PDF]
Unified Multi-Modal Latent Diffusion for
Joint Subject and Text Conditional Image Generation.
Yiyang Ma, Huan Yang, Wenjing Wang, Jianlong Fu, Jiaying Liu.
arXiv 2023. [PDF]
ELITE- Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generation.
Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, Wangmeng Zuo.
ICCV 2023. [PDF]
Designing an Encoder for Fast Personalization of Text-to-Image Models.
Rinon Gal, Moab Arar, Yuval Atzmon, Amit H. Bermano, Gal Chechik, Daniel Cohen-Or.
arXiv 2023. [PDF]
Is This Loss Informative? Faster Text-to-Image Customization by Tracking Objective Dynamics.
Anton Voronov, Mikhail Khoroshikh, Artem Babenko, Max Ryabinin.
NeurIPS 2023. [PDF]
Multi-Concept Customization of Text-to-Image Diffusion.
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, Jun-Yan Zhu.
CVPR 2023. [PDF]
DreamArtist: Towards Controllable One-Shot Text-to-Image Generation via Positive-Negative Prompt-Tuning.
Ziyi Dong, Pengxu Wei, Liang Lin.
arXiv 2022. [PDF]
Re-Imagen: Retrieval-Augmented Text-to-Image Generator.
Wenhu Chen, Hexiang Hu, Chitwan Saharia, William W. Cohen.
ICLR 2023. [PDF]
DreamBooth- Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, Kfir Aberman.
CVPR 2023. [PDF]
An Image is Worth One Word- Personalizing Text-to-Image Generation using Textual Inversion.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, Daniel Cohen-Or.
ICLR 2023. [PDF]
Face2Diffusion for Fast and Editable Face Personalization.
Kaede Shiohara, Toshihiko Yamasaki.
CVPR 2024. [PDF]
CapHuman: Capture Your Moments in Parallel Universes.
Chao Liang, Fan Ma, Linchao Zhu, Yingying Deng, Yi Yang.
arXiv 2024. [PDF]
StableIdentity: Inserting Anybody into Anywhere at First Sight.
Qinghe Wang, Xu Jia, Xiaomin Li, Taiqing Li, Liqian Ma, Yunzhi Zhuge, Huchuan Lu.
arXiv 2024. [PDF]
InstantID: Zero-shot Identity-Preserving Generation in Seconds.
Qixun Wang, Xu Bai, Haofan Wang, Zekui Qin, Anthony Chen, Huaxia Li, Xu Tang, Yao Hu.
arXiv 2024. [PDF]
Towards a Simultaneous and Granular Identity-Expression Control in Personalized Face Generation.
Renshuai Liu, Bowen Ma, Wei Zhang, Zhipeng Hu, Changjie Fan, Tangjie Lv, Yu Ding, Xuan Cheng.
arXiv 2024. [PDF]
PortraitBooth: A Versatile Portrait Model for Fast Identity-preserved Personalization.
Xu Peng, Junwei Zhu, Boyuan Jiang, Ying Tai, Donghao Luo, Jiangning Zhang, Wei Lin, Taisong Jin, Chengjie Wang, Rongrong Ji.
CVPR 2024. [PDF]
Stellar: Systematic Evaluation of Human-Centric Personalized Text-to-Image Methods.
Panos Achlioptas, Alexandros Benetatos, Iordanis Fostiropoulos, Dimitris Skourtis.
arXiv 2023. [PDF]
PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding.
Zhen Li, Mingdeng Cao, Xintao Wang, Zhongang Qi, Ming-Ming Cheng, Ying Shan.
arXiv 2023. [PDF]
DemoCaricature: Democratising Caricature Generation with a Rough Sketch.
Dar-Yen Chen, Subhadeep Koley, Aneeshan Sain, Pinaki Nath Chowdhury, Tao Xiang, Ayan Kumar Bhunia, Yi-Zhe Song.
arXiv 2023. [PDF]
ViscoNet: Bridging and Harmonizing Visual and Textual Conditioning for ControlNet.
Soon Yau Cheong, Armin Mustafa, Andrew Gilbert.
arXiv 2023. [PDF]
FaceStudio: Put Your Face Everywhere in Seconds.
Yuxuan Yan, Chi Zhang, Rui Wang, Yichao Zhou, Gege Zhang, Pei Cheng, Gang Yu, Bin Fu.
arXiv 2023. [PDF]
Retrieving Conditions from Reference Images for Diffusion Models.
Haoran Tang, Xin Zhou, Jieren Deng, Zhihong Pan, Hao Tian, Pratik Chaudhari.
arXiv 2023. [PDF]
When StyleGAN Meets Stable Diffusion- a W + Adapter for Personalized Image Generation.
Xiaoming Li, Xinyu Hou, Chen Change Loy.
arXiv 2023. [PDF]
High-fidelity Person-centric Subject-to-Image Synthesis.
Yibin Wang, Weizhong Zhang, Jianwei Zheng, Cheng Jin.
arXiv 2023. [PDF]
MagiCapture- High-Resolution Multi-Concept Portrait Customization.
Junha Hyung, Jaeyo Shin, Jaegul Choo.
arXiv 2023. [PDF]
PhotoVerse- Tuning-Free Image Customization with Text-to-Image Diffusion Models.
Li Chen, Mengyi Zhao, Yiheng Liu, Mingxu Ding, Yangyang Song, Shizun Wang, Xu Wang, Hao Yang, Jing Liu, Kang Du, Min Zheng.
arXiv 2023. [PDF]
HyperDreamBooth- HyperNetworks for Fast Personalization of Text-to-Image Models .
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingbo Hou, Yael Pritch, Neal Wadhwa, Michael Rubinstein, Kfir Aberman.
CVPR 2024. [PDF]
DreamIdentity- Improved Editability for Efficient Face-identity Preserved Image Generation .
Zhuowei Chen, Shancheng Fang, Wei Liu, Qian He, Mengqi Huang, Yongdong Zhang, Zhendong Mao.
arXiv 2023. [PDF]
Face0- Instantaneously Conditioning a Text-to-Image Model on a Face.
Dani Valevski, Danny Wasserman, Yossi Matias, Yaniv Leviathan.
arXiv 2023. [PDF]
Conditioning Diffusion Models via Attributes and Semantic Masks for Face Generation.
Nico Giambi, Giuseppe Lisanti.
arXiv 2023. [PDF]
FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attention.
Guangxuan Xiao, Tianwei Yin, William T. Freeman, Frédo Durand, Song Han.
arXiv 2023. [PDF]
Visual Style Prompting with Swapping Self-Attention.
Jaeseok Jeong, Junho Kim, Yunjey Choi, Gayoung Lee, Youngjung Uh.
arXiv 2024. [PDF]
Towards Accurate Guided Diffusion Sampling through Symplectic Adjoint Method.
Jiachun Pan, Hanshu Yan, Jun Hao Liew, Jiashi Feng, Vincent Y. F. Tan.
arXiv 2023. [PDF]
Style Aligned Image Generation via Shared Attention.
Amir Hertz, Andrey Voynov, Shlomi Fruchter, Daniel Cohen-Or.
CVPR 2024. [PDF]
ArtAdapter: Text-to-Image Style Transfer using Multi-Level Style Encoder and Explicit Adaptation.
Dar-Yen Chen, Hamish Tennent, Ching-Wen Hsu.
arXiv 2023. [PDF]
StyleCrafter- Enhancing Stylized Text-to-Video Generation with Style Adapter.
Gongye Liu, Menghan Xia, Yong Zhang, Haoxin Chen, Jinbo Xing, Xintao Wang, Yujiu Yang, Ying Shan.
arXiv 2023. [PDF]
StyleDrop: Text-to-Image Generation in Any Style.
Kihyuk Sohn, Nataniel Ruiz, Kimin Lee, Daniel Castro Chin, Irina Blok, Huiwen Chang, Jarred Barber, Lu Jiang, Glenn Entis, Yuanzhen Li, Yuan Hao, Irfan Essa, Michael Rubinstein, Dilip Krishnan.
arXiv 2023. [PDF]
InteractDiffusion: Interaction Control in Text-to-Image Diffusion Models.
Jiun Tian Hoe, Xudong Jiang, Chee Seng Chan, Yap-Peng Tan, Weipeng Hu.
arXiv 2023. [PDF]
MotionCrafter: One-Shot Motion Customization of Diffusion Models.
Yuxin Zhang, Fan Tang, Nisha Huang, Haibin Huang, Chongyang Ma, Weiming Dong, Changsheng Xu.
arXiv 2023. [PDF]
DreaMoving: A Human Dance Video Generation Framework based on Diffusion Models.
Mengyang Feng, Jinlin Liu, Kai Yu, Yuan Yao, Zheng Hui, Xiefan Guo, Xianhui Lin, Haolan Xue, Chen Shi, Xiaowen Li, Aojie Li, Xiaoyang Kang, Biwen Lei, Miaomiao Cui, Peiran Ren, Xuansong Xie.
arXiv 2023. [PDF]
Customizing Motion in Text-to-Video Diffusion Models.
Joanna Materzynska, Josef Sivic, Eli Shechtman, Antonio Torralba, Richard Zhang, Bryan Russell.
arXiv 2023. [PDF]
SAVE: Protagonist Diversification with Structure Agnostic Video Editing.
Yeji Song, Wonsik Shin, Junsoo Lee, Jeesoo Kim, Nojun Kwak.
arXiv 2023. [PDF]
LAMP: Learn A Motion Pattern for Few-Shot-Based Video Generation.
Ruiqi Wu, Liangyu Chen, Tong Yang, Chunle Guo, Chongyi Li, Xiangyu Zhang.
arXiv 2023. [PDF]
MotionDirector- Motion Customization of Text-to-Video Diffusion Models.
Rui Zhao, Yuchao Gu, Jay Zhangjie Wu, David Junhao Zhang, Jiawei Liu, Weijia Wu, Jussi Keppo, Mike Zheng Shou.
arXiv 2023. [PDF]
AnimateDiff- Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, Bo Dai.
ICLR2024 (6688). [PDF]
ReVersion- Diffusion-Based Relation Inversion from Images .
Ziqi Huang, Tianxing Wu, Yuming Jiang, Kelvin C. K. Chan, Ziwei Liu.
arXiv 2023. [PDF]
FreeControl: Training-Free Spatial Control of Any Text-to-Image Diffusion Model with Any Condition.
Sicheng Mo, Fangzhou Mu, Kuan Heng Lin, Yanli Liu, Bochen Guan, Yin Li, Bolei Zhou.
arXiv 2023. [PDF]
Context Diffusion: In-Context Aware Image Generation.
Ivona Najdenkoska, Animesh Sinha, Abhimanyu Dubey, Dhruv Mahajan, Vignesh Ramanathan, Filip Radenovic.
arXiv 2023. [PDF]
ViscoNet: Bridging and Harmonizing Visual and Textual Conditioning for ControlNet.
Soon Yau Cheong, Armin Mustafa, Andrew Gilbert.
arXiv 2023. [PDF]
IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models.
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, Wei Yang.
arXiv 2023. [PDF]
Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Models.
Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Lu Yuan, Kwan-Yee K. Wong.
NeurIPS 2023. [PDF]
Prompt-Free Diffusion: Taking “Text” out of Text-to-Image Diffusion Models.
Xingqian Xu, Jiayi Guo, Zhangyang Wang, Gao Huang, Irfan Essa, Humphrey Shi.
arXiv 2023. [PDF]
Versatile Diffusion: Text, Images and Variations All in One Diffusion Model.
Xingqian Xu, Zhangyang Wang, Eric Zhang, Kai Wang, Humphrey Shi.
ICCV 2023. [PDF]
Hierarchical Text-Conditional Image Generation with CLIP Latents.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, Mark Chen.
arXiv 2022. [PDF]
DreamDistribution: Prompt Distribution Learning for Text-to-Image Diffusion Models.
Brian Nlong Zhao, Yuhang Xiao, Jiashu Xu, Xinyang Jiang, Yifan Yang, Dongsheng Li, Laurent Itti, Vibhav Vineet, Yunhao Ge.
arXiv 2023. [PDF]
Concept-centric Personalization with Large-scale Diffusion Priors.
Pu Cao, Lu Yang, Feng Zhou, Tianrui Huang, Qing Song.
arXiv 2023. [PDF]
It's All About Your Sketch: Democratising Sketch Control in Diffusion Models.
Subhadeep Koley, Ayan Kumar Bhunia, Deeptanshu Sekhri, Aneeshan Sain, Pinaki Nath Chowdhury, Tao Xiang, Yi-Zhe Song.
arXiv 2024. [PDF]
NoiseCollage: A Layout-Aware Text-to-Image Diffusion Model Based on Noise Cropping and Merging.
Takahiro Shirakawa, Seiichi Uchida.
arXiv 2024. [PDF]
PLACE: Adaptive Layout-Semantic Fusion for Semantic Image Synthesis.
Zhengyao Lv, Yuxiang Wei, Wangmeng Zuo, Kwan-Yee K. Wong.
arXiv 2024. [PDF]
Box It to Bind It: Unified Layout Control and Attribute Binding in T2I Diffusion Models.
Ashkan Taghipour, Morteza Ghahremani, Mohammed Bennamoun, Aref Miri Rekavandi, Hamid Laga, Farid Boussaid.
arXiv 2024. [PDF]
CustomSketching: Sketch Concept Extraction for Sketch-based Image Synthesis and Editing.
Chufeng Xiao, Hongbo Fu.
arXiv 2024. [PDF]
Layout-to-Image Generation with Localized Descriptions using ControlNet with Cross-Attention Control.
Denis Lukovnikov, Asja Fischer.
arXiv 2024. [PDF]
MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesis.
Dewei Zhou, You Li, Fan Ma, Xiaoting Zhang, Yi Yang.
arXiv 2024. [PDF]
InstanceDiffusion: Instance-level Control for Image Generation.
Xudong Wang, Trevor Darrell, Sai Saketh Rambhatla, Rohit Girdhar, Ishan Misra.
arXiv 2024. [PDF]
Adversarial Supervision Makes Layout-to-Image Diffusion Models Thrive.
Yumeng Li, Margret Keuper, Dan Zhang, Anna Khoreva.
ICLR 2024. [PDF]
Towards Flexible, Scalable, and Adaptive Multi-Modal Conditioned Face Synthesis.
Jingjing Ren, Cheng Xu, Haoyu Chen, Xinran Qin, Chongyi Li, Lei Zhu.
arXiv 2023. [PDF]
SCEdit: Efficient and Controllable Image Diffusion Generation via Skip Connection Editing.
Zeyinzi Jiang, Chaojie Mao, Yulin Pan, Zhen Han, Jingfeng Zhang.
arXiv 2023. [PDF]
Local Conditional Controlling for Text-to-Image Diffusion Models.
Yibo Zhao, Liang Peng, Yang Yang, Zekai Luo, Hengjia Li, Yao Chen, Wei Zhao, qinglin lu, Boxi Wu, Wei Liu.
arXiv 2023. [PDF]
FreeControl: Training-Free Spatial Control of Any Text-to-Image Diffusion Model with Any Condition.
Sicheng Mo, Fangzhou Mu, Kuan Heng Lin, Yanli Liu, Bochen Guan, Yin Li, Bolei Zhou.
arXiv 2023. [PDF]
InteractDiffusion: Interaction Control in Text-to-Image Diffusion Models.
Jiun Tian Hoe, Xudong Jiang, Chee Seng Chan, Yap-Peng Tan, Weipeng Hu.
arXiv 2023. [PDF]
DemoCaricature: Democratising Caricature Generation with a Rough Sketch.
Dar-Yen Chen, Subhadeep Koley, Aneeshan Sain, Pinaki Nath Chowdhury, Tao Xiang, Ayan Kumar Bhunia, Yi-Zhe Song.
arXiv 2023. [PDF]
LOOSE CONTROL: Lifting ControlNet for Generalized Depth Conditioning.
Shariq Farooq Bhat, Niloy J. Mitra, Peter Wonka.
arXiv 2023. [PDF]
Layered Rendering Diffusion Model for Zero-Shot Guided Image Synthesis.
Zipeng Qi, Guoxi Huang, Zebin Huang, Qin Guo, Jinwen Chen, Junyu Han, Jian Wang, Gang Zhang, Lufei Liu, Errui Ding, Jingdong Wang.
arXiv 2023. [PDF]
AnyLens: A Generative Diffusion Model with Any Rendering Lens.
Andrey Voynov, Amir Hertz, Moab Arar, Shlomi Fruchter, Daniel Cohen-Or.
arXiv 2023. [PDF]
LoCo: Locally Constrained Training-Free Layout-to-Image Synthesis.
Peiang Zhao, Han Li, Ruiyang Jin, S. Kevin Zhou.
arXiv 2023. [PDF]
An Image is Worth Multiple Words- Multi-attribute Inversion for Constrained Text-to-Image Synthesis.
Aishwarya Agarwal, Srikrishna Karanam, Tripti Shukla, Balaji Vasan Srinivasan.
arXiv 2023. [PDF]
Enhancing Object Coherence in Layout-to-Image Synthesis.
Yibin Wang, Weizhong Zhang, Jianwei Zheng, Cheng Jin.
arXiv 2023. [PDF]
R&B: REGION AND BOUNDARY AWARE ZERO-SHOT
GROUNDED TEXT-TO-IMAGE GENERATION.
Jiayu Xiao, Henglei Lv, Liang Li, Shuhui Wang, Qingming Huang.
arXiv 2023. [PDF]
HyperHuman- Hyper-Realistic Human Generation with Latent Structural Diffusion.
Xian Liu, Jian Ren, Aliaksandr Siarohin, Ivan Skorokhodov, Yanyu Li, Dahua Lin, Xihui Liu, Ziwei Liu, Sergey Tulyakov.
ICLR 2024. [PDF]
JointNet: Extending Text-to-Image Diffusion for Dense Distribution Modeling.
Jingyang Zhang, Shiwei Li, Yuanxun Lu, Tian Fang, David McKinnon, Yanghai Tsin, Long Quan, Yao Yao.
ICLR 2024. [PDF]
Dense Text-to-Image Generation with Attention Modulation.
Yunji Kim, Jiyoung Lee, Jin-Hwa Kim, Jung-Woo Ha, Jun-Yan Zhu.
ICCV 2023. [PDF]
SSMG: Spatial-Semantic Map Guided Diffusion Model for Free-form Layout-to-Image Generation.
Chengyou Jia, Minnan Luo, Zhuohang Dang, Guang Dai, Xiaojun Chang, Mengmeng Wang, Jingdong Wang.
arXiv 2023. [PDF]
Localized Text-to-Image Generation For Free via Cross Attention Control.
Yutong He, Ruslan Salakhutdinov, J. Zico Kolter.
arXiv 2023. [PDF]
Zero-shot spatial layout conditioning for text-to-image diffusion models.
Guillaume Couairon, Marlène Careil, Matthieu Cord, Stéphane Lathuilière, Jakob Verbeek.
ICCV 2023. [PDF]
Grounded Text-to-Image Synthesis with Attention Refocusing.
Quynh Phung, Songwei Ge, Jia-Bin Huang.
arXiv 2023. [PDF]
GeoDiffusion: Text-Prompted Geometric Control for Object Detection Data Generation.
Kai Chen, Enze Xie, Zhe Chen, Yibo Wang, Lanqing Hong, Zhenguo Li, Dit-Yan Yeung.
ICLR 2024. [PDF]
Conditioning Diffusion Models via Attributes and Semantic Masks for Face Generation.
Nico Giambi, Giuseppe Lisanti.
arXiv 2023. [PDF]
Late-Constraint Diffusion Guidance for Controllable Image Synthesis.
Chang Liu, Dong Liu.
arXiv 2023. [PDF]
HumanSD: A Native Skeleton-Guided Diffusion Model for Human Image Generation.
Xuan Ju, Ailing Zeng, Chenchen Zhao, Jianan Wang, Lei Zhang, Qiang Xu.
ICCV 2023. [PDF]
LayoutDiffusion: Controllable Diffusion Model for
Layout-to-image Generation.
Guangcong Zheng, Xianpan Zhou, Xuewei Li, Zhongang Qi, Ying Shan, Xi Li.
CVPR 2023. [PDF]
Freestyle Layout-to-Image Synthesis.
Han Xue, Zhiwu Huang, Qianru Sun, Li Song, Wenjun Zhang.
CVPR 2023. [PDF]
FreeDoM: Training-Free Energy-Guided Conditional Diffusion Model.
Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, Jian Zhang.
arXiv 2023. [PDF]
Modulating Pretrained Diffusion Models for Multimodal Image Synthesis.
Cusuh Ham, James Hays, Jingwan Lu, Krishna Kumar Singh, Zhifei Zhang, Tobias Hinz.
SIGGRAPH 2023. [PDF]
LayoutDiffuse: Adapting Foundational Diffusion Models for Layout-to-Image Generation.
Jiaxin Cheng, Xiao Liang, Xingjian Shi, Tong He, Tianjun Xiao, Mu Li.
arXiv 2023. [PDF]
Universal Guidance for Diffusion Models.
Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, Tom Goldstein.
CVPRW 2023. [PDF]
GLIGEN: Open-Set Grounded Text-to-Image Generation.
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, Yong Jae Lee.
CVPR 2023. [PDF]
SpaText: Spatio-Textual Representation for Controllable Image Generation.
Omri Avrahami, Thomas Hayes, Oran Gafni, Sonal Gupta, Yaniv Taigman, Devi Parikh, Dani Lischinski, Ohad Fried, Xi Yin.
CVPR 2023. [PDF]
Sketch-Guided Text-to-Image Diffusion Models.
Andrey Voynov, Kfir Aberman, Daniel Cohen-Or.
SIGGRAPH 2023. [PDF]
eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers.
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, Tero Karras, Ming-Yu Liu.
arXiv 2022. [PDF]
PEA-Diffusion: Parameter-Efficient Adapter with Knowledge Distillation in non-English Text-to-Image Generation.
Jian Ma, Chen Chen, Qingsong Xie, Haonan Lu.
arXiv 2023. [PDF]
Paragraph-to-Image Generation with Information-Enriched Diffusion Model.
Weijia Wu, Zhuang Li, Yefei He, Mike Zheng Shou, Chunhua Shen, Lele Cheng, Yan Li, Tingting Gao, Di Zhang, Zhongyuan Wang.
arXiv 2023. [PDF]
Tailored Visions: Enhancing Text-to-Image Generation with Personalized Prompt Rewriting.
Zijie Chen, Lichao Zhang, Fangsheng Weng, Lili Pan, Zhenzhong Lan.
CVPR 2024. [PDF]
Linguistic Binding in Diffusion Models- Enhancing Attribute Correspondence through Attention Map Alignment.
Royi Rassin, Eran Hirsch, Daniel Glickman, Shauli Ravfogel, Yoav Goldberg, Gal Chechik.
NeurIPS 2023. [PDF]
Expressive Text-to-Image Generation with Rich Text.
Songwei Ge, Taesung Park, Jun-Yan Zhu, Jia-Bin Huang.
ICCV 2023. [PDF]
GlueGen: Plug and Play Multi-modal Encoders for X-to-image Generation.
Can Qin, Ning Yu, Chen Xing, Shu Zhang, Zeyuan Chen, Stefano Ermon, Yun Fu, Caiming Xiong, Ran Xu.
arXiv 2023. [PDF]
Attend-and-Excite- Attention-Based Semantic Guidance for Text-to-Image Diffusion Models.
Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, Daniel Cohen-Or.
TOG 2023. [PDF]
TRAINING-FREE STRUCTURED DIFFUSION GUIDANCE FOR COMPOSITIONAL TEXT-TO-I MAGE SYNTHESIS.
Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, William Yang Wang.
arXiv 2022. [PDF]
I.
Tianqi Chen, Yongfei Liu, Zhendong Wang, Jianbo Yuan, Quanzeng You, Hongxia Yang, Mingyuan Zhou.
arXiv 2023. [PDF]
In-Context Learning Unlocked for Diffusion Models.
Zhendong Wang, Yifan Jiang, Yadong Lu, Yelong Shen, Pengcheng He, Weizhu Chen, Zhangyang Wang, Mingyuan Zhou.
arXiv 2023. [PDF]
BrainVis: Exploring the Bridge between Brain and Visual Signals via Image Reconstruction.
Honghao Fu, Zhiqi Shen, Jing Jih Chin, Hao Wang.
arXiv 2023. [PDF]
DreamDiffusion: Generating High-Quality Images from Brain EEG Signals.
Yunpeng Bai, Xintao Wang, Yan-pei Cao, Yixiao Ge, Chun Yuan, Ying Shan.
arXiv 2023. [PDF]
Natural Image Reconstruction from fMRI Based on Self-supervised Representation Learning and Latent Diffusion Model.
Pengyu Ni, Yifeng Zhang.
ACM MM 2023.
MindDiffuser: Controlled Image Reconstruction from Human Brain Activity with Semantic and Structural Diffusion.
Yizhuo Lu, Changde Du, Dianpeng Wang, Huiguang He.
ACM MM 2023. [PDF]
Natural.
Furkan Ozcelik, Rufin VanRullen.
Scientific Reports 2023. [PDF]
High-Resolution Image Reconstruction With Latent Diffusion Models From Human Brain Activity.
Yu Takagi, Shinji Nishimoto.
CVPR 2023.
Seeing Beyond the Brain: Conditional Diffusion Model with Sparse Masked Modeling for Vision Decoding.
Zijiao Chen, Jiaxin Qing, Tiange Xiang, Wan Lin Yue, Juan Helen Zhou.
CVPR 2023. [PDF]
Align, Adapt and Inject: Sound-guided
Unified Image Generation.
Yue Yang, Kaipeng Zhang, Yuying Ge, Wenqi Shao, Zeyue Xue, Yu Qiao, Ping Luo.
arXiv 2023. [PDF]
GlueGen: Plug and Play Multi-modal Encoders for X-to-image Generation.
Can Qin, Ning Yu, Chen Xing, Shu Zhang, Zeyuan Chen, Stefano Ermon, Yun Fu, Caiming Xiong, Ran Xu.
arXiv 2023. [PDF]
Typographic Text Generation with Off-the-Shelf Diffusion Model.
KhayTze Peong, Seiichi Uchida, Daichi Haraguchi.
arXiv 2024. [PDF]
Brush Your Text: Synthesize Any Scene Text on Images via Diffusion Model.
Lingjun Zhang, Xinyuan Chen, Yaohui Wang, Yue Lu, Yu Qiao.
AAAI 2024. [PDF]
UDiffText: A Unified Framework for High-quality Text Synthesis in Arbitrary Images via Character-aware Diffusion Models.
Yiming Zhao, Zhouhui Lian.
arXiv 2023. [PDF]
TextDiffuser-2: Unleashing the Power of Language Models for Text Rendering.
Jingye Chen, Yupan Huang, Tengchao Lv, Lei Cui, Qifeng Chen, Furu Wei.
arXiv 2023. [PDF]
AnyText: Multilingual Visual Text Generation And Editing.
Yuxiang Tuo, Wangmeng Xiang, Jun-Yan He, Yifeng Geng, Xuansong Xie.
arXiv 2023. [PDF]
GlyphControl: Glyph Conditional Control for Visual Text Generation.
Yukang Yang, Dongnan Gui, Yuhui Yuan, Weicong Liang, Haisong Ding, Han Hu, Kai Chen.
NeurIPS 2023. [PDF]
TextDiffuser: Diffusion Models as Text Painters.
Jingye Chen, Yupan Huang, Tengchao Lv, Lei Cui, Qifeng Chen, Furu Wei.
arXiv 2023. [PDF]
GlyphDraw: Seamlessly Rendering Text with Intricate Spatial Structures in Text-to-Image.
Jian Ma, Mingjun Zhao, Chen Chen, Ruichen Wang, Di Niu, Haonan Lu, Xiaodong Lin.
arXiv 2023. [PDF]
Character-Aware Models Improve Visual Text Rendering.
Rosanne Liu, Dan Garrette, Chitwan Saharia, William Chan, Adam Roberts, Sharan Narang, Irina Blok, RJ Mical, Mohammad Norouzi, Noah Constant.
ACL 2022. [PDF]
Cocktail: Mixing Multi-Modality Controls for Text-Conditional Image Generation.
Minghui Hu, Jianbin Zheng, Daqing Liu, Chuanxia Zheng, Chaoyue Wang, Dacheng Tao, Tat-Jen Cham.
NeurIPS 2023. [PDF]
FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attention.
Guangxuan Xiao, Tianwei Yin, William T. Freeman, Frédo Durand, Song Han.
arXiv 2023. [PDF]
SVDiff- Compact Parameter Space for Diffusion Fine-Tuning.
Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, Feng Yang.
ICCV 2023. [PDF]
Composer: Creative and controllable image synthesis with composable conditions.
Lianghua Huang, Di Chen, Yu Liu, Yujun Shen, Deli Zhao, Jingren Zhou.
ICML 2023. [PDF]
Continual Diffusion with STAMINA: STack-And-Mask INcremental Adapters.
James Seale Smith, Yen-Chang Hsu, Zsolt Kira, Yilin Shen, Hongxia Jin.
arXiv 2023. [PDF]
Create Your World: Lifelong Text-to-Image
Diffusion.
Gan Sun, Wenqi Liang, Jiahua Dong, Jun Li, Zhengming Ding, Yang Cong.
arXiv 2023. [PDF]
Continual Diffusion- Continual Customization of Text-to-Image Diffusion with C-LoRA.
James Seale Smith, Yen-Chang Hsu, Lingyu Zhang, Ting Hua, Zsolt Kira, Yilin Shen, Hongxia Jin.
arXiv 2023. [PDF]
Orthogonal Adaptation for Modular Customization of Diffusion Models.
Ryan Po, Guandao Yang, Kfir Aberman, Gordon Wetzstein.
arXiv 2023. [PDF]
ZipLoRA- Any Subject in Any Style by Effectively Merging LoRAs.
Viraj Shah, Nataniel Ruiz, Forrester Cole, Erika Lu, Svetlana Lazebnik, Yuanzhen Li, Varun Jampani.
arXiv 2023. [PDF]
Mix-of-Show- Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models.
Yuchao Gu, Xintao Wang, Jay Zhangjie Wu, Yujun Shi, Yunpeng Chen, Zihan Fan, Wuyou Xiao, Rui Zhao, Shuning Chang, Weijia Wu, Yixiao Ge, Ying Shan, Mike Zheng Shou.
NeurIPS 2023. [PDF]
Cones- Concept Neurons in Diffusion Models for Customized Generation.
Zhiheng Liu, Ruili Feng, Kai Zhu, Yifei Zhang, Kecheng Zheng, Yu Liu, Deli Zhao, Jingren Zhou, Yang Cao.
ICML 2023. [PDF]
Multi-Concept Customization of Text-to-Image Diffusion.
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, Jun-Yan Zhu.
CVPR 2023. [PDF]
Cones 2- Customizable Image Synthesis with Multiple Subjects .
Zhiheng Liu, Yifei Zhang, Yujun Shen, Kecheng Zheng, Kai Zhu, Ruili Feng, Yu Liu, Deli Zhao, Jingren Zhou, Yang Cao.
arXiv 2023. [PDF]
Mix-of-Show- Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models.
Yuchao Gu, Xintao Wang, Jay Zhangjie Wu, Yujun Shi, Yunpeng Chen, Zihan Fan, Wuyou Xiao, Rui Zhao, Shuning Chang, Weijia Wu, Yixiao Ge, Ying Shan, Mike Zheng Shou.
NeurIPS 2023. [PDF]
Concept-centric Personalization with Large-scale Diffusion Priors.
Pu Cao, Lu Yang, Feng Zhou, Tianrui Huang, Qing Song.
arXiv 2023. [PDF]
High-fidelity Person-centric Subject-to-Image Synthesis.
Yibin Wang, Weizhong Zhang, Jianwei Zheng, Cheng Jin.
arXiv 2023. [PDF]
Decompose and Realign: Tackling Condition Misalignment in Text-to-Image Diffusion Models.
Luozhou Wang, Guibao Shen, Wenhang Ge, Guangyong Chen, Yijun Li, Ying-cong Chen.
arXiv 2023. [PDF]
Image Anything: Towards Reasoning-coherent and Training-free Multi-modal Image Generation.
Yuanhuiyi Lyu, Xu Zheng, Lin Wang.
arXiv 2024. [PDF]
UNIMO-G: Unified Image Generation through Multimodal Conditional Diffusion.
Wei Li, Xue Xu, Jiachen Liu, Xinyan Xiao.
arXiv 2024. [PDF]
Instruct-Imagen: Image Generation with Multi-modal Instruction.
Hexiang Hu, Kelvin C. K. Chan, Yu-Chuan Su, Wenhu Chen, Yandong Li, Kihyuk Sohn, Yang Zhao, Xue Ben, Boqing Gong, William Cohen, Ming-Wei Chang, Xuhui Jia.
arXiv 2024. [PDF]
Generative Multimodal Models are In-Context Learners.
Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, Xinlong Wang.
arXiv 2023. [PDF]
DiffBlender: Scalable and Composable Multimodal Text-to-Image Diffusion Models.
Sungnyun Kim, Junsoo Lee, Kibeom Hong, Daesik Kim, Namhyuk Ahn.
arXiv 2023. [PDF]
Towards Accurate Guided Diffusion Sampling through Symplectic Adjoint Method.
Jiachun Pan, Hanshu Yan, Jun Hao Liew, Jiashi Feng, Vincent Y. F. Tan.
arXiv 2023. [PDF]
FreeDoM: Training-Free Energy-Guided Conditional Diffusion Model.
Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, Jian Zhang.
arXiv 2023. [PDF]
Universal Guidance for Diffusion Models.
Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, Tom Goldstein.
CVPRW 2023. [PDF]