Feat/package and device compatibility

[paulasquin]

Refacto, Packaging & Apple Silicon compatibility

• Add poetry-style packaging
• Refacto code in Object Oriented Programming
• Add typing
• Add tests
• Add mps compatibility (tested on M3 Max 64Go)
• Add gradio app

To squash before merge

Solved issues

Nonsense inner thougts

In Apple Silicon, we are (were) getting nonsense from the model.generate methods

Payload

• Instruction: make the frame red
• Image:

Expected:

• Out:

If the frame of the glasses in the image were made red, the overall appearance of the scene would change significantly.The red frame would draw more attention to the glass and create a stronger contrast with the black frame.

• Res:

Obtained

• Out

Pres flash togful calledgot At commitilli split sent supports fir card projects course bunch mixture enc halery racc developed curves enjoydog memory seek Inside Wh sam closure served supports fir tripifest towardinn household finishing exact meaning ordinary treat drop whose invert Rem follow til Otherwise stal frames sequence lifted accomp entire variation government carriage uses eratrim condition Wild throne phys mutong B woods racc developed Le rename Ada laugh applying dess squ cit reference rad type refresh spr rud embedded agricult foot ax steps God close These

• Res:
  ~same as input

Fix

Latest llava weights that you can get from hugging face with git clone https://huggingface.co/liuhaotian/LLaVA-Lightning-7B-delta-v1-1 are just not working.
Solved using saved weights by tsujuifu, stored in GoogleDrive
-> A lot of time lost out of this. This is due to delta-vs-full LLava?

• Out

The image would feature a close-up view of a pair of black eyeglasses with a gold or metallic frame, placed on a gray background.The frame would be red, drawing attention to the glasses and making them the focal point of the image.

• Res

[xiaoqian-shen]

I also faced issues when trying to reproduce the results. Although no errors were displayed, the quality of the editing was not good as the paper. Could you please share the environment file so I can verify the versions of the critical packages?

[xiaoqian-shen]

I fix the problem by using your provided checkpoint in google drive. Thanks!

[paulasquin]

Hello @xiaoqian-shen

Indeed I suggest to use the models from my HuggingFace, which is from Tsu-Jui Fu's Google Drive link.
I do not have clear understanding of why original package weights aren't working.

Even if this isn't needed for you anymore, here are the package version if it can help others:
I'm sharing poetry run python -m pip freeze instead of poetry.lock file for readability

freeze.txt

[xiaoqian-shen]

Thanks for your reply! May I ask are you available to reproduce the result of MagicBrush in Table 2?

[GitHub1712]

My trained mgie_7b also not working. Was able to train and export mllm.pt and unet.pt but if running demo, ckpt has no 'emb' and my ckpt´s 'model.embed_tokens.weight' have different tensor size. So running training worked but result model not. With tsujuifu´s weights demo works.

[paulasquin]

Thanks for your reply! May I ask are you available to reproduce the result of MagicBrush in Table 2?

Hello @xiaoqian-shen
I have sometimes slight differences but I get mainly same level of quality, and a few times I got ugly results (phone and beach photos mainly)

Here are my before/after on the demo images

[lzw-lzw]

Thank you for your contribution. I wonder where can I find the ipr2ipr.pkl/tsv data in the code, that is, the summarized image-text pair, or do I need to construct it myself?