- bug fixes

- cleanup

Author: egeozsoy

.gitignore

@@ -2,13 +2,4 @@
 /synthetic_or_generation/dis_background_removal_helpers/saved_models/isnet-general-use.pth
 /evaluated_checkpoints.json
 /requirements_submission.txt
-/EGE_README.MD
-/LLaVA/slurm_config_multiview.conf
-/LLaVA/slurm_config_multiview_aug.conf
-/LLaVA/slurm_config_multiview_temporal.conf
-/LLaVA/slurm_config_multiview_temporal_curriculum.conf
-/LLaVA/slurm_config_symbolic.conf
-/LLaVA/slurm_config_symbolic_synthetic_mv.conf
-/LLaVA/slurm_config_symbolic_synthetic_removal_mv.conf
-/slurm_config.conf
-/slurm_config_generate_novel_entities.conf
+/EGE_README.MD

LLaVA/docs/Evaluation.md

@@ -43,11 +43,9 @@ scripts, and the prediction files with LLaVA v1.5. Extract to `./playground/data
 
 1. Download [`test2015`](http://images.cocodataset.org/zips/test2015.zip) and put it under `./playground/data/eval/vqav2`.
 2. Multi-GPU inference.
-
 ```Shell
 CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 bash scripts/v1_5/eval/vqav2.sh
 ```
-
 3. Submit the results to the [evaluation server](https://eval.ai/web/challenges/challenge-page/830/my-submission): `./playground/data/eval/vqav2/answers_upload`.
 
 ### GQA
@@ -56,7 +54,6 @@ CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 bash scripts/v1_5/eval/vqav2.sh
    and put under `./playground/data/eval/gqa/data`. You may need to modify `eval.py` as [this](https://gist.github.com/haotian-liu/db6eddc2a984b4cbcc8a7f26fd523187) due to the missing assets in the
    GQA v1.2 release.
 2. Multi-GPU inference.
-
 ```Shell
 CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 bash scripts/v1_5/eval/gqa.sh
 ```
@@ -66,18 +63,15 @@ CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 bash scripts/v1_5/eval/gqa.sh
 1. Download [`test.json`](https://vizwiz.cs.colorado.edu/VizWiz_final/vqa_data/Annotations.zip) and extract [`test.zip`](https://vizwiz.cs.colorado.edu/VizWiz_final/images/test.zip) to `test`. Put
    them under `./playground/data/eval/vizwiz`.
 2. Single-GPU inference.
-
 ```Shell
 CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/vizwiz.sh
 ```
-
 3. Submit the results to the [evaluation server](https://eval.ai/web/challenges/challenge-page/1911/my-submission): `./playground/data/eval/vizwiz/answers_upload`.
 
 ### ScienceQA
 
 1. Under `./playground/data/eval/scienceqa`, download `images`, `pid_splits.json`, `problems.json` from the `data/scienceqa` folder of the ScienceQA [repo](https://github.com/lupantech/ScienceQA).
 2. Single-GPU inference and evaluate.
-
 ```Shell
 CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/sqa.sh
 ```
@@ -87,7 +81,6 @@ CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/sqa.sh
 1. Download [`TextVQA_0.5.1_val.json`](https://dl.fbaipublicfiles.com/textvqa/data/TextVQA_0.5.1_val.json) and [images](https://dl.fbaipublicfiles.com/textvqa/images/train_val_images.zip) and extract
    to `./playground/data/eval/textvqa`.
 2. Single-GPU inference and evaluate.
-
 ```Shell
 CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/textvqa.sh
 ```
@@ -96,7 +89,6 @@ CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/textvqa.sh
 
 1. Download `coco` from [POPE](https://github.com/AoiDragon/POPE/tree/e3e39262c85a6a83f26cf5094022a782cb0df58d/output/coco) and put under `./playground/data/eval/pope`.
 2. Single-GPU inference and evaluate.
-
 ```Shell
 CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/pope.sh
 ```
@@ -107,7 +99,6 @@ CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/pope.sh
 2. Downloaded images to `MME_Benchmark_release_version`.
 3. put the official `eval_tool` and `MME_Benchmark_release_version` under `./playground/data/eval/MME`.
 4. Single-GPU inference and evaluate.
-
 ```Shell
 CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/mme.sh
 ```
@@ -116,43 +107,37 @@ CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/mme.sh
 
 1. Download [`mmbench_dev_20230712.tsv`](https://download.openmmlab.com/mmclassification/datasets/mmbench/mmbench_dev_20230712.tsv) and put under `./playground/data/eval/mmbench`.
 2. Single-GPU inference.
-
 ```Shell
 CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/mmbench.sh
 ```
-
 3. Submit the results to the [evaluation server](https://opencompass.org.cn/leaderboard-multimodal): `./playground/data/eval/mmbench/answers_upload/mmbench_dev_20230712`.
 
 ### MMBench-CN
 
 1. Download [`mmbench_dev_cn_20231003.tsv`](https://download.openmmlab.com/mmclassification/datasets/mmbench/mmbench_dev_cn_20231003.tsv) and put under `./playground/data/eval/mmbench`.
 2. Single-GPU inference.
-
 ```Shell
 CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/mmbench_cn.sh
 ```
-
 3. Submit the results to the evaluation server: `./playground/data/eval/mmbench/answers_upload/mmbench_dev_cn_20231003`.
 
+
 ### SEED-Bench
 
 1. Following the official [instructions](https://github.com/AILab-CVC/SEED-Bench/blob/main/DATASET.md) to download the images and the videos. Put images
    under `./playground/data/eval/seed_bench/SEED-Bench-image`.
 2. Extract the video frame in the middle from the downloaded videos, and put them under `./playground/data/eval/seed_bench/SEED-Bench-video-image`. We provide our script `extract_video_frames.py`
    modified from the official one.
 3. Multiple-GPU inference and evaluate.
-
 ```Shell
 CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 bash scripts/v1_5/eval/seed.sh
 ```
-
 4. Optionally, submit the results to the leaderboard: `./playground/data/eval/seed_bench/answers_upload` using the official jupyter notebook.
 
 ### LLaVA-Bench-in-the-Wild
 
 1. Extract contents of [`llava-bench-in-the-wild`](https://huggingface.co/datasets/liuhaotian/llava-bench-in-the-wild) to `./playground/data/eval/llava-bench-in-the-wild`.
 2. Single-GPU inference and evaluate.
-
 ```Shell
 CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/llavabench.sh
 ```
@@ -161,11 +146,9 @@ CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/llavabench.sh
 
 1. Extract [`mm-vet.zip`](https://github.com/yuweihao/MM-Vet/releases/download/v1/mm-vet.zip) to `./playground/data/eval/mmvet`.
 2. Single-GPU inference.
-
 ```Shell
 CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/mmvet.sh
 ```
-
 3. Evaluate the predictions in `./playground/data/eval/mmvet/results` using the official jupyter notebook.
 
 ## More Benchmarks
@@ -179,11 +162,9 @@ Below are awesome benchmarks for multimodal understanding from the research comm
 2. Download and extract [images](https://huggingface.co/datasets/nanyangtu/LLVisionQA-QBench/resolve/main/images_llvisionqa.tar) and put all the images directly
    under `./playground/data/eval/qbench/images_llviqionqa`.
 3. Single-GPU inference (change `dev` to `test` for evaluation on test set).
-
 ```Shell
 CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/qbench.sh dev
 ```
-
 4. Submit the results by instruction [here](https://github.com/VQAssessment/Q-Bench#option-1-submit-results): `./playground/data/eval/qbench/llvisionqa_dev_answers.jsonl`.
 
 ### Chinese-Q-Bench
@@ -194,9 +175,7 @@ CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/qbench.sh dev
 2. Download and extract [images](https://huggingface.co/datasets/nanyangtu/LLVisionQA-QBench/resolve/main/images_llvisionqa.tar) and put all the images directly
    under `./playground/data/eval/qbench/images_llviqionqa`.
 3. Single-GPU inference (change `dev` to `test` for evaluation on test set).
-
 ```Shell
 CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/qbench_zh.sh dev
 ```
-
 4. Submit the results by instruction [here](https://github.com/VQAssessment/Q-Bench#option-1-submit-results): `./playground/data/eval/qbench/llvisionqa_zh_dev_answers.jsonl`.

LLaVA/docs/LLaVA_from_LLaMA2.md

@@ -9,6 +9,7 @@ Llama 2 checkpoints, and release it to the community for the public use.
 
 You need to apply for and download the latest Llama 2 checkpoints to start your own training (apply [here](https://ai.meta.com/resources/models-and-libraries/llama-downloads/))
 
+
 ## Training
 
 Please
@@ -17,7 +18,6 @@ checkout [`pretrain.sh`](https://github.com/haotian-liu/LLaVA/blob/main/scripts/
 ## LLaVA (based on Llama 2), What is different?
 
 :volcano: How is the new LLaVA based on Llama 2 different from Llama 1? The comparisons of the training process are described:
-
 - **Pre-training**. The pre-trained base LLM is changed from Llama 1 to Llama 2
 - **Language instruction-tuning**. The previous LLaVA model starts with Vicuna, which is instruct tuned on ShareGPT data from Llama 1; The new LLaVA model starts with Llama 2 Chat, which is an
   instruct tuned checkpoint on dialogue data from Llama 2.

LLaVA/docs/LoRA.md

@@ -12,13 +12,11 @@ the base model to use. Please make sure the base model corresponds to the LoRA c
 already, follow the instructions [here](https://github.com/lm-sys/FastChat#vicuna-weights)).
 
 #### Launch a controller
-
 ```Shell
 python -m llava.serve.controller --host 0.0.0.0 --port 10000
 ```
 
 #### Launch a gradio web server.
-
 ```Shell
 python -m llava.serve.gradio_web_server --controller http://localhost:10000 --model-list-mode reload
 ```
@@ -27,7 +25,6 @@ You just launched the Gradio web interface. Now, you can open the web interface
 not launched any model worker yet. It will be automatically updated when you launch a model worker.
 
 #### Launch a model worker
-
 ```Shell
 python -m llava.serve.model_worker --host 0.0.0.0 --controller http://localhost:10000 --port 40000 --worker http://localhost:40000 --model-path liuhaotian/llava-vicuna-7b-v1.1-lcs_558k-instruct_80k_3e-lora-preview-alpha --model-base /path/to/vicuna-v1.1
 ```

LLaVA/docs/MODEL_ZOO.md

@@ -23,6 +23,7 @@ Base model: Vicuna v1.5. Training logs: [wandb](https://api.wandb.ai/links/lht/6
   LLaVA-1.5 achieves SoTA performance across 11 benchmarks.
 </p>
 
+
 ## LLaVA-v1
 
 *Note: We recommend using the most capable LLaVA-v1.5 series above for the best performance.*
@@ -33,6 +34,7 @@ Base model: Vicuna v1.5. Training logs: [wandb](https://api.wandb.ai/links/lht/6
 | LLaMA-2-13B-Chat | CLIP-L         | LCS-558K      | 1e                   | LLaVA-Instruct-80K | full_ft-1e          | 56.7             | 58.6               | 80.0                | 67.9                | [ckpt](https://huggingface.co/liuhaotian/llava-llama-2-13b-chat-lightning-preview)                                                                                                 |
 | LLaMA-2-7B-Chat  | CLIP-L         | LCS-558K      | 1e                   | LLaVA-Instruct-80K | lora-1e             | 51.2             | 58.9               | 71.6                | 62.8                | [LoRA](https://huggingface.co/liuhaotian/llava-llama-2-7b-chat-lightning-lora-preview)                                                                                             |
 
+
 ## Projector weights
 
 These are projector weights we have pretrained. You can use these projector weights for visual instruction tuning. They are just pretrained on image-text pairs and are NOT instruction-tuned, which
@@ -64,12 +66,14 @@ When using these projector weights to instruction-tune your LMM, please make sur
 | Vicuna-13B-v1.3  | CLIP-L         | Linear     | LCS-558K      | 1e                   | [projector](https://huggingface.co/liuhaotian/llava-pretrain-vicuna-13b-v1.3)                  |
 | Vicuna-7B-v1.3   | CLIP-L         | Linear     | LCS-558K      | 1e                   | [projector](https://huggingface.co/liuhaotian/llava-pretrain-vicuna-7b-v1.3)                   |
 
+
 ## Science QA Checkpoints
 
 | Base LLM        | Vision Encoder | Pretrain Data | Pretraining schedule | Finetuning Data | Finetuning schedule | Download                                                                          |
 |-----------------|----------------|---------------|----------------------|-----------------|---------------------|-----------------------------------------------------------------------------------|
 | Vicuna-13B-v1.3 | CLIP-L         | LCS-558K      | 1e                   | ScienceQA       | full_ft-12e         | [ckpt](https://huggingface.co/liuhaotian/llava-lcs558k-scienceqa-vicuna-13b-v1.3) |
 
+
 ## Legacy Models (merged weights)
 
 The model weights below are *merged* weights. You do not need to apply delta. The usage of LLaVA checkpoints should comply with the base LLM's model license.
@@ -78,6 +82,7 @@ The model weights below are *merged* weights. You do not need to apply delta. Th
 |-------------|----------------|---------------|----------------------|--------------------|---------------------|-----------------------------------------------------------------------------|
 | MPT-7B-Chat | CLIP-L         | LCS-558K      | 1e                   | LLaVA-Instruct-80K | full_ft-1e          | [preview](https://huggingface.co/liuhaotian/LLaVA-Lightning-MPT-7B-preview) |
 
+
 ## Legacy Models (delta weights)
 
 The model weights below are *delta* weights. The usage of LLaVA checkpoints should comply with the base LLM's model license: [LLaMA](https://github.com/facebookresearch/llama/blob/main/MODEL_CARD.md).

LLaVA/docs/ScienceQA.md

@@ -1,7 +1,6 @@
 ### ScienceQA
 
 #### Prepare Data
-
 1. Please see ScienceQA [repo](https://github.com/lupantech/ScienceQA) for setting up the dataset.
 2. Generate ScienceQA dataset for LLaVA conversation-style format.
 

LLaVA/docs/Windows.md

@@ -6,14 +6,12 @@ now. More functionalities on Windows is to be added soon, stay tuned.*
 ## Installation
 
 1. Clone this repository and navigate to LLaVA folder
-
 ```bash
 git clone https://github.com/haotian-liu/LLaVA.git
 cd LLaVA
 ```
 
 2. Install Package
-
 ```Shell
 conda create -n llava python=3.10 -y
 conda activate llava

LLaVA/docs/macOS.md

@@ -5,14 +5,12 @@
 ## Installation
 
 1. Clone this repository and navigate to LLaVA folder
-
 ```bash
 git clone https://github.com/haotian-liu/LLaVA.git
 cd LLaVA
 ```
 
 2. Install Package
-
 ```Shell
 conda create -n llava python=3.10 -y
 conda activate llava