DLCV Final Project ( STAR )

Setup & Run lamma-1.3B(ours)

setup env

git clone https://github.com/DLCV-Fall-2023/DLCV-Fall-2023-Final-1-r-b.git
conda create -n flipped-vqa python=3.8
conda activate flipped-vqa
cd DLCV-Fall-2023-Final-1-r-b/Flipped-VQA
sh setup.sh
pip3 install gdown timm torchinfo sentencepiece accelerate
bash data_download.sh

train

bash train.sh

model 會存在 checkpoint/star/ 裡

inference

bash inference.sh

.json 會存在 output_dir/ 裡

Setup & Run lamma-7B(leaderboard)

setup env

cd DLCV-Fall-2023-Final-1-r-b/Flipped-VQA-7B
sh setup.sh
pip3 install wget
sudo apt-get install unzip
bash download.sh

inference

python3 inference.py --resume ./star.pth
python3 trans.py

.json 會存在同一位置

How to run your code?

• TODO: Please provide the scripts for TAs to reproduce your results, including training and inference. For example,

bash train.sh <Path to videos folder> <annotation file> [additional path(s)...]
bash inference.sh <Path to videos folder> <annotation file>

• For the training code, feel free to add extra paths to your training script if you want to use additional data such as bounding boxes or hyper-graphs.
• You can modify .gitignore file to avoid uploading your data

Usage

To start working on this final project, you should clone this repository into your local machine by the following command:

git clone https://github.com/ntudlcv/DLCV-Fall-2023-Final-1-<team name>.git

Note that you should replace <team_name> with your own team name.

For more details, please click this link to view the slides of Final Project - STAR Benchmark. Note that video and introduction pdf files for final project can be accessed in your NTU COOL.

Dataset Overview

The following files are required for use in your training process.

Question, Multiple Choice Answers and Situation Graphs

• Questions and Answers (.json) : Train Val Test
• Split file (Optional): Train/Val/Test Split File (.json)

Video Data

• raw video data: recommend: Data(scaled up to 480p)

If you want to use additional data such as bounding boxes, hyper-graphs, ..., please refer to the following links for more information.

• Star Official Website
• GitHub

Submission Rules

Deadline

112/12/28 (Thur.) 23:59 (GMT+8)

Q&A

If you have any problems related to Final Project, you may

• Use TA hours
• Contact TAs by e-mail (ntudlcv@gmail.com)
• Post your question under Final Project FAQ section in NTU Cool Discussion