参考教程链接：注意查看 评论区相关内容
Q & A 汇总
赛事链接：天池 Better Synth 多模态大模型数据合成挑战赛
算力资源 挺烧钱的。。
- 之前的教程有些问题没考虑到，排坑花了 200 了。。
跑通 baseline 至少需要 120G 内存以上的 A10，整个流程可能需要 5 小时。
目标： 使用已有大模型和技术，在限定的计算资源下合成高质量训练数据，以增强多模态大模型的图像理解能力。
——————————
如何借助已有的强大的大模型，在有限的计算资源下为新的模型训练流程 高效合成优质的训练数据
探究合成数据对于多模态大模型训练的影响，
高效的数据合成方法与策略
多模态大模型在图片理解任务上的能力，核心任务是在给定的种子数据集的基础上，通过高效的数据合成方法与模型生成出更优的数据，并在给定计算量的约束下，实现对图像理解多模态大模型的高效训练
一站式大模型数据处理系统 Data-Juicer，提供了大量系统化、可复用的数据处理与生成算子和工具
线上赛评测提交截止时间为 2024 年 8 月 23 日18:00
10:00-22:00 每小时更新排行榜
跑通最简单的 baseline
了解竞赛核心考察点和难点
精读 baseline 并 了解进阶思路
1）这是一个什么场景下的赛题；
2）这个赛题要解决什么问题。
————————
比赛内容：
Better Synth 是一项以数据为中心的挑战赛，考察如何合成与清洗图文数据以在多模态大模型上取得更优的图片理解能力。
合成与清洗图文数据 【预训练（模态间对齐）阶段】 ——> 图片理解
本次比赛基于 Mini-Gemini 模型进行训练，只关注于预训练（模态间对齐）阶段的数据合成与清洗，指令微调阶段为固定数据集。
为了选手更高效地迭代数据合成方案，本次比赛选用 MGM-2B 规模的模型作为比赛模型。
主办方提供候选种子数据集，要求参赛者基于种子数据集进行数据合成与清洗，产出一份基于种子数据集的更高质量、更多样性的数据集，并在给定计算约束下进行训练。
本次比赛基于 Mini-Gemini 模型进行训练，只关注于预训练（模态间对齐）阶段的数据合成与清洗，指令微调阶段为固定数据集。
下载相关包 和 资源
创建实例：
- 用了免费试用资源的 CPU 资源，使用比赛镜像，扩容内存到 150 G。
- 感觉关于魔塔的部分可以直接跳过，在阿里云控制台创建实例就行。
——————————————
在阿里云 人工智能平台PAI 创建实例，如何进入平台页面可参考 链接
创建实例进行相关设置时要用：
镜像链接：dsw-registry-vpc.cn-hangzhou.cr.aliyuncs.com/pai-training-algorithm/data-juicer-better-synth:0.0.1
打开实例后，打开终端，按步骤运行相应指令即可。
1、下载赛事所需文件包
- 包括：baseline、必要的开发者套件、基础模型 和相关数据集等
git clone https://www.modelscope.cn/datasets/Datawhale/better_synth_challenge_baseline.git
2、下载多模态 baseline 模型及数据集 & 相关依赖软件
安装必要工具
apt update
apt install axel zip file
pip install modelscope
下载模型和相关数据集
大概需要 50 分钟
bash better_synth_challenge_baseline/download.sh
3、下载 BLIP 图片描述模型
约 20 分钟
python better_synth_challenge_baseline/download_blip.py
——————————
以上每个实例仅运行一次
训练和评估
以下每次修改环境后都要重新执行
数据合成和模型训练环节
修改实例配置 为 GPU，其相应内存规格为 188G
终端执行以下命令
ls 命令 确认当前位置
大概需要 1 小时
使用 data-jucier 来进行数据处理
dj-process --config solution/image_captioning.yaml
————————————————
重要的修改：！！！
终端：务必先修改上述的 train_mgm_2b_stage_1.sh 相关内容，不然 CUDA 会爆 memory
ls 命令 确认当前位置
cd toolkit
git clone https://github.com/modelscope/data-juicer.git
bash train_mgm_2b_stage_1.sh ### 大概需要 3 小时
16:09 - 22:13
没有推理结果
——> 修改上面的 batch-size
————————————————
打包构建符合提交要求的 zip 文件
ls 命令 确认当前位置
以下可以一起复制执行
cd /mnt/workspace/better_synth_challenge_baseline/submit cp -r /mnt/workspace/better_synth_challenge_baseline/solution .cp -r /mnt/workspace/better_synth_challenge_baseline/output/eval_results output/cp -r /mnt/workspace/better_synth_challenge_baseline/output/train.sh output/cp /mnt/workspace/better_synth_challenge_baseline/output/training_dirs/MGM-2B-Finetune-default/finetuning.log output/training_dirs/MGM-2B-Finetune-image_recaption/cp /mnt/workspace/better_synth_challenge_baseline/output/training_dirs/MGM-2B-Pretrain-default/pretrain.log output/training_dirs/MGM-2B-Pretrain-image_recaption/zip -r submit.zip solution output
zip 文件下载后，检查下文件目录是否和要求一致。
提交评测，评测成功，配置完成。
————————————————————
cp -r 递归复制，会覆盖
cp 不覆盖
——————————
- 在 better_synth_challenge_baseline/solution/readme 记录 处理要点。
删掉 某个文件夹 指令
rm -rf /mnt/workspace/better_synth_challenge_baseline/submit/output/eval_results
——————
终端命令：清除 GPU 缓存
nvidia-smi --gpu-reset
torch.cuda.empty_cache()
Traceback (most recent call last):File "/mnt/workspace/better_synth_challenge_baseline/toolkit/training/mgm/train/train_mem.py", line 4, in <module>train(attn_implementation="flash_attention_2")File "/dj_synth_challenge/toolkit/training/mgm/train/train.py", line 1344, in traintrainer.train()File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 1624, in trainreturn inner_training_loop(File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 1961, in _inner_training_looptr_loss_step = self.training_step(model, inputs)File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 2902, in training_steploss = self.compute_loss(model, inputs)File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 2925, in compute_lossoutputs = model(**inputs)File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1501, in _call_implreturn forward_call(*args, **kwargs)File "/usr/local/lib/python3.10/dist-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fnret_val = func(*args, **kwargs)File "/usr/local/lib/python3.10/dist-packages/deepspeed/runtime/engine.py", line 1807, in forwardloss = self.module(*inputs, **kwargs)File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1501, in _call_implreturn forward_call(*args, **kwargs)File "/dj_synth_challenge/toolkit/training/mgm/model/language_model/mgm_gemma.py", line 97, in forwardreturn super().forward(File "/usr/local/lib/python3.10/dist-packages/transformers/models/gemma/modeling_gemma.py", line 1094, in forwardloss = loss_fct(shift_logits, shift_labels)File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1501, in _call_implreturn forward_call(*args, **kwargs)File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/loss.py", line 1174, in forwardreturn F.cross_entropy(input, target, weight=self.weight,File "/usr/local/lib/python3.10/dist-packages/torch/nn/functional.py", line 3029, in cross_entropyreturn torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 3.91 GiB (GPU 0; 22.20 GiB total capacity; 18.73 GiB already allocated; 1.14 GiB free; 19.38 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF0%| | 0/93 [00:03<?, ?it/s]
[2024-08-11 08:24:54,609] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 1248
[2024-08-11 08:24:54,609] [ERROR] [launch.py:321:sigkill_handler] ['/usr/bin/python3', '-u', '/mnt/workspace/better_synth_challenge_baseline/toolkit/training/mgm/train/train_mem.py', '--local_rank=0', '--deepspeed', '/mnt/workspace/better_synth_challenge_baseline/toolkit/training/scripts/zero2_offload.json', '--model_name_or_path', '/mnt/workspace/better_synth_challenge_baseline/toolkit/training/model_zoo/LLM/gemma/gemma-2b-it', '--version', 'gemma', '--data_path', '/mnt/workspace/better_synth_challenge_baseline/toolkit/training/data/finetuning_stage_1_12k/mgm_instruction_stage_1_12k.json', '--image_folder', '/mnt/workspace/better_synth_challenge_baseline/toolkit/training/data/finetuning_stage_1_12k', '--vision_tower', '/mnt/workspace/better_synth_challenge_baseline/toolkit/training/model_zoo/OpenAI/clip-vit-large-patch14-336', '--vision_tower_aux', '/mnt/workspace/better_synth_challenge_baseline/toolkit/training/model_zoo/OpenAI/openclip-convnext-large-d-320-laion2B-s29B-b131K-ft-soup', '--pretrain_mm_mlp_adapter', '/mnt/workspace/better_synth_challenge_baseline/toolkit/../output/training_dirs/MGM-2B-Pretrain-default/mm_projector.bin', '--mm_projector_type', 'mlp2x_gelu', '--mm_vision_select_layer', '-2', '--mm_use_im_start_end', 'False', '--mm_use_im_patch_token', 'False', '--image_aspect_ratio', 'pad', '--group_by_modality_length', 'True', '--image_size_aux', '768', '--bf16', 'True', '--output_dir', '/mnt/workspace/better_synth_challenge_baseline/toolkit/../output/training_dirs/MGM-2B-Finetune-default', '--num_train_epochs', '1', '--per_device_train_batch_size', '4', '--per_device_eval_batch_size', '4', '--gradient_accumulation_steps', '32', '--evaluation_strategy', 'no', '--save_strategy', 'steps', '--save_steps', '100', '--save_total_limit', '1', '--learning_rate', '2e-5', '--weight_decay', '0.', '--warmup_ratio', '0.03', '--lr_scheduler_type', 'cosine', '--logging_steps', '1', '--tf32', 'True', '--model_max_length', '2048', '--gradient_checkpointing', 'True', '--dataloader_num_workers', '4', '--lazy_preprocess', 'True', '--report_to', 'none'] exits with return code = 1
Infer on TextVQA...
/usr/local/lib/python3.10/dist-packages/transformers/deepspeed.py:23: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrationswarnings.warn(
[2024-08-11 08:24:56,958] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
Traceback (most recent call last):File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_mainreturn _run_code(code, main_globals, None,File "/usr/lib/python3.10/runpy.py", line 86, in _run_codeexec(code, run_globals)File "/dj_synth_challenge/toolkit/training/mgm/eval/model_vqa_loader.py", line 191, in <module>eval_model(args)File "/dj_synth_challenge/toolkit/training/mgm/eval/model_vqa_loader.py", line 121, in eval_modeltokenizer, model, image_processor, context_len = load_pretrained_model(model_path, args.model_base, model_name, load_8bit=args.load_8bit)File "/dj_synth_challenge/toolkit/training/mgm/model/builder.py", line 72, in load_pretrained_modeltokenizer = AutoTokenizer.from_pretrained(model_path)File "/usr/local/lib/python3.10/dist-packages/transformers/models/auto/tokenization_auto.py", line 782, in from_pretrainedconfig = AutoConfig.from_pretrained(File "/usr/local/lib/python3.10/dist-packages/transformers/models/auto/configuration_auto.py", line 1111, in from_pretrainedconfig_dict, unused_kwargs = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)File "/usr/local/lib/python3.10/dist-packages/transformers/configuration_utils.py", line 633, in get_config_dictconfig_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs)File "/usr/local/lib/python3.10/dist-packages/transformers/configuration_utils.py", line 688, in _get_config_dictresolved_config_file = cached_file(File "/usr/local/lib/python3.10/dist-packages/transformers/utils/hub.py", line 369, in cached_fileraise EnvironmentError(
OSError: /mnt/workspace/better_synth_challenge_baseline/toolkit/eval/../../output/training_dirs/MGM-2B-Finetune-default does not appear to have a file named config.json. Checkout 'https://huggingface.co//mnt/workspace/better_synth_challenge_baseline/toolkit/eval/../../output/training_dirs/MGM-2B-Finetune-default/None' for available files.
/mnt/workspace/better_synth_challenge_baseline/toolkit/eval/textvqa.sh: line 28: /mnt/workspace/better_synth_challenge_baseline/toolkit/eval/../../output/eval_results/MGM-2B-Finetune-default/textvqa//bm1.jsonl: No such file or directory
/mnt/workspace/better_synth_challenge_baseline/toolkit/eval/textvqa.sh: line 32: /mnt/workspace/better_synth_challenge_baseline/toolkit/eval/../../output/eval_results/MGM-2B-Finetune-default/textvqa//bm1.jsonl: No such file or directory
Infer on MMBench...
/usr/local/lib/python3.10/dist-packages/transformers/deepspeed.py:23: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrationswarnings.warn(
[2024-08-11 08:25:01,233] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
Traceback (most recent call last):File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_mainreturn _run_code(code, main_globals, None,File "/usr/lib/python3.10/runpy.py", line 86, in _run_codeexec(code, run_globals)File "/dj_synth_challenge/toolkit/training/mgm/eval/model_vqa_mmbench.py", line 216, in <module>eval_model(args)File "/dj_synth_challenge/toolkit/training/mgm/eval/model_vqa_mmbench.py", line 59, in eval_modeltokenizer, model, image_processor, context_len = load_pretrained_model(model_path, args.model_base, model_name)File "/dj_synth_challenge/toolkit/training/mgm/model/builder.py", line 72, in load_pretrained_modeltokenizer = AutoTokenizer.from_pretrained(model_path)File "/usr/local/lib/python3.10/dist-packages/transformers/models/auto/tokenization_auto.py", line 782, in from_pretrainedconfig = AutoConfig.from_pretrained(File "/usr/local/lib/python3.10/dist-packages/transformers/models/auto/configuration_auto.py", line 1111, in from_pretrainedconfig_dict, unused_kwargs = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)File "/usr/local/lib/python3.10/dist-packages/transformers/configuration_utils.py", line 633, in get_config_dictconfig_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs)File "/usr/local/lib/python3.10/dist-packages/transformers/configuration_utils.py", line 688, in _get_config_dictresolved_config_file = cached_file(File "/usr/local/lib/python3.10/dist-packages/transformers/utils/hub.py", line 369, in cached_fileraise EnvironmentError(
OSError: /mnt/workspace/better_synth_challenge_baseline/toolkit/eval/../../output/training_dirs/MGM-2B-Finetune-default does not appear to have a file named config.json. Checkout 'https://huggingface.co//mnt/workspace/better_synth_challenge_baseline/toolkit/eval/../../output/training_dirs/MGM-2B-Finetune-default/None' for available files.
Traceback (most recent call last):File "/mnt/workspace/better_synth_challenge_baseline/toolkit/eval/../training/scripts/convert_mmbench_for_submission.py", line 23, in <module>for pred in open(os.path.join(args.result_dir, f"{args.experiment}.jsonl")):
FileNotFoundError: [Errno 2] No such file or directory: '/mnt/workspace/better_synth_challenge_baseline/toolkit/eval/../../output/eval_results/MGM-2B-Finetune-default/mmbench/answers/mmbench_dev_20230712/MGM-2B-Finetune-default.jsonl'
Training and Inference done.
Training checkpoints are stored in output/training_dirs/MGM-2B-Finetune-default.
Inference results are stored in output/eval_results/MGM-2B-Finetune-default.
确认当前所在位置
pwd 或 ls
