主要原因是hugging face 的数据缓存比较大,jupyter执行的时候,需要加一个环境变量
方法一:在linux中指定环境变量
export HF_DATASETS_CACHE="/disk/cache/"
export HF_HOME="/disk/cache/"
export HUGGINGFACE_HUB_CACHE="/disk/cache/"
export TRANSFORMERS_CACHE="/disk/cache/"
方法二:在代码中指定环境变量
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "1"
os.environ["HF_DATASETS_CACHE"] = "/disk/cache/"
os.environ["HF_HOME"] = "/disk/cache/"
os.environ["HUGGINGFACE_HUB_CACHE"] = "/disk/cache/"
os.environ["TRANSFORMERS_CACHE"] = "/disk/cache/"