大语言模型分词器

代码

python">import transformers
import tokenizersfrom transformers import AutoTokenizertokenizers = AutoTokenizer.from_pretrained("data/bert-base-uncased")text = "This is a test sentence."tokens = tokenizers.encode(text)
token_count = len(tokens)print("Tokens:", tokens)
print("len of Tokens:", len(tokens))

直接运行会报错

需要先导入包

pip install transformers tokenizers

再次运行
执行结果如下