1. 说明

假设我们有一个句子如下：You can now install TorchText using pip!
分词后可得：['you', 'can', 'now', 'install', 'torchtext', 'using', 'pip', '!']

pytorch_5">2. pytorch代码

python">import torchtext
from torchtext.data import get_tokenizertokenizer = get_tokenizer("basic_english")
tokens = tokenizer("You can now install TorchText using pip!")
print(tokens)

• 结果：

python">['you', 'can', 'now', 'install', 'torchtext', 'using', 'pip', '!']