[代码实践]利用LSTM构建基于conll2003数据集的命名实体实体识别NER模型

news/2024/11/8 20:29:14/

1. conll2003数据集介绍以及数据集预处理请看下面博客

conll2003数据集下载与预处理_茫茫人海一粒沙的博客-CSDN博客

2. 取预处理过的数据集

import tensorflow as tf
from keras.models import Model
from keras.layers import Input, Embedding, LSTM, Dense, TimeDistributed
import keras as keras
from keras.callbacks import EarlyStopping, ModelCheckpoint
from keras.layers import Embedding, Conv1D, GlobalMaxPooling1D, Dense
import numpy as np
from constant.config import *def load_dataset():dataset = np.load('dataset/dataset.npz')train_X = dataset['train_X']train_y = dataset['train_y']valid_X = dataset['valid_X']valid_y = dataset['valid_y']test_X = dataset['test_X']test_y = dataset['test_y']return train_X, train_y, valid_X, valid_y, test_X, test_y

3. 创建基于lstm网络的NER模型

max_len =64def create_model():word2idx = load_dict('dataset/word2idx.json')tag2idx = load_dict('dataset/idx2Label.json')num_words = len(word2idx) + 1num_tags = len(tag2idx)# Define the modelinput_layer = Input(shape=(None,))embedding_layer = Embedding(input_dim=num_words, output_dim=60, input_length=max_len)(input_layer)lstm_layer = LSTM(units=50, return_sequences=True, dropout=0.5)(embedding_layer)output_layer = TimeDistributed(Dense(num_tags, activation="softmax"))(lstm_layer)model = Model(input_layer, output_layer)return model

4. 训练模型

def train( model,  train_X, train_y, valid_X, valid_y):# 定义保存模型的路径和文件名model_path = './dataset/ner_model.h5'# 定义早停回调函数early_stop = EarlyStopping(monitor='val_accuracy', patience=3, mode='max', verbose=1)# 定义ModelCheckpoint回调函数checkpoint = ModelCheckpoint(model_path, monitor='val_accuracy', save_best_only=True, mode='max', verbose=1)# Compile and train the modelmodel.compile(optimizer="adam", loss="categorical_crossentropy", metrics=["accuracy"])print(np.array(train_X).shape)print(np.array(train_y).shape)print(np.array(valid_X).shape)print(np.array(valid_y).shape)model.fit(train_X, train_y, batch_size=32, epochs=20, validation_data=(valid_X, valid_y), callbacks=[early_stop, checkpoint])

5. 测试模型

def test(test_X, test_y ):model = keras.models.load_model('./dataset/ner_model.h5')# 评估模型scores = model.evaluate(test_X, test_y, verbose=0)print("Test accuracy:", scores[1])

6.  保存文件以及加载文件的方法

def save_dict(dict, file_path):import json# Saving the dictionary to a filewith open(file_path, 'w') as f:json.dump(dict, f)def load_dict(path_file):import json# Loading the dictionary from the filewith open(path_file, 'r') as f:loaded_dict = json.load(f)return loaded_dict;print(loaded_dict)  # Output: {'key1': 'value1', 'key2': 'value2'}

7. main方法来训练模型与测试模型

if __name__ == '__main__':train_X, train_y, valid_X, valid_y, test_X, test_y =load_dataset()model= create_model()train(model, np.concatenate([train_X, valid_X]), np.concatenate([train_y, valid_y]),test_X, test_y)test(test_X, test_y )# predict()

执行结果. 一共应用了12个Epoch，训练集上的准确率是99.7%，测试集上准确率是98%。

577/577 [==============================] - ETA: 0s - loss: 0.0115 - accuracy: 0.9964
Epoch 10: val_accuracy did not improve from 0.98385
577/577 [==============================] - 16s 28ms/step - loss: 0.0115 - accuracy: 0.9964 - val_loss: 0.0570 - val_accuracy: 0.9835
Epoch 11/20
575/577 [============================>.] - ETA: 0s - loss: 0.0103 - accuracy: 0.9968
Epoch 11: val_accuracy did not improve from 0.98385
577/577 [==============================] - 16s 28ms/step - loss: 0.0103 - accuracy: 0.9968 - val_loss: 0.0598 - val_accuracy: 0.9836
Epoch 12/20
577/577 [==============================] - ETA: 0s - loss: 0.0095 - accuracy: 0.9970
Epoch 12: val_accuracy did not improve from 0.98385
577/577 [==============================] - 16s 28ms/step - loss: 0.0095 - accuracy: 0.9970 - val_loss: 0.0584 - val_accuracy: 0.9837
Epoch 12: early stopping
Test accuracy: 0.9838491082191467

8. 预测模型

def predict():# Example sentences to predicttest_sentences = ["John Wilson works at Apple .","I have a meeting with Peter Blackburn tomorrow.","BRUSSELS","Peter Blackburn",'EU rejects German call to boycott British lamb.','The European Commission said on Thursday it disagreed with German advice to consumers to shun British']word2idx = load_dict('dataset/word2idx.json')tag2idx = load_dict('dataset/idx2Label.json')model = keras.models.load_model('./dataset/ner_model.h5')# Convert test sentences to numerical sequencestest_sequences = [[word2idx.get(word.lower(), 0) for word in sentence.split()] for sentence in test_sentences]print('test_sequences:',test_sequences)test_sequences = tf.keras.preprocessing.sequence.pad_sequences(maxlen=max_len, sequences=test_sequences, padding='post' , value=0)# Make predictionspredictions = model.predict(test_sequences)# print(predictions)predicted_tags = tf.argmax(predictions, axis=-1)# Convert predicted tags back to labelspredicted_labels = []for tags in predicted_tags:labels = [list(tag2idx.keys())[tag] for tag in tags if tag != 0]predicted_labels.append(labels)# Print the predicted labelsfor sentence, labels in zip(test_sentences, predicted_labels):print(f"Sentence: {sentence}")print(f"Predicted Labels: {labels}\n")

执行结果

Sentence: John Wilson works at Apple
Predicted Labels: ['I-PER', 'O', 'O', 'B-ORG', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O']Sentence: I have a meeting with Peter Blackburn tomorrow.
Predicted Labels: ['O', 'O', 'O', 'O', 'O', 'I-PER', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O']Sentence: BRUSSELS
Predicted Labels: ['B-LOC', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O']Sentence: Peter Blackburn
Predicted Labels: ['I-PER', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O']Sentence: EU rejects German call to boycott British lamb.
Predicted Labels: ['B-ORG', 'O', 'B-MISC', 'O', 'O', 'O', 'B-MISC', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O']Sentence: The European Commission said on Thursday it disagreed with German advice to consumers to shun British
Predicted Labels: ['O', 'B-MISC', 'I-ORG', 'O', 'O', 'O', 'O', 'O', 'O', 'B-MISC', 'O', 'O', 'O', 'O', 'O', 'B-MISC', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O']

9. 所有代码

import tensorflow as tf
from keras.models import Model
from keras.layers import Input, Embedding, LSTM, Dense, TimeDistributed
import keras as keras
from keras.callbacks import EarlyStopping, ModelCheckpoint
from keras.layers import Embedding, Conv1D, GlobalMaxPooling1D, Dense
import numpy as np
from constant.config import *def load_dataset():dataset = np.load('dataset/dataset.npz')train_X = dataset['train_X']train_y = dataset['train_y']valid_X = dataset['valid_X']valid_y = dataset['valid_y']test_X = dataset['test_X']test_y = dataset['test_y']return train_X, train_y, valid_X, valid_y, test_X, test_ymax_len =64def create_model():word2idx = load_dict('dataset/word2idx.json')tag2idx = load_dict('dataset/idx2Label.json')num_words = len(word2idx) + 1num_tags = len(tag2idx)# Define the modelinput_layer = Input(shape=(None,))embedding_layer = Embedding(input_dim=num_words, output_dim=60, input_length=max_len)(input_layer)lstm_layer = LSTM(units=50, return_sequences=True, dropout=0.5)(embedding_layer)output_layer = TimeDistributed(Dense(num_tags, activation="softmax"))(lstm_layer)model = Model(input_layer, output_layer)return modeldef train( model,  train_X, train_y, valid_X, valid_y):# 定义保存模型的路径和文件名model_path = './dataset/ner_model.h5'# 定义早停回调函数early_stop = EarlyStopping(monitor='val_accuracy', patience=3, mode='max', verbose=1)# 定义ModelCheckpoint回调函数checkpoint = ModelCheckpoint(model_path, monitor='val_accuracy', save_best_only=True, mode='max', verbose=1)# Compile and train the modelmodel.compile(optimizer="adam", loss="categorical_crossentropy", metrics=["accuracy"])print(np.array(train_X).shape)print(np.array(train_y).shape)print(np.array(valid_X).shape)print(np.array(valid_y).shape)model.fit(train_X, train_y, batch_size=32, epochs=20, validation_data=(valid_X, valid_y), callbacks=[early_stop, checkpoint])
def save_dict(dict, file_path):import json# Saving the dictionary to a filewith open(file_path, 'w') as f:json.dump(dict, f)def load_dict(path_file):import json# Loading the dictionary from the filewith open(path_file, 'r') as f:loaded_dict = json.load(f)return loaded_dict;print(loaded_dict)  # Output: {'key1': 'value1', 'key2': 'value2'}def test(test_X, test_y ):model = keras.models.load_model('./dataset/ner_model.h5')# 评估模型scores = model.evaluate(test_X, test_y, verbose=0)print("Test accuracy:", scores[1])def predict():# Example sentences to predicttest_sentences = ["John Wilson works at Apple .","I have a meeting with Peter Blackburn tomorrow.","BRUSSELS","Peter Blackburn",'EU rejects German call to boycott British lamb.','The European Commission said on Thursday it disagreed with German advice to consumers to shun British']word2idx = load_dict('dataset/word2idx.json')tag2idx = load_dict('dataset/idx2Label.json')model = keras.models.load_model('./dataset/ner_model.h5')# Convert test sentences to numerical sequencestest_sequences = [[word2idx.get(word.lower(), 0) for word in sentence.split()] for sentence in test_sentences]print('test_sequences:',test_sequences)test_sequences = tf.keras.preprocessing.sequence.pad_sequences(maxlen=max_len, sequences=test_sequences, padding='post' , value=0)# Make predictionspredictions = model.predict(test_sequences)# print(predictions)predicted_tags = tf.argmax(predictions, axis=-1)# Convert predicted tags back to labelspredicted_labels = []for tags in predicted_tags:labels = [list(tag2idx.keys())[tag] for tag in tags if tag != 0]predicted_labels.append(labels)# Print the predicted labelsfor sentence, labels in zip(test_sentences, predicted_labels):print(f"Sentence: {sentence}")print(f"Predicted Labels: {labels}\n")if __name__ == '__main__':train_X, train_y, valid_X, valid_y, test_X, test_y =load_dataset()model= create_model()train(model, np.concatenate([train_X, valid_X]), np.concatenate([train_y, valid_y]),test_X, test_y)test(test_X, test_y )# predict()


http://www.ppmy.cn/news/703827.html

相关文章

计算机桌面保护时间,电脑处于屏幕保护或者休眠状态的时间怎么自己设置?

计算机桌面保护时间,电脑处于屏幕保护或者休眠状态的时间怎么自己设置?

1.打开笔记本2113电脑，点击打开开始菜单-----选择控制面板或者5261打开我的电脑，4102选择点击控制1653面板2.进入到控制面板页面，我们点击选择——硬件和声音 并且点击进入硬件和声音3.点击进入硬件和声音后，你会看到下面的页面—…
阅读更多...
如何设置电脑的息屏时间

如何设置电脑的息屏时间

阅读更多...
Win10电脑每2分钟频繁锁屏，设置锁屏时间也无效解决办法

Win10电脑每2分钟频繁锁屏，设置锁屏时间也无效解决办法

我的win10系统的电脑不管你锁屏时间设置为多少，两分钟自动就会进入休眠，一遍一遍的解锁真的很麻烦，费时费力。在网上查了好久，终于找到了解决方法，问题的根源是在Windows电源选项的一个隐藏设置项“无人参与系统睡眠超…
阅读更多...
diffusion model

diffusion model

(正课)Diffusion Model 原理剖析 (1_4) (optional)_哔哩哔哩_bilibili(正课)Diffusion Model 原理剖析 (1_4) (optional)是【授权】李宏毅2023春机器学习课程的第42集视频，该合集共计64集，视频收藏或关注UP主，及时了解更多相关视频内容。http…
阅读更多...
Win7联想电脑休眠后唤醒不了原因和解决方法

Win7联想电脑休眠后唤醒不了原因和解决方法

2019独角兽企业重金招聘Python工程师标准>>> 一、电脑BIOS不支持造成的。 查看方法： 1、点击运行。 2、输入cmd打开dos命令窗口。 3、输入 powercfg -a。 4、查看是否支持休眠后唤醒。 如图就不支持休眠以后唤醒。就需要升级BIOS解决。 二、没有设置鼠标…
阅读更多...
win10计算机休眠设置在哪里,win10休眠时间设置在哪里_win10更改休眠时间的方法...

win10计算机休眠设置在哪里,win10休眠时间设置在哪里_win10更改休眠时间的方法...

win10休眠时间休眠在哪里设置？休眠状态能够是电脑处于误操作的状态，又避免了关机之后还要重新开机的麻烦，但很多使用win10精简版系统用户不知道哪里可以更改休眠时间，没关系，今天小编就来给大家分享win10更改休眠时间的…
阅读更多...
win10电脑进入睡眠时间过快，电源选项的设置无效

win10电脑进入睡眠时间过快，电源选项的设置无效

我的笔记本电脑使用的是Windows10操作系统，但是最近发现了一个很奇怪的问题：电脑无操作后自动进入睡眠时间过快。 不管如何设置，电脑总会在无任何操作大约2分多钟后自动进入睡眠状态同时关闭显示器。设置了10分钟后关闭显示器，30分…
阅读更多...
设置Ubuntu休眠时间

设置Ubuntu休眠时间

摘要：主要简介了如何设置Ubuntu18.04的休眠时间 原因：在学习Python过程中，在对照参考书编写代码时，看书时间稍长后，Ubuntu自动休眠，需要重新输入密码，麻烦，解决该问题方法如下。&…
阅读更多...
最新文章

Copyright @ 2022~2023