自然语言处理|自然语言处理（七）（AG_NEWS新闻分类任务（TORCHTEXT））自然语言处理|nlp|分类|深度学

自然语言处理笔记总目录关于新闻主题分类任务：以一段新闻报道中的文本描述内容为输入，使用模型帮助我们判断它最有可能属于哪一种类型的新闻，这是典型的文本分类问题,，我们这里假定每种类型是互斥的，即文本描述有且只有一种类型
本案例取自Pytorch官网的：TEXT CLASSIFICATION WITH THE TORCHTEXT LIBRARY，在此基础上增加了完整的注释以及通俗的讲解
本案例分为以下九个步骤
Step 1：Access to the raw dataset iterators AG_NEWS数据集介绍：

AG_NEWS：新闻语料库，包含4个大类新闻：World、Sports、Business、Sci/Tec。

AG_NEWS共包含120000条训练样本集（train.csv)， 7600测试样本数据集(test.csv)。每个类别分别拥有 30000 个训练样本及 1900 个测试样本。

import torch from torchtext.datasets import AG_NEWS train_iter = AG_NEWS(split='train')

返回的是一个训练集的迭代器，通过以下方法可以查看训练集的内容：

next(train_iter) >>> (3, "Wall St. Bears Claw Back Into the Black (Reuters) Reuters - Short-sellers, Wall Street's dwindling\\band of ultra-cynics, are seeing green again.")next(train_iter) >>> (3, 'Carlyle Looks Toward Commercial Aerospace (Reuters) Reuters - Private investment firm Carlyle Group,\\which has a reputation for making well-timed and occasionally\\controversial plays in the defense industry, has quietly placed\\its bets on another part of the market.')

Step 2：Prepare data processing pipelines 在训练之前，首先我们要处理新闻数据，对文本进行分词，构建词汇表vocab
使用get_tokenizer进行分词，同时build_vocab_from_iterator提供了使用迭代器构建词汇表的方法

from torchtext.data.utils import get_tokenizer from torchtext.vocab import build_vocab_from_iteratortokenizer = get_tokenizer('basic_english') # 基本的英文分词器 train_iter = AG_NEWS(split='train') # 训练数据迭代器# 分词生成器 def yield_tokens(data_iter): for _, text in data_iter: yield tokenizer(text)# 构建词汇表 vocab = build_vocab_from_iterator(yield_tokens(train_iter), specials=[""]) # 设置默认索引，当某个单词不在词汇表中，则返回0 vocab.set_default_index(vocab[""])

vocab(['here', 'is', 'an', 'example']) >>> [475, 21, 30, 5286] print(vocab(["haha", "hehe", "xixi"])) >>> [0, 0, 0]

接下来使用分词器以及词汇表构建Pipeline

text_pipeline = lambda x: vocab(tokenizer(x)) label_pipeline = lambda x: int(x) - 1

text_pipeline('here is an example') >>> [475, 21, 30, 5286] label_pipeline('10') >>> 9

Step 3：Generate data batch and iterator

from torch.utils.data import DataLoader # 使用GPU device = torch.device("cuda" if torch.cuda.is_available() else "cpu")# 定义collate_batch函数，在DataLoader中会使用，对传入的样本数据进行批量处理 def collate_batch(batch): # 存放label以及text的列表，offses存放每条text的偏移量 label_list, text_list, offsets = [], [], [0] for (_label, _text) in batch: label_list.append(label_pipeline(_label)) processed_text = torch.tensor(text_pipeline(_text), dtype=torch.int64) text_list.append(processed_text) # 将每一条数据的长度放入offsets列表当中 offsets.append(processed_text.size(0)) label_list = torch.tensor(label_list, dtype=torch.int64) # 计算出每一条text的偏移量 offsets = torch.tensor(offsets[:-1]).cumsum(dim=0) text_list = torch.cat(text_list) return label_list.to(device), text_list.to(device), offsets.to(device)train_iter = AG_NEWS(split='train') dataloader = DataLoader(train_iter, batch_size=8, shuffle=False, collate_fn=collate_batch)

cumsum()用于计算一个数组各行的累加值，示例如下：

>>>a = [1, 2, 3, 4, 5, 6, 7] >>>cumsum(a) array([1, 3, 6, 10, 15, 21, 28])

Step 4：Define the model 定义神经网络模型：由EmbeddingBag、隐藏层和全连接层组成

自然语言处理|自然语言处理（七）（AG_NEWS新闻分类任务（TORCHTEXT））

文章图片

from torch import nnclass TextClassificationModel(nn.Module):def __init__(self, vocab_size, embed_dim, num_class): super(TextClassificationModel, self).__init__() self.embedding = nn.EmbeddingBag(vocab_size, embed_dim, sparse=True) self.fc = nn.Linear(embed_dim, num_class) self.init_weights()def init_weights(self): initrange = 0.5 self.embedding.weight.data.uniform_(-initrange, initrange) self.fc.weight.data.uniform_(-initrange, initrange) self.fc.bias.data.zero_()def forward(self, text, offsets): embedded = self.embedding(text, offsets) return self.fc(embedded)

Step 5：Initiate an instance AG_NEWS 数据集有四个标签，因此类的数量是四个

1 : World 2 : Sports 3 : Business 4 : Sci/Tec

实例一个模型

train_iter = AG_NEWS(split='train') num_class = len(set([label for (label, text) in train_iter])) # 获取分类数量 vocab_size = len(vocab) # 词汇表大小 emsize = 64 # 词嵌入维度 model = TextClassificationModel(vocab_size, emsize, num_class).to(device)

Step 6：Define functions to train the model and evaluate results

import timedef train(dataloader): model.train() total_acc, total_count = 0, 0 log_interval = 500 start_time = time.time()for idx, (label, text, offsets) in enumerate(dataloader): optimizer.zero_grad() predicted_label = model(text, offsets) loss = criterion(predicted_label, label) loss.backward() torch.nn.utils.clip_grad_norm_(model.parameters(), 0.1) optimizer.step() total_acc += (predicted_label.argmax(1) == label).sum().item() total_count += label.size(0) if idx % log_interval == 0 and idx > 0: elapsed = time.time() - start_time print('| epoch {:3d} | {:5d}/{:5d} batches ' '| accuracy {:8.3f}'.format(epoch, idx, len(dataloader), total_acc/total_count)) total_acc, total_count = 0, 0 start_time = time.time()def evaluate(dataloader): model.eval() total_acc, total_count = 0, 0with torch.no_grad(): for idx, (label, text, offsets) in enumerate(dataloader): predicted_label = model(text, offsets) loss = criterion(predicted_label, label) total_acc += (predicted_label.argmax(1) == label).sum().item() total_count += label.size(0) return total_acc/total_count

梯度裁剪 torch.nn.utils.clip_grad_norm_() 的使用应该在loss.backward()之后，optimizer.step()之前.

注意这个方法只在训练的时候使用，在测试的时候验证和测试的时候不用。

Step 7：Split the dataset and run the model 拆分训练集：拆分比率为训练集95%，验证集5%，使用torch.utils.data.dataset.random_split函数
to_map_style_dataset函数是将数据集从iterator变为map的形式，可以直接索引

from torch.utils.data.dataset import random_split from torchtext.data.functional import to_map_style_dataset # Hyperparameters EPOCHS = 10 # epoch LR = 5# learning rate BATCH_SIZE = 64 # batch size for trainingcriterion = torch.nn.CrossEntropyLoss() optimizer = torch.optim.SGD(model.parameters(), lr=LR) scheduler = torch.optim.lr_scheduler.StepLR(optimizer, 1.0, gamma=0.1)total_accu = Nonetrain_iter, test_iter = AG_NEWS() train_dataset = to_map_style_dataset(train_iter) test_dataset = to_map_style_dataset(test_iter)num_train = int(len(train_dataset) * 0.95) split_train_, split_valid_ = \ random_split(train_dataset, [num_train, len(train_dataset) - num_train])train_dataloader = DataLoader(split_train_, batch_size=BATCH_SIZE, shuffle=True, collate_fn=collate_batch) valid_dataloader = DataLoader(split_valid_, batch_size=BATCH_SIZE, shuffle=True, collate_fn=collate_batch) test_dataloader = DataLoader(test_dataset, batch_size=BATCH_SIZE, shuffle=True, collate_fn=collate_batch)for epoch in range(1, EPOCHS + 1): epoch_start_time = time.time() train(train_dataloader) accu_val = evaluate(valid_dataloader) if total_accu is not None and total_accu > accu_val: scheduler.step() else: total_accu = accu_val print('-' * 59) print('| end of epoch {:3d} | time: {:5.2f}s | valid accuracy {:8.3f} ' .format(epoch, time.time() - epoch_start_time, accu_val)) print('-' * 59)

输出：

| epoch1 |500/ 1782 batches | accuracy0.689 | epoch1 |1000/ 1782 batches | accuracy0.856 | epoch1 |1500/ 1782 batches | accuracy0.876 ----------------------------------------------------------- | end of epoch1 | time:8.17s | valid accuracy0.882 ----------------------------------------------------------- | epoch2 |500/ 1782 batches | accuracy0.897 | epoch2 |1000/ 1782 batches | accuracy0.904 | epoch2 |1500/ 1782 batches | accuracy0.900 ----------------------------------------------------------- | end of epoch2 | time:8.39s | valid accuracy0.893 ----------------------------------------------------------- | epoch3 |500/ 1782 batches | accuracy0.914 | epoch3 |1000/ 1782 batches | accuracy0.916 | epoch3 |1500/ 1782 batches | accuracy0.913 ----------------------------------------------------------- | end of epoch3 | time:8.44s | valid accuracy0.903 ----------------------------------------------------------- | epoch4 |500/ 1782 batches | accuracy0.924 | epoch4 |1000/ 1782 batches | accuracy0.923 | epoch4 |1500/ 1782 batches | accuracy0.924 ----------------------------------------------------------- | end of epoch4 | time:8.43s | valid accuracy0.908 ----------------------------------------------------------- | epoch5 |500/ 1782 batches | accuracy0.932 | epoch5 |1000/ 1782 batches | accuracy0.930 | epoch5 |1500/ 1782 batches | accuracy0.926 ----------------------------------------------------------- | end of epoch5 | time:8.37s | valid accuracy0.903 ----------------------------------------------------------- | epoch6 |500/ 1782 batches | accuracy0.941 | epoch6 |1000/ 1782 batches | accuracy0.943 | epoch6 |1500/ 1782 batches | accuracy0.941 ----------------------------------------------------------- | end of epoch6 | time:8.14s | valid accuracy0.908 ----------------------------------------------------------- | epoch7 |500/ 1782 batches | accuracy0.944 | epoch7 |1000/ 1782 batches | accuracy0.942 | epoch7 |1500/ 1782 batches | accuracy0.944 ----------------------------------------------------------- | end of epoch7 | time:8.15s | valid accuracy0.907 ----------------------------------------------------------- | epoch8 |500/ 1782 batches | accuracy0.943 | epoch8 |1000/ 1782 batches | accuracy0.943 | epoch8 |1500/ 1782 batches | accuracy0.945 ----------------------------------------------------------- | end of epoch8 | time:8.15s | valid accuracy0.907 ----------------------------------------------------------- | epoch9 |500/ 1782 batches | accuracy0.943 | epoch9 |1000/ 1782 batches | accuracy0.944 | epoch9 |1500/ 1782 batches | accuracy0.945 ----------------------------------------------------------- | end of epoch9 | time:8.15s | valid accuracy0.907 ----------------------------------------------------------- | epoch10 |500/ 1782 batches | accuracy0.943 | epoch10 |1000/ 1782 batches | accuracy0.944 | epoch10 |1500/ 1782 batches | accuracy0.945 ----------------------------------------------------------- | end of epoch10 | time:8.15s | valid accuracy0.907 -----------------------------------------------------------

Step 8：Evaluate the model with test dataset 检验模型在测试集上的效能

print('Checking the results of test dataset.') accu_test = evaluate(test_dataloader) print('test accuracy {:8.3f}'.format(accu_test))

输出：

Checking the results of test dataset. test accuracy0.909

Step 9：Test on a random news 随机输入一段新闻，测试模型效果：

ag_news_label = {1: "World", 2: "Sports", 3: "Business", 4: "Sci/Tec"}def predict(text, pipeline): with torch.no_grad(): text = torch.tensor(pipeline(text)) output = model(text, torch.tensor([0])) return output.argmax(1).item() + 1ex_text_str = "MEMPHIS, Tenn. – Four days ago, Jon Rahm was \ enduring the season’s worst weather conditions on Sunday at The \ Open on his way to a closing 75 at Royal Portrush, which \ considering the wind and the rain was a respectable showing. \ Thursday’s first round at the WGC-FedEx St. Jude Invitational \ was another story. With temperatures in the mid-80s and hardly any \ wind, the Spaniard was 13 strokes better in a flawless round. \ Thanks to his best putting performance on the PGA Tour, Rahm \ finished with an 8-under 62 for a three-stroke lead, which \ was even more impressive considering he’d never played the \ front nine at TPC Southwind."model = model.to('cpu') res = predict(ex_text_str, text_pipeline) print("This is a %s news" % ag_news_label[res])

【自然语言处理|自然语言处理（七）（AG_NEWS新闻分类任务（TORCHTEXT））】结果：