基于PaddlePaddle框架利用RNN（循环神经网络）生成古诗句

基于PaddlePaddle框架利用RNN（循环神经网络）生成古诗句

在本项目中，将使用PaddlePaddle实现循环神经网络模型（即RNN模型，以下循环神经网络都称作RNN）,并实现基于RNN语言模型进行诗句的生成。

本项目利用全唐诗数据集对RNN语言模型进行训练，能够实现根据输入的前缀诗句，自动生成后续诗句。

本实验所用全唐诗数据集下载地址：https://pan.baidu.com/s/1OgIdxjO2jh5KC8XzG-j8ZQ

1.背景知识

RNN是一个序列模型，基本思路是：在时刻t，将前一时刻t?1的隐藏层输出和t时刻的词向量一起输入到隐藏层从而得到时刻t的特征表示，然后用这个特征表示得到t时刻的预测输出，如此在时间维上递归下去。可以看出RNN善于使用上文信息、历史知识，具有“记忆”功能。理论上RNN能实现“长依赖”（即利用很久之前的知识），但在实际应用中发现效果并不理想，研究提出了LSTM和GRU等变种，通过引入门机制对传统RNN的记忆单元进行了改进，弥补了传统RNN在学习长序列时遇到的难题。本次实验模型使用了LSTM或GRU，可通过配置进行修改。下图是RNN（广义上包含了LSTM、GRU等）语言模型“循环”思想的示意图：

本项目要用到的RNN语言模型(Language Model)是一个概率分布模型，简单来说，就是用来计算一个句子的概率的模型。利用它可以确定生成的哪个词序列的可能性更大，或者给定若干个词，可以预测下一个最可能出现的词。基于RNN语言模型的特点，因此，我们打算利用该模型进行古诗句的生成。

另外，语言模型也是自然语言处理领域里一个重要的基础模型。

简单了解以上原理后，我们就可以正式开始本项目了。

数据预处理：

这里简单介绍数据集及其结构。本实验使用的全唐诗数据集，数据集以txt文件的形式存储，已经过以下初步处理并直接提供。

• 清洗语料：去除了数据集中的诗的标题、标点符号、注释、乱码及特殊符号等。
• 内容格式：每首诗占一行；每行中的各个字之间使用一个空格符分开。

数据集样式为：

2.项目实现过程

2.1 引入相关库

 1 import os
 2 import sys
 3 import gzip
 4 import collections
 5 import logging
 6 from collections import defaultdict
 7 import numpy as np
 8 import math
 9 import config as conf  #config.py中定义了模型的参数变量
10
11 import paddle.v2 as paddle

2.2 设置默认logging配置

1 #设置默认logging配置
2 logger = logging.getLogger("paddle")
3 logger.setLevel(logging.DEBUG)

2.3 定义构建字典函数

 1 #定义构建字典函数
 2 def build_dict(data_file,
 3                save_path,
 4                max_word_num,
 5                cutoff_word_fre=5,
 6                insert_extra_words=["<unk>", "<e>"]):
 7     #使用defaultdict()任何未定义的key都会默认返回一个默认值，避免在使用不存在的key时dict()返回KeyError的问题。
 8     word_count = defaultdict(int)
 9     with open(data_file, "r") as f:
10         for idx, line in enumerate(f):
11             if not (idx + 1) % 100000:
12                 logger.debug("processing %d lines ... " % (idx + 1))
13             words = line.strip().lower().split()
14             for w in words:
15                 word_count[w] += 1
16     #将字按字频排序
17     sorted_words = sorted(
18         word_count.iteritems(), key=lambda x: x[1], reverse=True)
19
20     stop_pos = len(sorted_words) if sorted_words[-1][
21         1] > cutoff_word_fre else next(idx for idx, v in enumerate(sorted_words)
22                                        if v[1] < cutoff_word_fre)
23
24     stop_pos = min(max_word_num, stop_pos)
25     with open(save_path, "w") as fdict:
26         for w in insert_extra_words:
27             fdict.write("%s\t-1\n" % (w))
28         for idx, info in enumerate(sorted_words):
29             if idx == stop_pos: break
30             fdict.write("%s\t%d\n" % (info[0], info[-1]))

2.4  定义加载字典函数

 1 #定义加载字典函数
 2 def load_dict(dict_path):
 3     """
 4     load word dictionary from the given file. Each line of the give file is
 5     a word in the word dictionary. The first column of the line, seperated by
 6     TAB, is the key, while the line index is the value.
 7
 8     """
 9     return dict((line.strip().split("\t")[0], idx)
10                 for idx, line in enumerate(open(dict_path, "r").readlines()))
11
12
13 def load_reverse_dict(dict_path):
14     """
15     load word dictionary from the given file. Each line of the give file is
16     a word in the word dictionary. The line index is the key, while the first
17     column of the line, seperated by TAB, is the value.
18
19     """
20     return dict((idx, line.strip().split("\t")[0])
21                 for idx, line in enumerate(open(dict_path, "r").readlines()))

2.5 定义Beam search类

  1 #定义BeamSearch类
  2 class BeamSearch(object):
  3     """
  4     Generating sequence by beam search
  5     NOTE: this class only implements generating one sentence at a time.
  6     """
  7
  8     def __init__(self, inferer, word_dict_file, beam_size=1, max_gen_len=100):
  9         """
 10         constructor method.
 11
 12         :param inferer: object of paddle.Inference that represents the entire
 13             network to forward compute the test batch
 14         :type inferer: paddle.Inference
 15         :param word_dict_file: path of word dictionary file
 16         :type word_dict_file: str
 17         :param beam_size: expansion width in each iteration
 18         :type param beam_size: int
 19         :param max_gen_len: the maximum number of iterations
 20         :type max_gen_len: int
 21         """
 22         self.inferer = inferer
 23         self.beam_size = beam_size
 24         self.max_gen_len = max_gen_len
 25         self.ids_2_word = load_reverse_dict(word_dict_file)
 26         logger.info("dictionay len = %d" % (len(self.ids_2_word)))
 27
 28         try:
 29             self.eos_id = next(x[0] for x in self.ids_2_word.iteritems()
 30                                if x[1] == "<e>")
 31             self.unk_id = next(x[0] for x in self.ids_2_word.iteritems()
 32                                if x[1] == "<unk>")
 33         except StopIteration:
 34             logger.fatal(("the word dictionay must contain an ending mark "
 35                           "in the text generation task."))
 36
 37         self.candidate_paths = []
 38         self.final_paths = []
 39
 40     def _top_k(self, softmax_out, k):
 41         """
 42         get indices of the words with k highest probablities.
 43         NOTE: <unk> will be excluded if it is among the top k words, then word
 44         with (k + 1)th highest probability will be returned.
 45
 46         :param softmax_out: probablity over the dictionary
 47         :type softmax_out: narray
 48         :param k: number of word indices to return
 49         :type k: int
 50         :return: indices of k words with highest probablities.
 51         :rtype: list
 52         """
 53         ids = softmax_out.argsort()[::-1]
 54         return ids[ids != self.unk_id][:k]
 55
 56     def _forward_batch(self, batch):
 57         """
 58         forward a test batch.
 59
 60         :params batch: the input data batch
 61         :type batch: list
 62         :return: probablities of the predicted word
 63         :rtype: ndarray
 64         """
 65         return self.inferer.infer(input=batch, field=["value"])
 66
 67     def _beam_expand(self, next_word_prob):
 68         """
 69         In every iteration step, the model predicts the possible next words.
 70         For each input sentence, the top k words is added to end of the original
 71         sentence to form a new generated sentence.
 72
 73         :param next_word_prob: probablities of the next words
 74         :type next_word_prob: ndarray
 75         :return:  the expanded new sentences.
 76         :rtype: list
 77         """
 78         assert len(next_word_prob) == len(self.candidate_paths), (
 79             "Wrong forward computing results!")
 80         top_beam_words = np.apply_along_axis(self._top_k, 1, next_word_prob,
 81                                              self.beam_size)
 82         new_paths = []
 83         for i, words in enumerate(top_beam_words):
 84             old_path = self.candidate_paths[i]
 85             for w in words:
 86                 log_prob = old_path["log_prob"] + math.log(next_word_prob[i][w])
 87                 gen_ids = old_path["ids"] + [w]
 88                 if w == self.eos_id:
 89                     self.final_paths.append({
 90                         "log_prob": log_prob,
 91                         "ids": gen_ids
 92                     })
 93                 else:
 94                     new_paths.append({"log_prob": log_prob, "ids": gen_ids})
 95         return new_paths
 96
 97     def _beam_shrink(self, new_paths):
 98         """
 99         to return the top beam_size generated sequences with the highest
100         probabilities at the end of evey generation iteration.
101
102         :param new_paths: all possible generated sentences
103         :type new_paths: list
104         :return: a state flag to indicate whether to stop beam search
105         :rtype: bool
106         """
107
108         if len(self.final_paths) >= self.beam_size:
109             max_candidate_log_prob = max(
110                 new_paths, key=lambda x: x["log_prob"])["log_prob"]
111             min_complete_path_log_prob = min(
112                 self.final_paths, key=lambda x: x["log_prob"])["log_prob"]
113             if min_complete_path_log_prob >= max_candidate_log_prob:
114                 return True
115
116         new_paths.sort(key=lambda x: x["log_prob"], reverse=True)
117         self.candidate_paths = new_paths[:self.beam_size]
118         return False
119
120     def gen_a_sentence(self, input_sentence):
121         """
122         generating sequence for an given input
123
124         :param input_sentence: one input_sentence
125         :type input_sentence: list
126         :return: the generated word sequences
127         :rtype: list
128         """
129         self.candidate_paths = [{"log_prob": 0., "ids": input_sentence}]
130         input_len = len(input_sentence)
131
132         for i in range(self.max_gen_len):
133             next_word_prob = self._forward_batch(
134                 [[x["ids"]] for x in self.candidate_paths])
135             new_paths = self._beam_expand(next_word_prob)
136
137             min_candidate_log_prob = min(
138                 new_paths, key=lambda x: x["log_prob"])["log_prob"]
139
140             path_to_remove = [
141                 path for path in self.final_paths
142                 if path["log_prob"] < min_candidate_log_prob
143             ]
144             for p in path_to_remove:
145                 self.final_paths.remove(p)
146
147             if self._beam_shrink(new_paths):
148                 self.candidate_paths = []
149                 break
150
151         gen_ids = sorted(
152             self.final_paths + self.candidate_paths,
153             key=lambda x: x["log_prob"],
154             reverse=True)[:self.beam_size]
155         self.final_paths = []
156
157         def _to_str(x):
158             text = " ".join(self.ids_2_word[idx]
159                             for idx in x["ids"][input_len:])
160             return "%.4f\t%s" % (x["log_prob"], text)
161
162         return map(_to_str, gen_ids)

2.6  定义RNN网络结构函数

 1 #定义RNN网络结构函数
 2 def rnn_lm(vocab_dim,
 3            emb_dim,
 4            hidden_size,
 5            stacked_rnn_num,
 6            rnn_type="lstm",
 7            is_infer=False):
 8     """
 9     RNN language model definition.
10
11     """
12
13     # 定义输入层，input layers
14     input = paddle.layer.data(
15         name="input", type=paddle.data_type.integer_value_sequence(vocab_dim))
16     if not is_infer:
17         target = paddle.layer.data(
18             name="target",
19             type=paddle.data_type.integer_value_sequence(vocab_dim))
20
21     # 定义词向量层，embedding layer
22     input_emb = paddle.layer.embedding(input=input, size=emb_dim)
23
24     # 定义 RNN 层，预先提供“lstm"与"gru"两种类别的RNN，可在配置文件中选择 rnn layer
25     if rnn_type == "lstm":                                                    #本实验使用的是lstm
26         for i in range(stacked_rnn_num):
27             rnn_cell = paddle.networks.simple_lstm(
28                 input=rnn_cell if i else input_emb, size=hidden_size)
29     elif rnn_type == "gru":
30         for i in range(stacked_rnn_num):
31             rnn_cell = paddle.networks.simple_gru(
32                 input=rnn_cell if i else input_emb, size=hidden_size)
33     else:
34         raise Exception("rnn_type error!")
35
36     # 定义全连接输出层，fc(full connected) and output layer
37     output = paddle.layer.fc(input=[rnn_cell],
38                              size=vocab_dim,
39                              act=paddle.activation.Softmax())
40
41     if is_infer:
42         last_word = paddle.layer.last_seq(input=output)
43         return last_word
44     else:
45         cost = paddle.layer.classification_cost(input=output, label=target)    #定义损失函数
46
47         return cost

2.7 定义reader及一些参数

 1 # 限定读取文本的最短长度 sentence‘s min length.
 2 MIN_LEN = 3
 3
 4 # 限定读取文本的最长长度 sentence‘s max length.
 5 MAX_LEN = 100
 6
 7 # 定义RNNreader
 8 def rnn_reader(file_name, word_dict):
 9     """
10     create reader for RNN, each line is a sample.
11
12     """
13
14     def reader():
15         UNK_ID = word_dict[‘<unk>‘]
16         with open(file_name) as file:
17             for line in file:
18                 words = line.strip().lower().split()
19                 if len(words) < MIN_LEN or len(words) > MAX_LEN:
20                     continue
21                 ids = [word_dict.get(w, UNK_ID)
22                        for w in words] + [word_dict[‘<e>‘]]
23                 yield ids[:-1], ids[1:]
24
25     return reader

2.8  定义模型训练函数

 1 # 定义模型训练函数
 2 def train(topology,
 3           train_reader,
 4           test_reader,
 5           model_save_dir="models",
 6           num_passes=10):
 7     """
 8     train model.
 9
10     """
11     # 创建训练模型参数保存目录
12     if not os.path.exists(model_save_dir):
13         os.mkdir(model_save_dir)
14
15     # 初始化 PaddlePaddle
16     paddle.init(use_gpu=conf.use_gpu, trainer_count=conf.trainer_count)
17
18     # 创建 optimizer，使用Adam优化算法
19     adam_optimizer = paddle.optimizer.Adam(
20         learning_rate=1e-3,
21         regularization=paddle.optimizer.L2Regularization(rate=1e-3),
22         model_average=paddle.optimizer.ModelAverage(
23             average_window=0.5, max_average_window=10000))
24
25     # 创建 parameters
26     parameters = paddle.parameters.create(topology)
27
28     # 创建 sum evaluator
29     sum_eval = paddle.evaluator.sum(topology)
30
31     # 创建 trainer 其中paddle.trainer.SGD() 函数定义一个随机梯度下降trainer，
32     # 配置4个参数cost、parameters、update_equation、extra_layers，它们分别表示损失函数、参数、更新公式以及评估器。
33     trainer = paddle.trainer.SGD(cost=topology,
34                                  parameters=parameters,
35                                  update_equation=adam_optimizer,
36                                  extra_layers=sum_eval)
37
38     # 定义 event_handler 以打印训练进度
39     def event_handler(event):
40         if isinstance(event, paddle.event.EndIteration):
41             # 每隔一个 log_period 打印一下训练信息
42             if not event.batch_id % conf.log_period:
43                 logger.info("Pass %d, Batch %d, Cost %f, %s" % (
44                     event.pass_id, event.batch_id, event.cost, event.metrics))
45
46             #每隔一个 log_period 保存一次训练模型参数
47             if (not event.batch_id %
48                     conf.save_period_by_batches) and event.batch_id:
49                 save_name = os.path.join(model_save_dir,
50                                          "rnn_lm_pass_%05d_batch_%03d.tar.gz" %
51                                          (event.pass_id, event.batch_id))
52                 with gzip.open(save_name, "w") as f:
53                     trainer.save_parameter_to_tar(f)
54
55         if isinstance(event, paddle.event.EndPass):
56             if test_reader is not None:
57                 result = trainer.test(reader=test_reader)
58                 logger.info("Test with Pass %d, %s" %
59                             (event.pass_id, result.metrics))
60             save_name = os.path.join(model_save_dir, "rnn_lm_pass_%05d.tar.gz" %
61                                      (event.pass_id))
62             with gzip.open(save_name, "w") as f:
63                 trainer.save_parameter_to_tar(f)
64     # 输出训练运行日志
65     logger.info("start training...")
66     trainer.train(
67         reader=train_reader, event_handler=event_handler, num_passes=num_passes)
68
69     logger.info("Training is finished.")

2.9 开始训练

 1 # 进行训练工作
 2
 3 def main():
 4     # 构建并加载词典文件
 5     if not (os.path.exists(conf.vocab_file) and
 6             os.path.getsize(conf.vocab_file)):
 7         logger.info(("word dictionary does not exist, "
 8                      "build it from the training data"))
 9         build_dict(conf.train_file, conf.vocab_file, conf.max_word_num,
10                    conf.cutoff_word_fre)
11     logger.info("load word dictionary.")
12     word_dict = load_dict(conf.vocab_file)
13     logger.info("dictionay size = %d" % (len(word_dict)))
14
15     cost = rnn_lm(
16         len(word_dict), conf.emb_dim, conf.hidden_size, conf.stacked_rnn_num,
17         conf.rnn_type)
18
19     # 使用 reader
20     reader_args = {
21         "file_name": conf.train_file,
22         "word_dict": word_dict,
23     }
24     # paddle.reader.shuffle(train(), buf_size=102400)表示trainer从 rnn_reader 中读取了 buf_size=102400大小的数据并打乱顺序
25     # batch_size 表示从打乱的数据中再取出 batch_size=con.batch_size 大小的数据进行一次迭代训练
26     train_reader = paddle.batch(
27         paddle.reader.shuffle(
28             rnn_reader(**reader_args), buf_size=102400),
29         batch_size=conf.batch_size)
30     test_reader = None
31     if os.path.exists(conf.test_file) and os.path.getsize(conf.test_file):
32         test_reader = paddle.batch(
33             paddle.reader.shuffle(
34                 rnn_reader(**reader_args), buf_size=65536),
35             batch_size=conf.batch_size)
36     # 正式开始训练,相关参数调用自 config.py 文件
37     train(
38         topology=cost,
39         train_reader=train_reader,
40         test_reader=test_reader,
41         model_save_dir=conf.model_save_dir,
42         num_passes=conf.num_passes)
43
44
45
46 main()

2.10 定义生成诗句函数

 1 # 定义生成诗句函数
 2 def rnn_generate(gen_input_file, model_path, max_gen_len, beam_size,
 3                  word_dict_file):
 4     """
 5     use RNN model to generate sequences.
 6
 7     """
 8
 9     assert os.path.exists(gen_input_file), "test file does not exist!"
10     assert os.path.exists(model_path), "trained model does not exist!"
11     assert os.path.exists(
12         word_dict_file), "word dictionary file does not exist!"
13
14     # 加载词典文件
15     word_2_ids = load_dict(word_dict_file)
16     try:
17         UNK_ID = word_2_ids["<unk>"]
18     except KeyError:
19         logger.fatal("the word dictionary must contain a <unk> token!")
20         sys.exit(-1)
21
22     # 初始化 PaddlePaddle
23     paddle.init(use_gpu=conf.use_gpu, trainer_count=conf.trainer_count)
24
25     # 加载训练好的模型
26     pred_words = rnn_lm(
27         len(word_2_ids),
28         conf.emb_dim,
29         conf.hidden_size,
30         conf.stacked_rnn_num,
31         conf.rnn_type,
32         is_infer=True)
33     # 使用训练完成的模型参数
34     parameters = paddle.parameters.Parameters.from_tar(
35         gzip.open(model_path, "r"))
36
37     inferer = paddle.inference.Inference(
38         output_layer=pred_words, parameters=parameters)
39
40     generator = BeamSearch(inferer, word_dict_file, beam_size, max_gen_len)
41     # 生成诗句文本
42     with open(conf.gen_file, "r") as fin, open(conf.gen_result, "w") as fout:
43         for idx, line in enumerate(fin):
44             fout.write("%d\t%s" % (idx, line))
45             for gen_res in generator.gen_a_sentence([
46                     word_2_ids.get(w, UNK_ID)
47                     for w in line.lower().strip().split()
48             ]):
49                 fout.write("%s\n" % gen_res)
50             fout.write("\n")
51
52
53
54 rnn_generate(conf.gen_file, conf.model_path, conf.max_gen_len,
55                  conf.beam_size, conf.vocab_file)

2.11 打印生成的诗句

1 # 查看看一下生成的诗句效果
2 print("生成诗句展示：\n")
3 for i in open("data/gen_result.txt"):
4     print i

生成的诗句展示：

0	黄 河 远 上 白 云 间

-12.4769	今 日 逢 君 泪 满 缨

-12.7528	今 日 逢 君 泪 满 衣

-13.8208	今 日 逢 君 泪 满 巾

-13.8563	今 日 将 军 汉 水 边

-14.0999	今 日 逢 君 不 见 春

1	羌 笛 何 须 怨 杨 柳

-8.9801	不 知 何 处 草 萋 萋

-9.3334	不 知 何 处 是 春 风

-10.4879	不 知 何 处 是 花 枝

-11.1031	不 知 人 不 知 何 人

-11.2889	不 知 何 处 是 春 山

2	无 边 落 木 萧 萧 下

-12.0765	日 暮 孤 舟 何 处 来

-12.5019	独 自 撄 宁 逆 鼓 鼙

-13.0668	日 暮 孤 舟 何 处 期

-13.7688	日 暮 孤 舟 何 处 村

-13.8054	日 暮 孤 舟 何 处 寻

3	凤 凰 台 上 凤 凰 游

-14.9325	年 少 恩 深 万 岁 同

-14.9472	年 少 恩 华 事 可 知

-15.1605	年 少 恩 深 万 里 馀

-15.1802	年 少 恩 深 万 里 同

-15.1948	年 少 恩 深 万 里 人

4	日 出 江 花 红 胜 火-10.9732	春 风 袅 袅 百 花 枝

-11.9631	春 风 袅 袅 春 风 起

-12.2523	春 风 袅 袅 满 池 塘

-12.8227	春 风 袅 袅 百 花 时

-12.9489	春 风 袅 袅 百 花 飞
3.相关补充对于导入模块与库过程中，引入的config.py文件，其全部代码如下，可以根据自己的语料与期望生成调整相关参数，其中examples.txt文件可自拟，但是由于本项目是按字训练，因此examples.txt文件要注意每两字之间用空格隔开：

 1 import os
 2
 3 ################## for building word dictionary  ##################
 4
 5 max_word_num = 51200 - 2
 6 cutoff_word_fre = 0
 7
 8 ################## for training task  #########################
 9 # path of training data
10 train_file = "data/poetrys.txt"
11
12 # path of testing data, if testing file does not exist,
13 # testing will not be performed at the end of each training pass
14 test_file = ""
15
16 # path of word dictionary, if this file does not exist,
17 # word dictionary will be built from training data.
18 vocab_file = "data/word_vocab.txt"
19
20 # directory to save the trained model
21 # create a new directory if the directoy does not exist
22 model_save_dir = "models"
23
24 batch_size = 32  # the number of training examples in one forward/backward pass
25 num_passes = 50  # how many passes to train the model
26
27 log_period = 50
28 save_period_by_batches = 50
29
30 use_gpu = False  # to use gpu or not
31 trainer_count = 1  # number of trainer
32
33 ##################  for model configuration  ##################
34 rnn_type = "lstm"  # "gru" or "lstm"
35 emb_dim = 256
36 hidden_size = 256
37 stacked_rnn_num = 2
38
39 ##################  for poetry generation  ##################
40 gen_file = "data/examples.txt"
41 gen_result = "data/gen_result.txt"
42 max_gen_len = 7  # the max number of words to generate
43 beam_size = 5
44 model_path = "models/rnn_lm_pass_00044_batch_750.tar.gz"
45
46 if not os.path.exists(model_save_dir):
47     os.mkdir(model_save_dir)

原文地址：https://www.cnblogs.com/youngawesome/p/9091207.html