TensorFlow使用记录 (三）： Learning rate

file: tensorflow/python/training/learning_rate_decay.py

参考：tensorflow中常用学习率更新策略

神经网络中通过超参数 learning rate，来控制每次参数更新的幅度。学习率太小会降低网络优化的速度，增加训练时间；学习率太大则可能导致可能导致参数在局部最优解两侧来回振荡，网络不能收敛。

tensorflow 定义了很多的 学习率衰减方式：

指数衰减 tf.train.exponential_decay()

指数衰减是比较常用的衰减方法，学习率是跟当前的训练轮次指数相关的。

tf.train.exponential_decay(
    learning_rate,     # 初始学习率
    global_step,       # 当前训练轮次
    decay_steps,       # 衰减周期
    decay_rate,        # 衰减率系数
    staircase=False,   # 定义是否是阶梯型衰减，还是连续衰减，默认是 False
    name=None
)
‘‘‘
decayed_learning_rate = learning_rate *
                      decay_rate ^ (global_step / decay_steps)
‘‘‘

示例代码：

import tensorflow as tf
import matplotlib.pyplot as plt
style1 = []
style2 = []
N = 200

with tf.Session() as sess:
  sess.run(tf.global_variables_initializer())
  for step in range(N):
    # 标准指数型衰减
    learing_rate1 = tf.train.exponential_decay(
      learning_rate=0.5, global_step=step, decay_steps=10, decay_rate=0.9, staircase=False)
    # 阶梯型衰减
    learing_rate2 = tf.train.exponential_decay(
      learning_rate=0.5, global_step=step, decay_steps=10, decay_rate=0.9, staircase=True)
    lr1 = sess.run([learing_rate1])
    lr2 = sess.run([learing_rate2])
    style1.append(lr1)
    style2.append(lr2)

step = range(N)

plt.plot(step, style1, ‘g-‘, linewidth=2, label=‘exponential_decay‘)
plt.plot(step, style2, ‘r--‘, linewidth=1, label=‘exponential_decay_staircase‘)
plt.title(‘exponential_decay‘)
plt.xlabel(‘step‘)
plt.ylabel(‘learing rate‘)
plt.legend(loc=‘upper right‘)
plt.tight_layout()
plt.show()

分段常数衰减 tf.train.piecewise_constant()

tf.train.piecewise_constant_decay(
    x,                # 当前训练轮次
    boundaries,       # 学习率应用区间
    values,           # 学习率常数列表
    name=None
)
‘‘‘
learning_rate value is `values[0]` when `x <= boundaries[0]`,
    `values[1]` when `x > boundaries[0]` and `x <= boundaries[1]`, ...,
    and values[-1] when `x > boundaries[-1]`.
‘‘‘

示例代码：

import tensorflow as tf
import matplotlib.pyplot as plt
boundaries = [10, 20, 30]
learing_rates = [0.1, 0.07, 0.025, 0.0125]

style = []
N = 40

with tf.Session() as sess:
  sess.run(tf.global_variables_initializer())
  for step in range(N):
    learing_rate = tf.train.piecewise_constant(step, boundaries=boundaries, values=learing_rates)
    lr = sess.run([learing_rate])
    style.append(lr)

step = range(N)

plt.plot(step, style, ‘r-‘, linewidth=2)
plt.title(‘piecewise_constant‘)
plt.xlabel(‘step‘)
plt.ylabel(‘learing rate‘)
plt.tight_layout()
plt.show()

多项式衰减 tf.train.polynomial_decay()

tf.train.polynomial_decay(
    learning_rate,               # 初始学习率
    global_step,                 # 当前训练轮次
    decay_steps,                 # 大衰减周期
    end_learning_rate=0.0001,    # 最小的学习率
    power=1.0,                   # 多项式的幂
    cycle=False,                 # 学习率是否循环
    name=None)
‘‘‘
global_step = min(global_step, decay_steps)
decayed_learning_rate = (learning_rate - end_learning_rate) *
                        (1 - global_step / decay_steps) ^ (power) +
                         end_learning_rate
‘‘‘

示例代码：

import tensorflow as tf
import matplotlib.pyplot as plt
style1 = []
style2 = []
N = 200

with tf.Session() as sess:
  sess.run(tf.global_variables_initializer())
  for step in range(N):
    # cycle=False
    learing_rate1 = tf.train.polynomial_decay(
      learning_rate=0.1, global_step=step, decay_steps=50,
      end_learning_rate=0.01, power=0.5, cycle=False)
    # cycle=True
    learing_rate2 = tf.train.polynomial_decay(
      learning_rate=0.1, global_step=step, decay_steps=50,
      end_learning_rate=0.01, power=0.5, cycle=True)
    lr1 = sess.run([learing_rate1])
    lr2 = sess.run([learing_rate2])
    style1.append(lr1)
    style2.append(lr2)

steps = range(N)

plt.plot(steps, style1, ‘g-‘, linewidth=2, label=‘polynomial_decay‘)
plt.plot(steps, style2, ‘r--‘, linewidth=1, label=‘polynomial_decay_cycle‘)
plt.title(‘polynomial_decay‘)
plt.xlabel(‘step‘)
plt.ylabel(‘learing rate‘)
plt.legend(loc=‘upper right‘)
plt.tight_layout()
plt.show()

自然指数衰减 tf.train.natural_exp_decay()

tf.train.natural_exp_decay(
    learning_rate,        # 初始学习率
    global_step,          # 当前训练轮次
    decay_steps,          # 衰减周期
    decay_rate,           # 衰减率系数
    staircase=False,      # 定义是否是阶梯型衰减，还是连续衰减，默认是 False
    name=None
)
‘‘‘
decayed_learning_rate = learning_rate * exp(-decay_rate * global_step)
‘‘‘

示例代码：

import tensorflow as tf
import matplotlib.pyplot as plt
style1 = []
style2 = []
N = 200

with tf.Session() as sess:
  sess.run(tf.global_variables_initializer())
  for step in range(N):
    # 标准指数型衰减
    learing_rate1 = tf.train.natural_exp_decay(
      learning_rate=0.5, global_step=step, decay_steps=10, decay_rate=0.9, staircase=False)
    # 阶梯型衰减
    learing_rate2 = tf.train.natural_exp_decay(
      learning_rate=0.5, global_step=step, decay_steps=10, decay_rate=0.9, staircase=True)
    lr1 = sess.run([learing_rate1])
    lr2 = sess.run([learing_rate2])
    style1.append(lr1)
    style2.append(lr2)

step = range(N)

plt.plot(step, style1, ‘g-‘, linewidth=2, label=‘natural_exp_decay‘)
plt.plot(step, style2, ‘r--‘, linewidth=1, label=‘natural_exp_decay_staircase‘)
plt.title(‘natural_exp_decay‘)
plt.xlabel(‘step‘)
plt.ylabel(‘learing rate‘)
plt.legend(loc=‘upper right‘)
plt.tight_layout()
plt.show()

倒数衰减 tf.train.inverse_time_decay()

tf.train.inverse_time_decay(
    learning_rate,     # 初始学习率
    global_step,       # 当前训练轮次
    decay_steps,       # 衰减周期
    decay_rate,        # 衰减率系数
    staircase=False,   # 定义是否是阶梯型衰减，还是连续衰减，默认是 False
    name=None
)
‘‘‘
decayed_learning_rate = learning_rate / (1 + decay_rate * global_step / decay_step)
‘‘‘

示例代码：

import tensorflow as tf
import matplotlib.pyplot as plt
style1 = []
style2 = []
N = 200

with tf.Session() as sess:
  sess.run(tf.global_variables_initializer())
  for step in range(N):
    # 标准指数型衰减
    learing_rate1 = tf.train.inverse_time_decay(
      learning_rate=0.5, global_step=step, decay_steps=20, decay_rate=0.2, staircase=False)
    # 阶梯型衰减
    learing_rate2 = tf.train.inverse_time_decay(
      learning_rate=0.5, global_step=step, decay_steps=20, decay_rate=0.2, staircase=True)
    lr1 = sess.run([learing_rate1])
    lr2 = sess.run([learing_rate2])
    style1.append(lr1)
    style2.append(lr2)

step = range(N)

plt.plot(step, style1, ‘g-‘, linewidth=2, label=‘inverse_time_decay‘)
plt.plot(step, style2, ‘r--‘, linewidth=1, label=‘inverse_time_decay_staircase‘)
plt.title(‘inverse_time_decay‘)
plt.xlabel(‘step‘)
plt.ylabel(‘learing rate‘)
plt.legend(loc=‘upper right‘)
plt.tight_layout()
plt.show()

余弦衰减 tf.train.cosine_decay()

tf.train.cosine_decay(
    learning_rate,   # 初始学习率
    global_step,     # 当前训练轮次
    decay_steps,     # 衰减周期
    alpha=0.0,       # 最小的学习率
    name=None
)
‘‘‘
global_step = min(global_step, decay_steps)
cosine_decay = 0.5 * (1 + cos(pi * global_step / decay_steps))
decayed = (1 - alpha) * cosine_decay + alpha
decayed_learning_rate = learning_rate * decayed
‘‘‘

改进的余弦衰减方法还有：
线性余弦衰减，对应函数 tf.train.linear_cosine_decay()
噪声线性余弦衰减，对应函数 tf.train.noisy_linear_cosine_decay()

示例代码：

import tensorflow as tf
import matplotlib.pyplot as plt
style1 = []
style2 = []
style3 = []
N = 200

with tf.Session() as sess:
  sess.run(tf.global_variables_initializer())
  for step in range(N):
    # 余弦衰减
    learing_rate1 = tf.train.cosine_decay(
      learning_rate=0.1, global_step=step, decay_steps=50)
    # 线性余弦衰减
    learing_rate2 = tf.train.linear_cosine_decay(
      learning_rate=0.1, global_step=step, decay_steps=50)
    # 噪声线性余弦衰减
    learing_rate3 = tf.train.noisy_linear_cosine_decay(
      learning_rate=0.1, global_step=step, decay_steps=50,
      initial_variance=0.01, variance_decay=0.1, num_periods=0.2, alpha=0.5, beta=0.2)
    lr1 = sess.run([learing_rate1])
    lr2 = sess.run([learing_rate2])
    lr3 = sess.run([learing_rate3])
    style1.append(lr1)
    style2.append(lr2)
    style3.append(lr3)

step = range(N)

plt.plot(step, style1, ‘g-‘, linewidth=2, label=‘cosine_decay‘)
plt.plot(step, style2, ‘r--‘, linewidth=1, label=‘linear_cosine_decay‘)
plt.plot(step, style3, ‘b--‘, linewidth=1, label=‘linear_cosine_decay‘)
plt.title(‘cosine_decay‘)
plt.xlabel(‘step‘)
plt.ylabel(‘learing rate‘)
plt.legend(loc=‘upper right‘)
plt.tight_layout()
plt.show()

余弦衰减 tf.train.cosine_decay_restarts()

这是在 fast.ai 中强推的衰减方式

tf.train.cosine_decay_restarts(
    learning_rate,      # 初始学习率
    global_step,        # 当前训练轮次
    first_decay_steps,  # 首次衰减周期
    t_mul=2.0,          # 随后每次衰减周期倍数
    m_mul=1.0,          # 随后每次初始学习率倍数
    alpha=0.0,          # 最小的学习率
    name=None
)
‘‘‘
See [Loshchilov & Hutter, ICLR2016], SGDR: Stochastic Gradient Descent
with Warm Restarts. https://arxiv.org/abs/1608.03983
The learning rate multiplier first decays
from 1 to `alpha` for `first_decay_steps` steps. Then, a warm
restart is performed. Each new warm restart runs for `t_mul` times more steps
and with `m_mul` times smaller initial learning rate.
‘‘‘

示例代码：

import tensorflow as tf
import matplotlib.pyplot as plt
style1 = []
style2 = []
N = 200

with tf.Session() as sess:
  sess.run(tf.global_variables_initializer())
  for step in range(N):
    # 余弦衰减
    learing_rate1 = tf.train.cosine_decay_restarts(
      learning_rate=0.1, global_step=step, first_decay_steps=50,
      )
    # 循环余弦衰减
    learing_rate2 = tf.train.cosine_decay(
      learning_rate=0.1, global_step=step, decay_steps=50)
    lr1 = sess.run([learing_rate1])
    lr2 = sess.run([learing_rate2])
    style1.append(lr1)
    style2.append(lr2)

step = range(N)

plt.plot(step, style1, ‘g-‘, linewidth=2, label=‘cosine_decay_restarts‘)
plt.plot(step, style2, ‘r--‘, linewidth=1, label=‘cosine_decay‘)
plt.title(‘cosine_decay_restarts‘)
plt.xlabel(‘step‘)
plt.ylabel(‘learing rate‘)
plt.legend(loc=‘upper right‘)
plt.tight_layout()
plt.show()

原文地址：https://www.cnblogs.com/xuanyuyt/p/11621532.html

时间： 2024-10-20 15:46:08

TensorFlow使用记录 (三）： Learning rate的相关文章

Windows API 编程学习记录&lt;三&gt;

恩,开始写API编程的第三节,其实马上要考试了,但是不把这节写完,心里总感觉不舒服啊.写完赶紧去复习啊       在前两节中,我们介绍了Windows API 编程的一些基本概念和一个最基本API函数 MessageBox的使用,在这节中,我们就来正式编写一个Windows的窗口程序. 在具体编写代码之前,我们必须先要了解一下API 编写窗口程序具体的三个基本步骤:             1. 注册窗口类:             2.创建窗口:             3.显示窗口: 恩,

关于Increased rates of convergence through learning rate adaptation一文的理解

原文地址:http://www.researchgate.net/profile/Robert_Jacobs9/publication/223108796_Increased_rates_of_convergence_through_learning_rate_adaptation/links/0deec525d8f8dd5ade000000.pdf 已经看了CNN,rbm,sae等网络及算法,所有网络在训练时都需要一个learning rate,一直以来都觉得这个量设为定值即可,现在才发现其实

learning rate+feature +polynomial regression

well ,I have to say it's quite wise to trans cube into mutiple feature.Like you have y=2+3x+4x^2 +...and you choose to use y=a+3x1+4x2+....You have one feature ,but you rather more feature than cubed(three times).Also U have to use feature scaling ca

Sql Server删除数据表中重复记录 三种方法

本文介绍了Sql Server数据库中删除数据表中重复记录的方法. ［项目］数据库中users表,包含u_name,u_pwd两个字段,其中u_name存在重复项,现在要实现把重复的项删除!［分析］1.生成一张临时表new_users,表结构与users表一样:2.对users表按id做一个循环,每从users表中读出一个条记录,判断new_users中是否存在有相同的u_name,如果没有,则把它插入新表:如果已经有了相同的项,则忽略此条记录:3.把users表改为其它的名称,把new_use

梯度下降实用技巧II之学习率 Gradient descent in practice II -- learning rate

梯度下降实用技巧II之学习率 Gradient descent in practice II -- learning rate 梯度下降算法中的学习率(learning rate)很难确定,下面介绍一些寻找的实用技巧.首先看下如何确定你的梯度下降算法正在正常工作:一般是要画出代价函数 和迭代次数之间的图像,如下图所示.如果随着迭代次数的增加不断下降,那么说明梯度下降算法工作的很好,当到达一定迭代次数后基本持平时,说明已经收敛了,可以选取此时的作为参数.一般选取一个数小于,如果下降的幅度小于,则认

Spring Boot学习记录(三)--整合Mybatis

Spring Boot学习记录(三)–整合Mybatis 标签(空格分隔): spring-boot 控制器,视图解析器前面两篇都已弄好,这一篇学习持久层框架整合. 1.数据源配置 数据源使用druid,maven引入相关依赖,包括spring-jdbc依赖,mysql依赖 1.转换问题 配置的过程要学会为什么这样配置,而不是只学会了配置.这里我们可以和以前的配置方式对比: 以前版本 <!--配置数据库连接池Druid--> <bean id="dataSource"

machine learning (4)---learning rate

degugging:make sure gradient descent is working correctly cost function(J(θ)) of Number of iteration 运行错误的图象是什么样子的 运行正确的图象是什么样子的 how to choose learning rate(∂) 若learning rate太小 若learning rate太大 可供选择的一些learning rate值 如何选择最佳的learning rate

oracle 查询当天记录 三种方法效率比较

-- 查询一表中当天生成的数据 -- 原表mobilefrends中的cdate字段上有索引,创建索引语句是:create index mobilefrends_cdate_idx on mobilefrends(cdate); --------------------------------------------------------------------------------------------------------------------- -- 方法一:用to_char()

Batchsize与learning rate

https://www.zhihu.com/question/64134994 1.增加batch size会使得梯度更准确,但也会导致variance变小,可能会使模型陷入局部最优: 2.因此增大batch size通常要增大learning rate,比如batch size增大m倍,lr增大m倍或者sqrt(m)倍,但并不固定: 3.learning rate的增加通常不能直接增加太大,一般会通过warm up逐步增大: 4.warm up策略参考 Bag of Freebies for