TFLearn实现深度残差收缩网络(MNIST数据集)

深度残差收缩网络是一种新颖的深度学习算法,实际上是深度残差网络的升级版本,能够在一定程度上提高深度学习方法在含噪数据上的特征学习效果。

首先,简单回顾一下深度残差网络,深度残差网络的基本模块如下图所示。相较于普通的卷积神经网络,深度残差网络引入了跨层的恒等映射,来减小模型训练的难度,提高准确率。

然后,在深度残差网络的基础上,深度残差收缩网络引入了一个小型的子网络,用这个子网络学习得到一组阈值,对特征图的各个通道进行软阈值化。这个过程其实可以看成一个可训练的特征选择的过程。具体而言,就是通过前面的卷积层将重要的特征转换成绝对值较大的值,将冗余信息所对应的特征转换成绝对值较小的值;通过子网络学习得到二者之间的界限,并且通过软阈值化将冗余特征置为零,同时使重要的特征有着非零的输出。

深度残差收缩网络其实是一种通用的方法,不仅可以用于含噪数据的特征学习,也可以用于不含噪声数据的特征学习。这是因为,深度残差收缩网络中的阈值是根据样本情况自适应确定的。换言之,如果样本中不含冗余信息、不需要软阈值化,那么阈值可以被训练得非常接近于零,从而软阈值化就相当于不存在了。

最后,堆叠一定数量的基本模块,就得到了完整的网络结构。

利用深度残差收缩网络进行MNIST数据集的分类,可以看到,效果还是不错的。下面是深度残差收缩网络的程序:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Created on Thu Dec 26 07:46:00 2019

Implemented using TensorFlow 1.0 and TFLearn 0.3.2

M. Zhao, S. Zhong, X. Fu, et al., Deep Residual Shrinkage Networks for Fault Diagnosis,
IEEE Transactions on Industrial Informatics, 2019, DOI: 10.1109/TII.2019.2943898

@author: me
"""

import tflearn
import tensorflow as tf
from tflearn.layers.conv import conv_2d

# Data loading
from tflearn.datasets import mnist
X, Y, testX, testY = mnist.load_data(one_hot=True)
X = X.reshape([-1,28,28,1])
testX = testX.reshape([-1,28,28,1])

def residual_shrinkage_block(incoming, nb_blocks, out_channels, downsample=False,
                   downsample_strides=2, activation=‘relu‘, batch_norm=True,
                   bias=True, weights_init=‘variance_scaling‘,
                   bias_init=‘zeros‘, regularizer=‘L2‘, weight_decay=0.0001,
                   trainable=True, restore=True, reuse=False, scope=None,
                   name="ResidualBlock"):

    # residual shrinkage blocks with channel-wise thresholds

    residual = incoming
    in_channels = incoming.get_shape().as_list()[-1]

    # Variable Scope fix for older TF
    try:
        vscope = tf.variable_scope(scope, default_name=name, values=[incoming],
                                   reuse=reuse)
    except Exception:
        vscope = tf.variable_op_scope([incoming], scope, name, reuse=reuse)

    with vscope as scope:
        name = scope.name #TODO

        for i in range(nb_blocks):

            identity = residual

            if not downsample:
                downsample_strides = 1

            if batch_norm:
                residual = tflearn.batch_normalization(residual)
            residual = tflearn.activation(residual, activation)
            residual = conv_2d(residual, out_channels, 3,
                             downsample_strides, ‘same‘, ‘linear‘,
                             bias, weights_init, bias_init,
                             regularizer, weight_decay, trainable,
                             restore)

            if batch_norm:
                residual = tflearn.batch_normalization(residual)
            residual = tflearn.activation(residual, activation)
            residual = conv_2d(residual, out_channels, 3, 1, ‘same‘,
                             ‘linear‘, bias, weights_init,
                             bias_init, regularizer, weight_decay,
                             trainable, restore)

            # get thresholds and apply thresholding
            abs_mean = tf.reduce_mean(tf.reduce_mean(tf.abs(residual),axis=2,keep_dims=True),axis=1,keep_dims=True)
            scales = tflearn.fully_connected(abs_mean, out_channels//4, activation=‘linear‘,regularizer=‘L2‘,weight_decay=0.0001,weights_init=‘variance_scaling‘)
            scales = tflearn.batch_normalization(scales)
            scales = tflearn.activation(scales, ‘relu‘)
            scales = tflearn.fully_connected(scales, out_channels, activation=‘linear‘,regularizer=‘L2‘,weight_decay=0.0001,weights_init=‘variance_scaling‘)
            scales = tf.expand_dims(tf.expand_dims(scales,axis=1),axis=1)
            thres = tf.multiply(abs_mean,tflearn.activations.sigmoid(scales))
            residual = tf.multiply(tf.sign(residual), tf.maximum(tf.abs(residual)-thres,0))

            # Downsampling
            if downsample_strides > 1:
                identity = tflearn.avg_pool_2d(identity, 1,
                                               downsample_strides)

            # Projection to new dimension
            if in_channels != out_channels:
                if (out_channels - in_channels) % 2 == 0:
                    ch = (out_channels - in_channels)//2
                    identity = tf.pad(identity,
                                      [[0, 0], [0, 0], [0, 0], [ch, ch]])
                else:
                    ch = (out_channels - in_channels)//2
                    identity = tf.pad(identity,
                                      [[0, 0], [0, 0], [0, 0], [ch, ch+1]])
                in_channels = out_channels

            residual = residual + identity

    return residual

# Real-time data preprocessing
img_prep = tflearn.ImagePreprocessing()
img_prep.add_featurewise_zero_center(per_channel=True)

# Building A Deep Residual Shrinkage Network
net = tflearn.input_data(shape=[None, 28, 28, 1])
net = tflearn.conv_2d(net, 8, 3, regularizer=‘L2‘, weight_decay=0.0001)
net = residual_shrinkage_block(net, 1,  8, downsample=True)
net = tflearn.batch_normalization(net)
net = tflearn.activation(net, ‘relu‘)
net = tflearn.global_avg_pool(net)
# Regression
net = tflearn.fully_connected(net, 10, activation=‘softmax‘)
mom = tflearn.Momentum(0.1, lr_decay=0.1, decay_step=40000, staircase=True)
net = tflearn.regression(net, optimizer=mom, loss=‘categorical_crossentropy‘)
# Training
model = tflearn.DNN(net, checkpoint_path=‘model_mnist‘,
                    max_checkpoints=10, tensorboard_verbose=0,
                    clip_gradients=0.)

model.fit(X, Y, n_epoch=200, snapshot_epoch=False, snapshot_step=500,
          show_metric=True, batch_size=100, shuffle=True, run_id=‘model_mnist‘)

training_acc = model.evaluate(X, Y)[0]
validation_acc = model.evaluate(testX, testY)[0]

接下来是深度残差网络ResNet的程序:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Created on Thu Dec 26 07:46:00 2019

Implemented using TensorFlow 1.0 and TFLearn 0.3.2
K. He, X. Zhang, S. Ren, J. Sun, Deep Residual Learning for Image Recognition, CVPR, 2016.

@author: me
"""

import tflearn

# Data loading
from tflearn.datasets import mnist
X, Y, testX, testY = mnist.load_data(one_hot=True)
X = X.reshape([-1,28,28,1])
testX = testX.reshape([-1,28,28,1])

# Real-time data preprocessing
img_prep = tflearn.ImagePreprocessing()
img_prep.add_featurewise_zero_center(per_channel=True)

# Building a deep residual network
net = tflearn.input_data(shape=[None, 28, 28, 1])
net = tflearn.conv_2d(net, 8, 3, regularizer=‘L2‘, weight_decay=0.0001)
net = tflearn.residual_block(net, 1,  8, downsample=True)
net = tflearn.batch_normalization(net)
net = tflearn.activation(net, ‘relu‘)
net = tflearn.global_avg_pool(net)
# Regression
net = tflearn.fully_connected(net, 10, activation=‘softmax‘)
mom = tflearn.Momentum(0.1, lr_decay=0.1, decay_step=40000, staircase=True)
net = tflearn.regression(net, optimizer=mom, loss=‘categorical_crossentropy‘)
# Training
model = tflearn.DNN(net, checkpoint_path=‘model_mnist‘,
                    max_checkpoints=10, tensorboard_verbose=0,
                    clip_gradients=0.)

model.fit(X, Y, n_epoch=200, snapshot_epoch=False, snapshot_step=500,
          show_metric=True, batch_size=100, shuffle=True, run_id=‘model_mnist‘)

training_acc = model.evaluate(X, Y)[0]
validation_acc = model.evaluate(testX, testY)[0]

上述两个程序构建了只有1个基本模块的小型网络,MNIST数据集中没有添加噪声。准确率如下表所示(每次运行结果会有些波动),可以看到,即使是对于不含噪声的数据,深度残差收缩网络的结果也是挺不错的:

参考文献:

M. Zhao, S. Zhong, X. Fu, et al., Deep residual shrinkage networks for fault diagnosis, IEEE Transactions on Industrial Informatics, 2019, DOI: 10.1109/TII.2019.2943898

https://ieeexplore.ieee.org/document/8850096

原文地址:https://www.cnblogs.com/yc-9527/p/12113173.html

时间: 2024-11-10 06:05:57

TFLearn实现深度残差收缩网络(MNIST数据集)的相关文章

Keras快速搭建深度残差收缩网络(及深度残差网络)

从本质上讲,深度残差收缩网络属于卷积神经网络,是深度残差网络(deep residual network, ResNet)的一个变种.它的核心思想在于,在深度学习进行特征学习的过程中,剔除冗余信息是非常重要的:软阈值化是一种非常灵活的.删除冗余信息的方式. 1.深度残差网络 首先,在介绍深度残差收缩网络的时候,经常需要从深度残差网络开始讲起.下图展示了深度残差网络的基本模块,包括一些非线性层(残差路径)和一个跨层的恒等连接.恒等连接是深度残差网络的核心,是其优异性能的一个保障. 2.深度残差收缩

深度残差收缩网络:(一)背景知识

深度残差收缩网络(Deep Residual Shrinkage Network)是深度残差学习(Deep Residual Network, ResNet)的一种改进,发表在IEEE Transactions on Industrial Informatics上,面向的是数据包含噪声的情况. 简单地讲,深度残差收缩网络就是,将软阈值化作为可训练的模块,嵌入到ResNet之中.接下来结合自己的理解,解读一下相关的背景知识. (1)噪声的含义 如上所述,深度残差收缩网络面向的是数据包含噪声的情况.

深度残差收缩网络:(三)网络结构

(1)回顾一下深度残差网络的结构 在下图中,(a)-(c)分别是三种残差模块,(d)是深度残差网络的整体示意图.BN指的是批标准化(Batch Normalization),ReLU指的是整流线性单元激活函数(Rectifier Linear Unit),Conv指的是卷积层(Convolutional layer),Identity shortcut指的是跨层的恒等映射,RBU指的是残差模块(Residual Building Unit),GAP是全局均值池化(Global Average P

深度残差收缩网络:(四)注意力机制下的阈值设置

对于基于深度学习的分类算法,其关键不仅在于提取与标签相关的目标信息,剔除无关的信息也是非常重要的,所以要在深度神经网络中引入软阈值化.阈值的自动设置,是深度残差收缩网络的核心贡献.需要注意的是,软阈值化中的阈值,需要满足一定的条件.这篇文章中的阈值设置,事实上,是在注意力机制下进行的.下面分别介绍阈值需要满足的条件.注意力机制以及具体的阈值设置方法. (1)阈值需要满足的条件 在软阈值化中,阈值的取值有一定的要求:首先,阈值必须是正数:其次,阈值不能太大,否则输出会全部为零. 更重要的是,每个样

深度残差收缩网络:(五)实验验证

实验部分将所提出的两种深度残差收缩网络,即“通道之间共享阈值的深度残差收缩网络(Deep Residual Shrinkage Networks with Channel-shared Thresholds,简称DRSN-CS)”,和“逐通道不同阈值的深度残差收缩网络(Deep Residual Shrinkage Networks with Channel-wise Thresholds,简称DRSN-CW)”,与传统的卷积神经网络(Convolutional Neural Networks,

基于深度残差收缩网络的故障诊断

论文题目:Deep residual shrinkage networks for fault diagnosis 题目翻译:基于深度残差收缩网络的故障诊断 期刊:IEEE Transactions on Industrial Informatics 面向问题:强噪声(或冗余信息繁多) 创新:①在深度残差网络中引入软阈值化,②并且自动设置阈值 本质:在注意力机制下实现特征的软阈值化 M. Zhao, S, Zhong, X. Fu, et al. Deep residual shrinkage

CNN卷积神经网络_深度残差网络 ResNet——解决神经网络过深反而引起误差增加的根本问题

from:https://blog.csdn.net/diamonjoy_zone/article/details/70904212 环境:Win8.1 TensorFlow1.0.1 软件:Anaconda3 (集成Python3及开发环境) TensorFlow安装:pip install tensorflow (CPU版) pip install tensorflow-gpu (GPU版) TFLearn安装:pip install tflearn 参考: Deep Residual Le

Dual Path Networks(DPN)——一种结合了ResNet和DenseNet优势的新型卷积网络结构。深度残差网络通过残差旁支通路再利用特征,但残差通道不善于探索新特征。密集连接网络通过密集连接通路探索新特征,但有高冗余度。

如何评价Dual Path Networks(DPN)? 论文链接:https://arxiv.org/pdf/1707.01629v1.pdf在ImagNet-1k数据集上,浅DPN超过了最好的ResNeXt-101(64×4d),具有26%更小的模型尺寸,25%的计算成本和8%的更低的内存消耗 5 个回答 xiaozhi CV.ML.DL 1.针对视觉识别任务的"网络工程"一直是研究的重点,其重在设计更为高效的网络拓扑结构,一方面考虑更好的特征表示学习,另一方面尽可能减少计算复杂度

【TensorFlow/简单网络】MNIST数据集-softmax、全连接神经网络,卷积神经网络模型

初学tensorflow,参考了以下几篇博客: soft模型 tensorflow构建全连接神经网络 tensorflow构建卷积神经网络 tensorflow构建卷积神经网络 tensorflow构建CNN[待学习] 全连接+各种优化[待学习] BN层[待学习] 先解释以下MNIST数据集,训练数据集有55,000 条,即X为55,000 * 784的矩阵,那么Y为55,000 * 10的矩阵,每个图片是28像素*28像素,带有标签,Y为该图片的真实数字,即标签,每个图片10个数字,1所在位置