受限玻尔兹曼机(Restricted Boltzmann Machine)

受限玻尔兹曼机(Restricted Boltzmann Machine)

作者:凯鲁嘎吉 - 博客园 http://www.cnblogs.com/kailugaji/

1. 生成模型

2. 参数学习

3. 对比散度学习算法

由于受限玻尔兹曼机的特殊结构,因此可以使用一种比吉布斯采样更有效 的学习算法,即对比散度(Contrastive Divergence)对比散度算法仅需k步吉布斯采样。为了提高效率,对比散度算法用一个训练样本作为可观测向量的初始值。然后,交替对可观测向量和隐藏向量进行吉布斯采样,不需要等到收敛,只需要k步就足够了。这就是CD-k 算法。通常,k = 1就可以学得很好。对比散度的流程如算法12.1所示。

4. MATLAB程序解读

% Version 1.000
%
% Code provided by Geoff Hinton and Ruslan Salakhutdinov
%
% Permission is granted for anyone to copy, use, modify, or distribute this
% program and accompanying programs and documents for any purpose, provided
% this copyright notice is retained and prominently displayed, along with
% a note saying that the original programs are available from our
% web page.
% The programs and documents are distributed without any warranty, express or
% implied.  As the programs were written for research purposes only, they have
% not been tested to the degree that would be advisable in any important
% application.  All use of these programs is entirely at the user‘s own risk.
%
% This program trains Restricted Boltzmann Machine in which
% visible, binary, stochastic pixels are connected to
% hidden, binary, stochastic feature detectors using symmetrically
% weighted connections. Learning is done with 1-step Contrastive Divergence.
% The program assumes that the following variables are set externally:
% maxepoch  -- 最大迭代次数maximum number of epochs
% numhid    -- 隐含层神经元数number of hidden units
% batchdata -- 分批后的训练数据集the data that is divided into batches (numcases numdims numbatches)
% restart   -- 如果从第1层开始学习,就置restart为1set to 1 if learning starts from beginning 

%作用:训练RBM,利用1步CD算法 直接调用权值迭代公式不使用反向传播
%可见的、二元的、随机的像素通过对称加权连接连接到隐藏的、二元的、随机的特征检测器
epsilonw      = 0.1;   % Learning rate for weights 权重学习率 alpha
epsilonvb     = 0.1;   % Learning rate for biases of visible units 可视层偏置学习率 alpha
epsilonhb     = 0.1;   % Learning rate for biases of hidden units 隐藏层偏置学习率 alpha
weightcost  = 0.0002;   %权衰减,用于防止出现过拟合
initialmomentum  = 0.5; %动量项学习率,用于克服收敛速度和算法的不稳定性之间的矛盾
finalmomentum    = 0.9;

[numcases numdims numbatches]=size(batchdata);%[numcases numdims numbatches]=[每批中的样本数 每个样本的维数 训练样本批数]

if restart ==1  %是否为重新开始即从头训练
  restart=0;
  epoch=1;

% Initializing symmetric weights and biases. 初始化权重和两层偏置
  vishid     = 0.1*randn(numdims, numhid);% 连接权值Wij 784*1000
  hidbiases  = zeros(1,numhid);% 隐含层偏置项bi
  visbiases  = zeros(1,numdims);% 可视化层偏置项aj

  poshidprobs = zeros(numcases,numhid); %样本数*隐藏层NN数,隐藏层输出p(h1|v0)对应每个样本有一个输出 100*1000
  neghidprobs = zeros(numcases,numhid);  %重构数据驱动的隐藏层
  posprods    = zeros(numdims,numhid);  % 表示p(h1|v0)*v0,用于更新Wij即<vihj>data 784*1000
  negprods    = zeros(numdims,numhid);  %<vihj>recon
  vishidinc  = zeros(numdims,numhid);  % 权值更新的增量 ΔW
  hidbiasinc = zeros(1,numhid);  % 隐含层偏置项更新的增量 1*1000 Δb
  visbiasinc = zeros(1,numdims);  % 可视化层偏置项更新的增量 1*784 Δa
  batchposhidprobs=zeros(numcases,numhid,numbatches);  % 整个数据隐含层的输出  每批样本数*隐含层维度*批数
end

for epoch = epoch:maxepoch  %每个迭代周期
 fprintf(1,‘epoch %d\r‘,epoch);
 errsum=0;
 for batch = 1:numbatches  %每一批样本
 fprintf(1,‘epoch %d batch %d\r‘,epoch,batch);
%%CD-1
%%%%%%%%% START POSITIVE PHASE 正向梯度%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
  data = batchdata(:,:,batch);  %data里是100个图片数据
  poshidprobs = 1./(1 + exp(-data*vishid - repmat(hidbiases,numcases,1)));  %隐藏层输出p(h=1|v0)=sigmod函数=1/(1+exp(-wx-b)) 根据这个分布采集一个隐变量h
  batchposhidprobs(:,:,batch)=poshidprobs;  %将输出存入一个三位数组
  posprods    = data‘ * poshidprobs;  %p(h|v0)*v0 更新权重时会使用到 计算正向梯度vh‘
  poshidact   = sum(poshidprobs);  %隐藏层中神经元概率和,在更新隐藏层偏置时会使用到
  posvisact = sum(data);  %可视层中神经元概率和,在更新可视层偏置时会使用到
%%%%%%%%% END OF POSITIVE PHASE  %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%gibbs采样
  poshidstates = poshidprobs > rand(numcases,numhid);  %将隐藏层输出01化表示,大于随机概率的置1,小于随机概率的置0,gibbs抽样,设定状态 

%%%%%%%%% START NEGATIVE PHASE 反向梯度%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
  negdata = 1./(1 + exp(-poshidstates*vishid‘ - repmat(visbiases,numcases,1)));  %01化表示之后算vt=p(vt|ht-1)重构的数据 p(v=1|h)=sigmod(W*h+a)  采集重构的可见变量v‘
  neghidprobs = 1./(1 + exp(-negdata*vishid - repmat(hidbiases,numcases,1)));    %ht=p(h|vt)使用重构数据隐藏层的输出 p(h=1|v)=sigmod(W‘*v+b) 采样一个h‘
  negprods  = negdata‘*neghidprobs;  %计算反向梯度v‘h‘;
  neghidact = sum(neghidprobs);
  negvisact = sum(negdata);
%%%%%%%%% END OF NEGATIVE PHASE %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%更新参数
  err= sum(sum( (data-negdata).^2 ));  %整批数据的误差 ||v-v‘||^2
  errsum = err + errsum;

   if epoch>5  %迭代次数不同调整冲量
     momentum=finalmomentum;
   else
     momentum=initialmomentum;
   end

%%%%%%%%% UPDATE WEIGHTS AND BIASES 更新权重和偏置%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    vishidinc = momentum*vishidinc + ...
                epsilonw*( (posprods-negprods)/numcases - weightcost*vishid); %权重的增量 ΔW=alpha*(vh‘-v‘h‘)
    visbiasinc = momentum*visbiasinc + (epsilonvb/numcases)*(posvisact-negvisact);  %可视层增量 Δa=alpha*(v-v‘)
    hidbiasinc = momentum*hidbiasinc + (epsilonhb/numcases)*(poshidact-neghidact);  %隐含层增量 Δb=alpha*(h-h‘)

    vishid = vishid + vishidinc; %a=a+Δa
    visbiases = visbiases + visbiasinc; %W=W+ΔW
    hidbiases = hidbiases + hidbiasinc;  %b=b+Δb
%%%%%%%%%%%%%%%% END OF UPDATES %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 

  end
  fprintf(1, ‘epoch %4i error %6.1f  \n‘, epoch, errsum);
end

5. 玻尔兹曼机与受限玻尔兹曼机

6. 参考文献

[1] 邱锡鹏, 神经网络与深度学习[M]. 2019.

[2] Salakhutdinov R, Hinton G. Deep boltzmann machines[C]//Artificial intelligence and statistics. 2009: 448-455.

[3] Hinton, Training a deep autoencoder or a classifieron MNIST digits. 2006.

[4] Hinton G E. Training products of experts by minimizing contrastive divergence[J]. Neural computation, 2002, 14(8): 1771-1800.

[5] Hinton G E. A practical guide to training restricted Boltzmann machines[M]//Neural networks: Tricks of the trade. Springer, Berlin, Heidelberg, 2012: 599-619.

原文地址:https://www.cnblogs.com/kailugaji/p/11595876.html

时间: 2024-10-06 17:03:52

受限玻尔兹曼机(Restricted Boltzmann Machine)的相关文章

受限玻尔兹曼机(Restricted Boltzmann Machine,RBM)代码1

环境:python 2, 32位 https://www.cnblogs.com/tuhooo/p/5440473.html 备注:这个python代码需要用到psyco包,psyco包目前只有python2 32位版本.在windows 64+python 3环境下,如果下载psyco的源代码安装,比较麻烦. https://blog.csdn.net/qq_36965134/article/details/80039026 """ Continuous Restricte

Restricted Boltzmann Machine(to be continued)

1,受限玻尔兹曼机,思想似乎是源自于热力学,因为有一个玻尔兹曼分布律的东西,具体还没学习,不过可见机器学习中不同学科的思想融合,往往是idea/innovation 的发源地. 2,想迅速入门,受知乎指引看了Hugo Larochelle在YouTube上的神经网络课第五章的Restricted Boltzmann machine. 在[5.2]中讲到了P(h|x)的条件概率的推导,感觉不错,截图如下: 然后就可以很容易地推出隐层是可见层的logistic函数了: 接着,不知道怎么和local

限制Boltzmann机(Restricted Boltzmann Machine)

限制Boltzmann机(Restricted Boltzmann Machine) 起源:Boltzmann神经网络 Boltzmann神经网络的结构是由Hopfield递归神经网络改良过来的,Hopfield中引入了统计物理学的能量函数的概念. 即,cost函数由统计物理学的能量函数给出,随着网络的训练,能量函数会逐渐变小. 可视为一动力系统,其能量函数的极小值对应系统的稳定平衡点. Hinton发明的Boltzmann中乘热打铁,对神经元输出引入了随机概率重构的概念.其想法来自于模拟退火算

《神经网络与深度学习》(四) 受限玻尔兹曼机

转自:http://blog.csdn.net/qiao1245/article/details/50932519 谈到神经网络,就不得不提到最近非常火的深度学习.那么什么是深度学习?它与神经网络的关系是什么?深度学习的基本原理和常用算法是什么?我将在这篇中详细解答. 什么是深度学习 深度学习来源于人工神经网络,它的思想是组合低层次的特征从而构建出更加抽象的深层次特征.深度学习的概念由Hinton等人于2006年提出, 它是一个新的机器学习领域,通过更"深"层次的训练,它可以更加良好的

受限波兹曼机导论Introduction to Restricted Boltzmann Machines

Suppose you ask a bunch of users to rate a set of movies on a 0-100 scale. In classical factor analysis, you could then try to explain each movie and user in terms of a set of latent factors. For example, movies like Star Wars and Lord of the Rings m

深度学习深入浅出:受限玻尔兹曼机RBM(一)基本概念

技术交流QQ群:433250724,欢迎对算法.技术.应用感兴趣的同学加入. 最近在复习经典机器学习算法的同时,也仔细看了一些深度学习的典型算法.深度学习是机器学习的"新浪潮",它的成功主要得益于深度"神经网络模型"的优异效果.这个小系列打算深入浅出地记录一下深度学习中常用的一些算法.第一篇先写一下"受限玻尔兹曼机"RBM,会分若干个小段写,这是第一段,关于RBM的基本概念. 网上有很多关于RBM的介绍,但是很多写的比较简略,跳过了很多细节,本文

受限玻尔兹曼机与推荐系统

与传统的浅层机器学习相比, 深度学习具有优秀的自动提取抽象特征的能力,并且随着分布式计算平台的快速发展,大数据的处理能力得到极大的提升,使得近年来DL在工程界得到广泛的应用,包括图像识别,语音识别,自然语言处理等领域,并取得比传统机器学习更好的效果提升.另一方面,智能推荐系统,本质上是从一堆看似杂乱无章的原始流水数据中,抽象出用户的兴趣因子,挖掘出用户的偏好,因此把深度学习与推荐系统相结合成为近年来DL发展的一个新热点,事实上,Spotify,Netflix,Facebook,Google等公司

受限玻尔兹曼机(RBM)原理总结

https://blog.csdn.net/l7H9JA4/article/details/81463954 授权转发自:刘建平<受限玻尔兹曼机(RBM)原理总结> 地址:http://www.cnblogs.com/pinard/p/6530523.html 前  言 本文主要关注于这类模型中的受限玻尔兹曼机(Restricted Boltzmann Machine,以下简称RBM),RBM模型及其推广在工业界比如推荐系统中得到了广泛的应用.章节目录 RBM模型结构 RBM概率分布 RBM模

【转载】Boltzmann machine

Boltzmann machine Geoffrey E. Hinton (2007), Scholarpedia, 2(5):1668. doi:10.4249/scholarpedia.1668 revision #91075 [link to/cite this article] Post-publication activity Curator: Geoffrey E. Hinton Dr. Geoffrey E. Hinton, University of Toronto, CANAD