Deep Learning: Assuming a deep neural network is properly regulated, can adding more layers actually make the performance degrade?

Deep Learning: Assuming a deep neural network is properly regulated, can adding more layers actually make the performance degrade?

I found this to be really puzzling. A deeper NN is supposed to be more powerful or at least equal to a shallower NN. I have already used dropout to prevent overfitting. How can the performance be degraded?

Re-Ask

Follow17

Yoshua‘s Answer

View 2 Other Answers

Yoshua Bengio, My lab has been one of the three that started the deep learning approach, bac...

Upvoted by Prateek Tandon, Robotics and Strong Artificial Intelligence Researcher• Paul King, Computational Neuroscientist, Technology Entrepreneur • Jack Rae,Google DeepMind Research Engineer

Yoshua has 25 endorsements in Deep Learning.

If you do not change the size of the layers and just add more layers, capacity should increase, so you could be overfitting. However, you should check whether training error increases or decreases. If it increases (which is also very plausible), it means that adding the layer made the optimization harder, with the optimization methods and initialization that you are using.  That could also explain your problem. However, if training error decreases and test error increases, you are overfitting.

时间： 2024-08-13 11:09:29

Deep Learning: Assuming a deep neural network is properly regulated, can adding more layers actually make the performance degrade?的相关文章

【转帖】【面向代码】学习 Deep Learning（一）Neural Network

最近一直在看Deep Learning,各类博客.论文看得不少 但是说实话,这样做有些疏于实现,一来呢自己的电脑也不是很好,二来呢我目前也没能力自己去写一个toolbox 只是跟着Andrew Ng的UFLDL tutorial 写了些已有框架的代码(这部分的代码见github) 后来发现了一个matlab的Deep Learning的toolbox,发现其代码很简单,感觉比较适合用来学习算法 再一个就是matlab的实现可以省略掉很多数据结构的代码,使算法思路非常清晰 所以我想在解读这个too

Deep Learning 论文笔记 (2): Neural network regularization via robust weight factorization

under review as a conference paper at ICLR 2015. Motivation: 本文提出来一种regularization的方法,叫做FaMe (Factored Mean training). The proposed FaMe model aims to apply a similar strategy, yet learns a factorization of each weight matrix such that the factors ar

Andrew Ng 的 Machine Learning 课程学习 (week5) Neural Network Learning

这学期一直在跟进 Coursera上的 Machina Learning 公开课, 老师Andrew Ng是coursera的创始人之一,Machine Learning方面的大牛.这门课程对想要了解和初步掌握机器学习的人来说是不二的选择.这门课程涵盖了机器学习的一些基本概念和方法,同时这门课程的编程作业对于掌握这些概念和方法起到了巨大的作用. 课程地址 https://www.coursera.org/learn/machine-learning 笔记主要是简要记录下课程内容,以及MATLAB

Deep learning与Neural Network

该文章转自深度学习微信公众号 深度学习是机器学习研究中的一个新的领域,其动机在于建立.模拟人脑进行分析学习的神经网络,它模仿人脑的机制来解释数据,例如图像,声音和文本.深度学习是无监督学习的一种. 深度学习的概念源于人工神经网络的研究.含多隐层的多层感知器就是一种深度学习结构.深度学习通过组合低层特征形成更加抽象的高层表示属性类别或特征,以发现数据的分布式特征表示. Deep learning本身算是machine learning的一个分支,简单可以理解为neural network的发展.大

[C3] Andrew Ng - Neural Networks and Deep Learning

About this Course If you want to break into cutting-edge AI, this course will help you do so. Deep learning engineers are highly sought after, and mastering deep learning will give you numerous new career opportunities. Deep learning is also a new "s

机器学习001 deeplearning.ai 深度学习课程 Neural Networks and Deep Learning 第一周总结

Deep Learning Specialization 吴恩达老师最近在coursera上联合deeplearning.ai 推出了有关深度学习的一系列课程,相对于之前的machine learning课程,这次的课程更加实用,作业语言也有matlab改为了python从而更加贴合目前的趋势.在此将对这个系列课程做一个学习笔记. 而这次的Deep Learning Specialization分为五门课程,分别为:Neural Networks and Deep Learning,Improv

Neural Networks and Deep Learning

Neural Networks and Deep Learning This is the first course of the deep learning specialization at Coursera which is moderated by moderated by DeepLearning.ai. The course is taught by Andrew Ng. Introduction to deep learning Be able to explain the maj

(转) Deep Learning in a Nutshell: Reinforcement Learning

Deep Learning in a Nutshell: Reinforcement Learning Share: Posted on September 8, 2016by Tim Dettmers No CommentsTagged Deep Learning, Deep Neural Networks, Machine Learning,Reinforcement Learning This post is Part 4 of the Deep Learning in a Nutshel

Deep Learning（深度学习）学习笔记整理系列之（三）

Deep Learning(深度学习)学习笔记整理系列 [email protected] http://blog.csdn.net/zouxy09 作者:Zouxy version 1.0 2013-04-08 声明: 1)该Deep Learning的学习系列是整理自网上很大牛和机器学习专家所无私奉献的资料的.具体引用的资料请看参考文献.具体的版本声明也参考原文献. 2)本文仅供学术交流,非商用.所以每一部分具体的参考资料并没有详细对应.如果某部分不小心侵犯了大家的利益,还望海涵,并联系博主