<!doctype html>
反向传播算法html {overflow-x: initial !important;}:root { --bg-color:#ffffff; --text-color:#333333; --select-text-bg-color:#B5D6FC; --select-text-font-color:auto; --monospace:"Lucida Console",Consolas,"Courier",monospace; }
html { font-size: 14px; background-color: var(--bg-color); color: var(--text-color); font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; -webkit-font-smoothing: antialiased; }
body { margin: 0px; padding: 0px; height: auto; bottom: 0px; top: 0px; left: 0px; right: 0px; font-size: 1rem; line-height: 1.42857; overflow-x: hidden; background: inherit; tab-size: 4; }
iframe { margin: auto; }
a.url { word-break: break-all; }
a:active, a:hover { outline: 0px; }
.in-text-selection, ::selection { text-shadow: none; background: var(--select-text-bg-color); color: var(--select-text-font-color); }
write { margin: 0px auto; height: auto; width: inherit; word-break: normal; word-wrap: break-word; position: relative; white-space: normal; overflow-x: visible; padding-top: 40px; }
write.first-line-indent p { text-indent: 2em; }
write.first-line-indent li p, #write.first-line-indent p * { text-indent: 0px; }
write.first-line-indent li { margin-left: 2em; }
.for-image #write { padding-left: 8px; padding-right: 8px; }
body.typora-export { padding-left: 30px; padding-right: 30px; }
.typora-export .footnote-line, .typora-export li, .typora-export p { white-space: pre-wrap; }
@media screen and (max-width: 500px) {
body.typora-export { padding-left: 0px; padding-right: 0px; }
#write { padding-left: 20px; padding-right: 20px; }
.CodeMirror-sizer { margin-left: 0px !important; }
.CodeMirror-gutters { display: none !important; }
}
write li > figure:first-child { margin-top: -20px; }
write ol, #write ul { position: relative; }
img { max-width: 100%; vertical-align: middle; }
button, input, select, textarea { color: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; font-stretch: inherit; font-size: inherit; line-height: inherit; font-family: inherit; }
input[type="checkbox"], input[type="radio"] { line-height: normal; padding: 0px; }
*, ::after, ::before { box-sizing: border-box; }
write h1, #write h2, #write h3, #write h4, #write h5, #write h6, #write p, #write pre { width: inherit; }
write h1, #write h2, #write h3, #write h4, #write h5, #write h6, #write p { position: relative; }
h1, h2, h3, h4, h5, h6 { break-after: avoid-page; break-inside: avoid; orphans: 2; }
p { orphans: 4; }
h1 { font-size: 2rem; }
h2 { font-size: 1.8rem; }
h3 { font-size: 1.6rem; }
h4 { font-size: 1.4rem; }
h5 { font-size: 1.2rem; }
h6 { font-size: 1rem; }
.md-math-block, .md-rawblock, h1, h2, h3, h4, h5, h6, p { margin-top: 1rem; margin-bottom: 1rem; }
.hidden { display: none; }
.md-blockmeta { color: rgb(204, 204, 204); font-weight: 700; font-style: italic; }
a { cursor: pointer; }
sup.md-footnote { padding: 2px 4px; background-color: rgba(238, 238, 238, 0.7); color: rgb(85, 85, 85); border-radius: 4px; cursor: pointer; }
sup.md-footnote a, sup.md-footnote a:hover { color: inherit; text-transform: inherit; text-decoration: inherit; }
write input[type="checkbox"] { cursor: pointer; width: inherit; height: inherit; }
figure { overflow-x: auto; margin: 1.2em 0px; max-width: calc(100% + 16px); padding: 0px; }
figure > table { margin: 0px !important; }
tr { break-inside: avoid; break-after: auto; }
thead { display: table-header-group; }
table { border-collapse: collapse; border-spacing: 0px; width: 100%; overflow: auto; break-inside: auto; text-align: left; }
table.md-table td { min-width: 32px; }
.CodeMirror-gutters { border-right: 0px; background-color: inherit; }
.CodeMirror-linenumber { user-select: none; }
.CodeMirror { text-align: left; }
.CodeMirror-placeholder { opacity: 0.3; }
.CodeMirror pre { padding: 0px 4px; }
.CodeMirror-lines { padding: 0px; }
div.hr:focus { cursor: none; }
write pre { white-space: pre-wrap; }
write.fences-no-line-wrapping pre { white-space: pre; }
write pre.ty-contain-cm { white-space: normal; }
.CodeMirror-gutters { margin-right: 4px; }
.md-fences { font-size: 0.9rem; display: block; break-inside: avoid; text-align: left; overflow: visible; white-space: pre; background: inherit; position: relative !important; }
.md-diagram-panel { width: 100%; margin-top: 10px; text-align: center; padding-top: 0px; padding-bottom: 8px; overflow-x: auto; }
write .md-fences.mock-cm { white-space: pre-wrap; }
.md-fences.md-fences-with-lineno { padding-left: 0px; }
write.fences-no-line-wrapping .md-fences.mock-cm { white-space: pre; overflow-x: auto; }
.md-fences.mock-cm.md-fences-with-lineno { padding-left: 8px; }
.CodeMirror-line, twitterwidget { break-inside: avoid; }
.footnotes { opacity: 0.8; font-size: 0.9rem; margin-top: 1em; margin-bottom: 1em; }
.footnotes + .footnotes { margin-top: 0px; }
.md-reset { margin: 0px; padding: 0px; border: 0px; outline: 0px; vertical-align: top; background: 0px 0px; text-decoration: none; text-shadow: none; float: none; position: static; width: auto; height: auto; white-space: nowrap; cursor: inherit; -webkit-tap-highlight-color: transparent; line-height: normal; font-weight: 400; text-align: left; box-sizing: content-box; direction: ltr; }
li div { padding-top: 0px; }
blockquote { margin: 1rem 0px; }
li .mathjax-block, li p { margin: 0.5rem 0px; }
li { margin: 0px; position: relative; }
blockquote > :last-child { margin-bottom: 0px; }
blockquote > :first-child, li > :first-child { margin-top: 0px; }
.footnotes-area { color: rgb(136, 136, 136); margin-top: 0.714rem; padding-bottom: 0.143rem; white-space: normal; }
write .footnote-line { white-space: pre-wrap; }
@media print {
body, html { border: 1px solid transparent; height: 99%; break-after: avoid; break-before: avoid; }
#write { margin-top: 0px; padding-top: 0px; border-color: transparent !important; }
.typora-export * { -webkit-print-color-adjust: exact; }
html.blink-to-pdf { font-size: 13px; }
.typora-export #write { padding-left: 32px; padding-right: 32px; padding-bottom: 0px; break-after: avoid; }
.typora-export #write::after { height: 0px; }
@page { margin: 20mm 0px; }
}
.footnote-line { margin-top: 0.714em; font-size: 0.7em; }
a img, img a { cursor: pointer; }
pre.md-meta-block { font-size: 0.8rem; min-height: 0.8rem; white-space: pre-wrap; background: rgb(204, 204, 204); display: block; overflow-x: hidden; }
p > .md-image:only-child:not(.md-img-error) img, p > img:only-child { display: block; margin: auto; }
p > .md-image:only-child { display: inline-block; width: 100%; }
write .MathJax_Display { margin: 0.8em 0px 0px; }
.md-math-block { width: 100%; }
.md-math-block:not(:empty)::after { display: none; }
[contenteditable="true"]:active, [contenteditable="true"]:focus { outline: 0px; box-shadow: none; }
.md-task-list-item { position: relative; list-style-type: none; }
.task-list-item.md-task-list-item { padding-left: 0px; }
.md-task-list-item > input { position: absolute; top: 0px; left: 0px; margin-left: -1.2em; margin-top: calc(1em - 10px); border: none; }
.math { font-size: 1rem; }
.md-toc { min-height: 3.58rem; position: relative; font-size: 0.9rem; border-radius: 10px; }
.md-toc-content { position: relative; margin-left: 0px; }
.md-toc-content::after, .md-toc::after { display: none; }
.md-toc-item { display: block; color: rgb(65, 131, 196); }
.md-toc-item a { text-decoration: none; }
.md-toc-inner:hover { text-decoration: underline; }
.md-toc-inner { display: inline-block; cursor: pointer; }
.md-toc-h1 .md-toc-inner { margin-left: 0px; font-weight: 700; }
.md-toc-h2 .md-toc-inner { margin-left: 2em; }
.md-toc-h3 .md-toc-inner { margin-left: 4em; }
.md-toc-h4 .md-toc-inner { margin-left: 6em; }
.md-toc-h5 .md-toc-inner { margin-left: 8em; }
.md-toc-h6 .md-toc-inner { margin-left: 10em; }
@media screen and (max-width: 48em) {
.md-toc-h3 .md-toc-inner { margin-left: 3.5em; }
.md-toc-h4 .md-toc-inner { margin-left: 5em; }
.md-toc-h5 .md-toc-inner { margin-left: 6.5em; }
.md-toc-h6 .md-toc-inner { margin-left: 8em; }
}
a.md-toc-inner { font-size: inherit; font-style: inherit; font-weight: inherit; line-height: inherit; }
.footnote-line a:not(.reversefootnote) { color: inherit; }
.md-attr { display: none; }
.md-fn-count::after { content: "."; }
code, pre, samp, tt { font-family: var(--monospace); }
kbd { margin: 0px 0.1em; padding: 0.1em 0.6em; font-size: 0.8em; color: rgb(36, 39, 41); background: rgb(255, 255, 255); border: 1px solid rgb(173, 179, 185); border-radius: 3px; box-shadow: rgba(12, 13, 14, 0.2) 0px 1px 0px, rgb(255, 255, 255) 0px 0px 0px 2px inset; white-space: nowrap; vertical-align: middle; }
.md-comment { color: rgb(162, 127, 3); opacity: 0.8; font-family: var(--monospace); }
code { text-align: left; vertical-align: initial; }
a.md-print-anchor { white-space: pre !important; border-width: initial !important; border-style: none !important; border-color: initial !important; display: inline-block !important; position: absolute !important; width: 1px !important; right: 0px !important; outline: 0px !important; background: 0px 0px !important; text-decoration: initial !important; text-shadow: initial !important; }
.md-inline-math .MathJax_SVG .noError { display: none !important; }
.html-for-mac .inline-math-svg .MathJax_SVG { vertical-align: 0.2px; }
.md-math-block .MathJax_SVG_Display { text-align: center; margin: 0px; position: relative; text-indent: 0px; max-width: none; max-height: none; min-height: 0px; min-width: 100%; width: auto; overflow-y: hidden; display: block !important; }
.MathJax_SVG_Display, .md-inline-math .MathJax_SVG_Display { width: auto; margin: inherit; display: inline-block !important; }
.MathJax_SVG .MJX-monospace { font-family: var(--monospace); }
.MathJax_SVG .MJX-sans-serif { font-family: sans-serif; }
.MathJax_SVG { display: inline; font-style: normal; font-weight: 400; line-height: normal; zoom: 90%; text-indent: 0px; text-align: left; text-transform: none; letter-spacing: normal; word-spacing: normal; word-wrap: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0px; min-height: 0px; border: 0px; padding: 0px; margin: 0px; }
.MathJax_SVG * { transition: none; }
.MathJax_SVG_Display svg { vertical-align: middle !important; margin-bottom: 0px !important; }
.os-windows.monocolor-emoji .md-emoji { font-family: "Segoe UI Symbol", sans-serif; }
.md-diagram-panel > svg { max-width: 100%; }
[lang="mermaid"] svg, [lang="flow"] svg { max-width: 100%; }
[lang="mermaid"] .node text { font-size: 1rem; }
table tr th { border-bottom: 0px; }
video { max-width: 100%; display: block; margin: 0px auto; }
iframe { max-width: 100%; width: 100%; border: none; }
.highlight td, .highlight tr { border: 0px; }
:root { --active-file-bg-color: rgba(32, 43, 51, 0.63); --active-file-text-color: white; --bg-color: #fff; --text-color: #333; --side-bar-bg-color: #f5f5f5; --control-text-color: #666; }
html { color: rgb(51, 51, 51); background: rgb(255, 255, 255); text-size-adjust: 100%; text-rendering: optimizeLegibility; font-size: 14px; -webkit-font-smoothing: initial; }
write { max-width: 960px; padding-top: 2em; padding-left: 60px; padding-right: 60px; min-height: calc(100vh - 6em); -webkit-font-smoothing: antialiased; font-size: 16px; }
.typora-node #write { min-height: calc(100% - 6em); }
pre.md-meta-block { background: rgb(245, 245, 245); padding: 1em; border-radius: 3px; font-size: 14px; }
@media screen and (max-width: 800px) {
html { font-size: 14px; }
#write { padding-left: 30px; padding-right: 30px; font-size: 14px; }
}
@media screen and (min-width: 1100px) {
body, #footer-word-count-info { background: rgb(245, 245, 245); }
body.pin-outline, .pin-outline #footer-word-count-info, .pin-outline footer { background: rgb(255, 255, 255); }
#write { max-width: 1000px; padding: 40px 60px; background: rgb(255, 255, 255); margin: 3em auto; border-style: solid; border-color: rgb(221, 221, 221); border-image: initial; border-width: 0px 1px; }
.pin-outline #write { max-width: 1000px; background: rgb(255, 255, 255); margin: 0px; border: 0px; padding-left: 60px; padding-right: 60px; }
footer { background-color: transparent; }
}
@media screen and (min-width: 1300px) {
body.pin-outline, .pin-outline #footer-word-count-info, .pin-outline footer { background: rgb(245, 245, 245); }
.pin-outline #write { max-width: 1000px; padding: 40px 60px; background: rgb(255, 255, 255); margin: 3em auto; border-style: solid; border-color: rgb(221, 221, 221); border-image: initial; border-width: 0px 1px; }
.pin-outline footer { background-color: transparent; }
#footer-word-count-info { background: rgb(245, 245, 245); }
}
html.borderbox *, html.borderbox ::before, html.borderbox ::after { box-sizing: border-box; }
body, dl, dt, dd, ul, ol, li, h1, h2, h3, h4, h5, h6, code, form, fieldset, legend, input, textarea, p, blockquote, th, td, hr, button, article, aside, details, figcaption, figure, footer, header, menu, nav, section { margin: 0px; padding: 0px; }
article, aside, details, figcaption, figure, footer, header, menu, nav, section { display: block; }
audio, canvas, video { display: inline-block; }
body, button, input, select, textarea { font-style: normal; font-variant: normal; font-weight: 300; font-stretch: normal; font-size: 1em; line-height: 1.8; font-family: "PingFang SC", "Lantinghei SC", "Microsoft Yahei", "Hiragino Sans GB", "Microsoft Sans Serif", "WenQuanYi Micro Hei", sans; }
body { font-family: "PingFang SC", "Lantinghei SC", "Microsoft Yahei", "Hiragino Sans GB", "Microsoft Sans Serif", "WenQuanYi Micro Hei", sans; }
h1, h2, h3, h4, h5, h6 { font-family: TimesNewRomanPS-ItalicMT, "PingFang SC", "Lantinghei SC", "Microsoft Yahei", "Hiragino Sans GB", "Microsoft Sans Serif", "WenQuanYi Micro Hei", sans; -webkit-font-smoothing: initial; font-weight: 100; color: var(--text-color); line-height: 1.35; font-variant-numeric: lining-nums; margin-bottom: 1em; }
em { font-family: Georgia-Italic, STSongti-SC-Light, serif; }
strong em, em strong { font-family: Georgia-BoldItalic, STSongti-SC-Regular, serif; }
table { border-collapse: collapse; border-spacing: 0px; }
fieldset, img { border: 0px; }
blockquote { position: relative; color: rgb(153, 153, 153); font-weight: 400; border-left: 1px solid rgb(26, 188, 156); padding-left: 1em; margin: 1em 3em 1em 2em; }
@media only screen and (max-width: 640px) {
blockquote { margin: 1em 0px; }
}
acronym, abbr { border-bottom: 1px dotted; font-variant: normal; }
abbr { cursor: help; }
address, caption, cite, code, dfn, th, var { font-style: normal; font-weight: 400; }
ul, ol { list-style: none; }
caption, th { text-align: left; }
q::before, q::after { content: ""; }
sub, sup { font-size: 75%; line-height: 0; position: relative; }
:root sub, :root sup { vertical-align: baseline; }
sup { top: -0.5em; }
sub { bottom: -0.25em; }
a { color: rgb(26, 188, 156); }
a:hover { text-decoration: underline; }
write a { border-bottom: 1px solid rgb(26, 188, 156); }
write a:hover { border-bottom-color: rgb(85, 85, 85); color: rgb(85, 85, 85); text-decoration: none; }
ins, a { text-decoration: none; }
mark { background: rgb(255, 253, 209); border-bottom: 1px solid rgb(255, 237, 206); padding: 2px; margin: 0px 5px; }
pre, code, pre tt { font-family: Courier, "Courier New", monospace; }
write .md-fences { border: 1px solid rgb(221, 221, 221); padding: 1em 0.5em; display: block; }
hr { border-top: none; border-right: none; border-left: none; border-image: initial; border-bottom: 1px solid rgb(207, 207, 207); margin-bottom: 0.8em; height: 10px; }
write strong { font-weight: bolder; color: rgb(0, 0, 0); }
.code-tooltip.md-hover-tip strong { color: white; }
write p, #write .md-fences, #write ul, #write ol, #write dl, #write form, #write hr, #write figure, #write-p, #write-pre, #write-ul, #write-ol, #write-dl, #write-form, #write-hr, #write-table, blockquote { margin-bottom: 1.2em; }
html { font-family: "PingFang SC", Verdana, "Helvetica Neue", "Microsoft Yahei", "Hiragino Sans GB", "Microsoft Sans Serif", "WenQuanYi Micro Hei", sans-serif; }
write h1, #write h2, #write h3, #write h4, #write h5, #write h6, #write-h1, #write-h2, #write-h3, #write-h4, #write-h5, #write-h6 { margin-top: 1.2em; margin-bottom: 0.6em; line-height: 1.35; color: rgb(0, 0, 0); }
write h1, #write-h1 { font-size: 2.4em; padding-bottom: 1em; border-bottom: 3px double rgb(238, 238, 238); }
write h2, #write-h2 { font-size: 1.8em; }
write h3, #write-h3 { font-size: 1.6em; }
write h4, #write-h4 { font-size: 1.4em; }
write h5, #write h6, #write-h5, #write-h6 { font-size: 1.2em; }
write ul, #write-ul { margin-left: 1.3em; list-style: disc; }
write ol, #write-ol { list-style: decimal; margin-left: 1.9em; }
write li ul, #write li ol, #write-ul ul, #write-ul ol, #write-ol ul, #write-ol ol { margin-bottom: 0.8em; margin-left: 2em; }
write li ul, #write-ul ul, #write-ol ul { list-style: circle; }
write table th, #write table td { border: 1px solid rgb(221, 221, 221); padding: 0.5em 1em; color: rgb(102, 102, 102); }
write table .md-table-edit th { border: none; padding: 0px; color: inherit; }
write table th, #write-table th { background: rgb(251, 251, 251); }
write table thead th, #write-table thead th { background: rgb(241, 241, 241); }
write table caption { border-bottom: none; }
write em { font-weight: inherit; font-style: inherit; }
li > p { margin-bottom: 0px !important; }
write img { max-width: 100%; }
a.md-toc-inner { border-bottom: 0px !important; }
.md-toc-h1:first-of-type:last-of-type { display: none; }
.md-toc { font-size: inherit; }
.md-toc-h1 .md-toc-inner { font-weight: normal; }
.md-table-edit th { padding: 0px !important; border: 0px !important; }
.mac-seamless-mode #write { min-height: calc((100vh - 6em) - 20px); }
.typora-quick-open-item.active { color: var(--active-file-text-color); }
.in-text-selection, ::selection { background: var(--active-file-bg-color); text-shadow: none; color: white; }
.btn-primary { background-color: rgb(45, 45, 45); border-color: rgb(2, 2, 2); }
.btn-primary:hover, .btn-primary:focus, .btn-primary.focus, .btn-primary:active, .btn-primary.active, .open > .dropdown-toggle.btn-primary { background-color: rgb(78, 76, 78); border: rgb(78, 76, 78); }
preference-dialog .modal-content { background: rgb(110, 117, 122); --bg-color: #6e757a; --text-color: #f1f1f1; color: rgb(241, 241, 241); }
typora-source, .typora-sourceview-on { --bg-color: #eee; background: rgb(238, 238, 238); }
.cm-s-typora-default .cm-header, .cm-s-typora-default .cm-property { color: rgb(17, 96, 152); }
.cm-s-typora-default .cm-link { color: rgb(17, 152, 125); }
.cm-s-typora-default .cm-em { font-family: Georgia-Italic, STSongti-SC-Light, serif; color: rgb(111, 100, 0); }
.cm-s-typora-default .cm-em { color: rgb(0, 22, 45); }
.CodeMirror.cm-s-typora-default div.CodeMirror-cursor { border-left: 3px solid rgb(110, 117, 122); }
.cm-s-typora-default .CodeMirror-selectedtext, .typora-sourceview-on .CodeMirror-focused .CodeMirror-selected { background: rgb(110, 117, 122); color: white; }
.file-node-icon.fa.fa-folder::before { color: rgba(32, 43, 51, 0.49); }
preference-dialog .megamenu-menu-panel h1 { margin-bottom: 1em; }
::-webkit-scrollbar-corner { display: none; background: transparent; }
.typora-export li, .typora-export p, .typora-export, .footnote-line {white-space: normal;}
反向传播算法
参考：http://neuralnetworksanddeeplearning.com/chap2.html
BP 算法所关注的是损失函数 C 关于神经网络每一个权重 w 和偏置 b 的偏导。BP 算法向我们展示权值和偏置对损失值的影响。BP 不仅仅是一个快速的算法，其同时为我们提供了一个视角，让我们观察权值和偏置是如何影响网络输出的。
热身：使用矩阵计算网络输出
先介绍一个网络权重和偏置值的数学表示：，左侧的数学表达式表示第 层网络的第 个元素和第 层第 个元素之间的权重。同样的， 表示第 层第个元素的偏置值， 表示 层第 个元素的激活函数输出。利用这种数学表示， 可以表示为：
使用矩阵形式表示上述表达式：
定义 为激活函数的输入值则可以将上面表达式 表示为：
损失函数的两个要求
BP 算法用于计算网络中所有权重 和偏置 关于损失函数 的偏导数 和。为了使BP算法正常运行，我们需要对损失函数做两条约束。我们先给出一个常用的均方差损失函数，如下式：
表达式 中元素解释如下， 是训练网络的样本个数； 是训练样本 的期望值（也就是样本的标签）； 标识着网络的层数； 是网络在输入为 时输出层激活函数的输出。
现在描述我们对损失函数的要求。首先，损失函数可以写成所有训练样本损失值均值的形式： 。
我们做上面的要求是因为训练的过程中我们常常使用批训练的方式，而不是每次只训练一个样本。批训练之后我们求当前批次样本损失值的平均数来更新权重和偏置。
其次，损失函数可以使用网络输出层作为参数：， 是网络输出层的输出。以均方差损失函数为例，当样本为时，网络的损失值为：
上式中所有元素的值都是已知的， 是标签、 是网络输出层的输出。
Hadamard 积，
Hadamardd 积（哈达玛积）表示矩阵按对应元素做乘法：，例如：
BP 算法依赖的四个方程
BP 算法用于计算网络中权值与偏置关于损失值的偏导，也就是计算：和 。在给出计算方法前我们先引入一个中间变量 ，这个变量表示网络第 层第 个元素的值对整个网络损失的贡献。BP 算法可以帮我们计算出 ，然后我们就可以得到和 。
为了便于理解，这里我们假设网络中的某个神经元（第 层 第 个）旁边住着一个小恶魔，它可以帮我们调整神经元的输出。小恶魔不出手时这个神经元的输入为：，小恶魔出手后，当前神经元的输入为：，输出为 。小恶魔对当前神经元的调整最终将影响整个网络的输出，小恶魔出手后对整个网络损失值的影响为：。
如果这是一个善良的小恶魔，那么它可以帮我们调整输入一减少损失函数的值。如果 的值不为0，那么调整，将减小整个网络的损失值。如果的值为0，那么小恶魔的调整对网络损失的贡献就非常小，除非比较大。
我们定义如下：
时的矩阵表示方法为：。BP 算法可以帮助我们计算网络中每一层的 。
?
?
?
?
?
?
原文地址：https://www.cnblogs.com/jiahu-Blog/p/11408982.html