Linear Regression ----- Stanford Machine Learning（by Andrew NG）Course Notes

Andrew NG的Machine learning课程地址为：https://www.coursera.org/course/ml

在Linear Regression部分出现了一些新的名词，这些名词在后续课程中会频繁出现：

Model
Representation

　　m: number of
training examples

　　x⁽ⁱ⁾:
input (features) of i^th training example

　　x_j⁽ⁱ⁾:
value of feature j in i^th training
example

　　y⁽ⁱ⁾:
“output” variable / “target” variable of i^th training
example

　　n:
number of features

　　θ:
parameters

　　Hypothesis:
h_θ(x) = θ₀ +
θ₁x₁ +
θ₂x₂ + …
+θ_nx_n

Cost
Function

 
IDEA: Choose θso that h_θ(x) is close to y for our training
examples (x, y).

A.Linear Regression with One Variable Cost
Function

Cost Function:    

Goal:    

Contour Plot:

B.Linear
Regression with Multiple Variable Cost Function

Cost Function:  

Goal:   

Gradient
Descent 

Outline

Gradient
Descent Algorithm

迭代过程收敛图可能如下：

(此为等高线图，中间为最小值点，图中蓝色弧线为可能的收敛路径。)

Learning
Rate α:

1) If α is too small, gradient descent can be slow to
converge;

2) If α is too large, gradient descent may not decrease on every
iteration or may not converge;

3) For sufficiently small α , J(θ) should decrease on
every iteration;

Choose
Learning Rate α: Debug, 0.001, 0.003, 0.006,
0.01, 0.03, 0.06, 0.1, 0.3, 0.6,
1.0;

“Batch”
Gradient Descent: Each step of gradient descent uses all the training
examples;

“Stochastic” gradient descent:
Each step of gradient descent uses only one training
examples.

Normal Equation

IDEA:
Method to solve for θ analytically.

 for
every j, then   

Restriction:
Normal Equation does not work when (X^TX) is
non-invertible.

PS:
当矩阵为满秩矩阵时，该矩阵可逆。列向量（feature）线性无关且行向量（样本）线性无关的个数大于列向量的个数（特征个数n）.

Gradient
Descent Algorithm VS. Normal Equation 

Gradient
Descent:

a) Need to choose α;

 
        b) Needs many iterations;

c) Works well even when n is large;
(n > 1000 is appropriate)

Normal
Equation:

a) No need to choose α;

b) Don’t need to iterate;

c) Need to compute ;

d) Slow if n is very large. (n
< 1000 is OK)

Feature Scaling

IDEA:
Make sure features are on a similar scale.

好处:
减少迭代次数，有利于快速收敛

Example:
If we need to get every feature into approximately a -1
≤ x_i ≤ 1 range, feature values located in [-3,
3] or [-1/3, 1/3] fields are acceptable.

Mean
normalization:  

HOMEWORK

好了，既然看完了视频课程，就来做一下作业吧，下面是Linear Regression部分作业的核心代码：

1.computeCost.m/computeCostMulti.m

J=1/(2*m)*sum((theta‘*X‘-y‘).^2);

2.gradientDescent.m/gradientDescentMulti.m

h=X*theta-y;

v=X‘*h;

v=v*alpha/m;

theta1=theta;

theta=theta-v;

Linear Regression ----- Stanford Machine Learning（by Andrew
NG）Course Notes,码迷,mamicode.com

Linear Regression ----- Stanford Machine Learning（by Andrew
NG）Course Notes