Machine Learning - WEEK 1 2 3- 线性回归、逻辑回归、梯度下降法及其优化算法、传统方法、 Octave 入门

本文为个人笔记，只记了重要内容，不适合新手入手

线性回归

样本 $(x^{(i)}, y^{(i)}) ， i \in 1, 2, \dots, m$
$x^{(i)} = (x_{1}^{(i)}, x_{2}^{(i)}, \dots, x_{n}^{(i)})$
假想函数（目标函数）：
$h θ (x (i)) = θ 0 + θ 1 x (i) 1 + θ 2 x (i) 2 + \dots + θ n$
$h_{θ} (x^{(i)})$
线性回归 中 线性 的含义是参数 $θ_{j}$
$θ = (θ_{0}, θ_{1}, θ_{2}, \dots, θ_{n})$
为此我们提出了一个衡量参数取值好坏的函数——代价函数：
$J (θ) = 1 2 m \sum i = 1 m ( h θ ( x ( i ) ) - y ( i ) ) 2$
现在问题转变为了求使得代价函数 $J (θ)$

梯度下降法

时间复杂度 $O (k n^{2})$

θ j := θ j - α \partial J ( θ ) \partial θ j

$α$
注意要先求出所有的 $t e m p_{j} = θ_{j} - α \frac{\partial J (θ)}{\partial θ_{j}}$

若取 $x_{0}^{(i)} = 1$

θ j := θ j - α m \sum i = 1 m [ ( h θ ( x ( i ) ) - y ( i ) )

用矩阵来表述的话：

θ := θ - α m [ X T \cdot ( X \cdot θ - y ) ]

$θ \in R^{(n + 1) \times 1}, X \in R^{m \times (n + 1)}, y \in R^{m \times 1}$

特征优化：

尽量让 $- 1 < x_{i} < 1$

x i := x i - μ i s i

$μ_{i}$

传统方法

时间复杂度 $O (n^{3})$

直接令 $\frac{\partial J (θ)}{\partial θ_{i}} = 0$

θ = (X T X) - 1 X T y

* $θ \in R^{(n + 1) \times 1}, X \in R^{m \times (n + 1)}, y \in R^{m \times 1}$

code

featureNormalize

function [X_norm, mu, sigma] = featureNormalize(X)
mu = zeros(1, size(X, 2));
sigma = zeros(1, size(X, 2));
for iter = 1:size(X, 2)
    mu(iter) = mean(X(:, iter));
    sigma(iter) = std(X(:, iter));
    X_norm(:, iter) = (X(:, iter) - mu(iter)) / sigma(iter);
end
end

computeCostMulti

function J = computeCost(X, y, theta)
m = length(y); % number of training examples
J = 1 / (2 * m) * sum((X * theta - y) .^ 2);
end

gradientDescentMulti

function [theta, J_history] = gradientDescentMulti(X, y, theta, alpha, num_iters)
m = length(y); % number of training examples
J_history = zeros(num_iters, 1);
for iter = 1:num_iters
    theta -= alpha / m * X' * (X * theta - y);   
    J_history(iter) = computeCostMulti(X, y, theta);
end
end

normalEqn

function [theta] = normalEqn(X, y)
theta = zeros(size(X, 2), 1);
theta = pinv(X' * X) * X' * y;
end

Octave

https://www.gnu.org/software/octave/

Ubuntu Install:

sudo apt-add-repository ppa:octave/stable
sudo apt-get update
sudo apt-get install octave

Octave 入门:

逻辑回归

线性回归是用来预测某个点的取值，逻辑回归是预测某个点具有某种特征的概率

为了达到我们的目的，重现建立模型：

h θ (x) g (z) J (θ) = g (θ T x) = 1 1 +

C o s t (h θ (x, y)) = ⎧⎩⎨⎪⎪ - l o g (h θ (x)) - l o g (

$θ \in R^{(n + 1) \times 1}, X \in R^{m \times (n + 1)}, y \in R^{m \times 1}$

可以写成：

C o s t (h θ (x, y)) = - [y \cdot l o g (h θ (x)) + (1 - y) \cdot l o g (1 - h θ (

对代价函数求偏倒后发现和线性回归代价函数求偏倒的结果形式上是完全一样的:

\partial J ( θ ) \partial θ j = 1 m \sum i = 1 m [ ( h θ ( x ( i ) ) - y

cost function code

function [J, grad] = costFunction(theta, X, y)
m = length(y); % number of training examples
tmp = sigmoid(X*theta);
J = -1 / m * (y'*log(tmp)+(1-y)'*log(1-tmp));
grad = 1 / m * X' * (sigmoid(X * theta) - y);
end

高级优化算法

Optimization algorithms:
- Gradient descent
- Conjugate gradient
- BFGS
- L-BFGS

Advantages:
- No need to manually pick $α$

disadvantages:
- More complex

调用 Octave 优化算法 example :

[first] 定义 cost function:

costFunction.m

function [jval, gradient] = costFunction(theta, X, y)
    % jval := J(theta)
    % gradient := grad J(theta)

[then] 键入命令

options = optimset('GrandObj', 'on', 'MaxIter', '100');
initialTheta = zeros(n + 1, 1)
[optTheta, functionVal, exitFlag] = fminunc(@(t)costFunction(t, X, y), initialTheta, options);

正则化项

线性回归

为了防止 overfitting（过度拟合），对 cost function 引入了正则化项 $\frac{λ}{2 m} \sum_{i = 1}^{n} θ_{i}^{2}$

J (θ) = 1 2 m [ \sum i = 1 m ( h θ ( x ( i ) ) - y ( i ) ) 2 +

$θ \in R^{(n + 1) \times 1}, X \in R^{m \times (n + 1)}, y \in R^{m \times 1}$

\partial J ( θ ) \partial θ j = ⎧⎩⎨⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪ 1 m \sum i = 1 m ( h θ ( x ( i )

$θ \in R^{(n + 1) \times 1}, X \in R^{m \times (n + 1)}, y \in R^{m \times 1}$

梯度下降法

Repeat {

θ 0 θ j := θ 0 - α 1 m \sum i = 1 m ( h θ ( x ( i ) ) -

$θ \in R^{(n + 1) \times 1}, X \in R^{m \times (n + 1)}, y \in R^{m \times 1}$

对上整理后：

θ j := θ j (1 - α λ m ) - α 1 m \sum i = 1 m ( h θ ( x ( i ) ) - y

由于 $1 - α \frac{λ}{m} < 1$

常规方法

直接令 $\frac{\partial J (θ)}{\partial θ_{i}} = 0$

θ = (X T X + λ \cdot ⎡⎣⎢⎢⎢⎢⎢⎢⎢ 0 0 0 ⋮ 0 0 1 0 ⋮ 0

Suppose $m \leq n$
在 $λ > 0$
$θ \in R^{(n + 1) \times 1}, X \in R^{m \times (n + 1)}, y \in R^{m \times 1}$

逻辑回归

对 cost function 引入了正则化项 $\frac{λ}{2 m} \sum_{i = 1}^{n} θ_{i}^{2}$

J (θ) = 1 m \sum i = 1 m C o s t ( h θ ( x ( i ) , y

$θ \in R^{(n + 1) \times 1}, X \in R^{m \times (n + 1)}, y \in R^{m \times 1}$

\partial J ( θ ) \partial θ j = ⎧⎩⎨⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪ 1 m \sum i = 1 m ( h θ ( x ( i )

注意！！！ $θ_{0}$

cost function code

function [J, grad] = costFunctionReg(theta, X, y, lambda)
tmp = sigmoid(X*theta);
J = -1 / m * (y'*log(tmp)+(1-y)'*log(1-tmp)) + lambda / (2 * m) * sum(theta(2:size(theta, 1),1) .^ 2);
theta(1) = 0;
grad = 1 / m * (X' * (tmp - y) + lambda * theta);
end

https://www.coursera.org/learn/machine-learning
教学方： Andrew Ng, Co-founder, Coursera; Adjunct Professor, Stanford University; formerly head of Baidu AI Group/Google Brain

posted @ 2018-05-09 15:31 jude_python 阅读(306) 评论(0) 编辑收藏举报

刷新页面返回顶部

Machine Learning - WEEK 1 2 3- 线性回归 、逻辑回归、梯度下降法及其优化算法、传统方法、 Octave 入门

线性回归

梯度下降法

特征优化：

传统方法

code

Octave

Ubuntu Install:

Octave 入门:

逻辑回归

高级优化算法

调用 Octave 优化算法 example :

正则化项

线性回归

梯度下降法

常规方法

逻辑回归

公告

Machine Learning - WEEK 1 2 3- 线性回归、逻辑回归、梯度下降法及其优化算法、传统方法、 Octave 入门