My naive machine learning notes

Notes:

This page records my naive machine learning notes. 

 

• is learning feasible ?
  • 　　Hoeffding inequaility : link
    • 　　Hoeffding inequality formular's left side is about something bad happending. You don't want this bad thing to happen, so that you can use a upper bound to constraint it. The right side of the Hoeffding inequality is the restriction. As you can see, it is either/both the larger the sample number (N) you have, or the smaller tolerance (epslon) you set that can make the upperbound smaller. 
    • on the other hand, if your hyphothesis set size M which is large (say infinity), the upper bound of the Hoeffding inequality needs to multiply this M (according to some math), then the upper bound becomes infinity. So we need to abstract quantity from this infinity to make it finite.  
    • the feasibility of learning is split into two questions:
    1. can we make sure that E_out(g) is close enought to E_in(g), where E_in(g) is the hypothesis g's in-sample error,  E_out(g) is hypothesis g's out-sample error. --> Hoeffding inequality answers this.
    2. can we make E_in(g) small enough ? --> depends on the complexity of H -- the number (M) of the hypothesis in the hypothesis set H, and the complexity of the target function f -- learning a non-linear target function is more prone to make E_in(g) bigger. 
  • 　　a model corresponds to a hypothesis set (H), a H contains a set of hypothesis (h), you choose one h and it is not called g, which (you believe) is approximate to the target function f. 
    • 　　how you pick g depends on the algorithm, hypothesis set and data you use, take perceptron for example, a g is picked util all the points are classified. There are multiple hypothesis that classify points correct, so how do you pick up the g?