mahout分类

分类看起来比聚类和推荐麻烦多了

分类算法与聚类和推荐算法的不同：必须是有明确结果的，必须是有监督的，主要用于预测和检测

Mahout的优势 mahout的分类算法对资源的要求不会快于训练数据和测试数据的增长速度，而且可以转换为分布式应用（数据规模如果不够大 Mahout表现可能不及其他类型的系统）

关键词表：

Key idea	Description
Model	A computer program that makes decisions; in classification, the output of the training algorithm is a model
Training Data	Subset of training examples labeled with the value of the target variable and used as input to the learning algorithm to produce the model
Test Data	Withheld portion of training examples given to the model without the value for the target variable (although the value is known) and used to evaluate the model
Training	Learning process that uses training data to produce a model. That model can then compute estimates of the target variable given the predictor variables as inputs.
Training example	Entity with features that will be used as input for learning algorithm
Feature	A known characteristic of a training or new example; a “feature” is equivalent to saying a “characteristic”.
Variable	In this context, a variable is equivalent to a the value of a feature or a function of several features. This usage is somewhat different from a normal variable in a computer program.
Record	A container where an example is stored; such a record is composed of fields.
Field	Part of a record that contains the value of a feature (variable)
Predictor variable	Feature selected for use as input for a classification model. Not all features need be used. Some features may be algorithmic combinations of other features.
Target variable	Feature that the classification model is attempting to estimate: the target variable is categorical and its determination is the aim of the classification system.

一般来说：80-90%的数据作为training Data 其他数据作为Test Data数据

 

Mahout分类中的四种数据类型

 

Type of Value	Description
Continuous	This type of value is a floating point value. This might be a price, a weight, a time, a value or anything else that has a numerical magnitude and where this magnitude is the key property of the value.
Categorical	A categorical value can have one of a pre-specified set of values. Typically the set of categorical values is relatively small and may be as small as two, although the set can be quite large. Boolean values are generally treated as categorical values. Another example might be a vendor id.
Word-like	A word-like value is like a categorical value, but it has an open-ended set of possible values.
Text-like	A text-like value is a sequence of word-like values, all of the same kind. Text is the classic example of a text-like value, but a list of email addresses or URL’s is also text-like.

 

数据类型

Name	Type	Value
from-address	word-like	George <george@fumble-tech.com>
in-address-book?	categorical(TRUE, FALSE)	TRUE
non-spam-words	text-like	“Ted”, “Mahout”, “User”, “lunch”
spam-words	text-like	“available”
unknown-words	continuous	0
message-length	continuous	31

 

分类的应用步骤

Stage	Step
1. Training the model	Define target variable Collect historical data Define predictor variables Select a learning algorithm Use learning algorithm to train model
2. Evaluating the model	Run test data Adjust input (different predictor variables and/or algorithm)
3. Using model in production	Input the new examples to estimate unknown target values Retrain model as needed

 

Mahout的命令行工具

$ $MAHOUT_HOME/bin/mahout

An example program must be given as the first argument.

Valid program names are:

canopy: : Canopy clustering

cat : Print a file or resource as the logistic regression models would see it

...

runlogistic : Run a logistic regression model against CSV data

...

trainlogistic : Train a logistic regression using stochastic gradient descent

 

demo：

cat 查看一个文件

$ bin/mahout cat donut.csv

"x","y","shape","color","k","k0","xx","xy","yy","a","b","c","bias"

0.923307513352484,0.0135197141207755,21,2,4,8,0.852496764213146,...,1

0.711011884035543,0.909141522599384,22,2,3,9,0.505537899239772,...,1

...

0.67132937326096,0.571220482233912,23,1,5,2,0.450683127402953,...,1

0.548616112209857,0.405350996181369,24,1,5,3,0.300979638576258,...,1

0.677980388281867,0.993355110753328,25,2,3,9,0.459657406894831,...,1

$

 

Trainlogistic：根据数据训练生成model

$ $MAHOUT_HOME/bin/mahout trainlogistic --input donut.csv \

--output ./model \

--target color --categories 2 \

--predictors x y --types numeric \

--features 20 --passes 100 --rate 50

...

color ~ -0.157*Intercept Term + -0.678*x + -0.416*y

Intercept Term -0.15655

x -0.67841

y -0.41587

...

Option	Whatit Does
--quiet	Produce less status and progress output
--input <file-or-resource>	Use the specified file or resource as input
--output <file-for-model>	Put the model into the specified file
--target <variable>	Use the specified variable as the target
--categories <n>	How many categories does the target variable have?
--predictors <v1> ... <vn>	A list of the names of the predictor variables
--types <t1> ... <tm>	A list of the types of the predictor variables. Each type should be one of numeric, word or text. Types can be abbreviated to their first letter. If too few types are given, the last one is used again as necessary. Use word for categorical variables.
--passes	The number of times the input data should be re-examined during training. Small input files may need to be examined dozens of times. Very large input files probably don’t even need to be completely examined
--lambda	Controls how much the algorithm tries to eliminate variables from the final model. A value of 0 indicates no effort is made. Typical values are on the order of 0.00001 or less.
--rate	The initial learning rate. This can be large if you have lots of data or use lots of passes because it is decreased progressively as data is examined.
--noBias	Do not use the built-in constant in the model (this eliminates the Intercept Term from the model. Occasionally this is a good idea, but generally it is not since the SGD learning algorithm can usually eliminate the intercept term if warranted.
--features	The size of the internal feature vector to use in building the model. A larger value here can be helpful, especially with text-like input data.

 

Runlogistic model评价

$ bin/mahout runlogistic --input donut.csv --model ./model \

--auc --confusion

AUC = 0.57

confusion: [[27.0, 13.0], [0.0, 0.0]]

AUC 和confusion表示分类准确率 AUC（readingData的正确率越接近1越好） confusion（识别率和误识率）

 

参数说明

Option	What it Does
--quiet	Produce less status and progress output
--auc	Print out AUC score for model versus input data after reading data
--scores	Print target variable value and scores for each input example
--threshold <t>	Set the threshold for confusion matrix computation to t (default 0.5)
--confusion	Print out confusion matrix for a particular threshold (See --threshold)
--input <input>	Read data records from specified file or resource
--model <model>	Read model from specified file

 

分类看起来比聚类和推荐麻烦多了

分类算法与聚类和推荐算法的不同：必须是有明确结果的，必须是有监督的，主要用于预测和检测

Mahout的优势 mahout的分类算法对资源的要求不会快于训练数据和测试数据的增长速度，而且可以转换为分布式应用（数据规模如果不够大 Mahout表现可能不及其他类型的系统）

关键词表：

Key idea	Description
Model	A computer program that makes decisions; in classification, the output of the training algorithm is a model
Training Data	Subset of training examples labeled with the value of the target variable and used as input to the learning algorithm to produce the model
Test Data	Withheld portion of training examples given to the model without the value for the target variable (although the value is known) and used to evaluate the model
Training	Learning process that uses training data to produce a model. That model can then compute estimates of the target variable given the predictor variables as inputs.
Training example	Entity with features that will be used as input for learning algorithm
Feature	A known characteristic of a training or new example; a “feature” is equivalent to saying a “characteristic”.
Variable	In this context, a variable is equivalent to a the value of a feature or a function of several features. This usage is somewhat different from a normal variable in a computer program.
Record	A container where an example is stored; such a record is composed of fields.
Field	Part of a record that contains the value of a feature (variable)
Predictor variable	Feature selected for use as input for a classification model. Not all features need be used. Some features may be algorithmic combinations of other features.
Target variable	Feature that the classification model is attempting to estimate: the target variable is categorical and its determination is the aim of the classification system.

一般来说：80-90%的数据作为training Data 其他数据作为Test Data数据

 

Mahout分类中的四种数据类型

 

Type of Value	Description
Continuous	This type of value is a floating point value. This might be a price, a weight, a time, a value or anything else that has a numerical magnitude and where this magnitude is the key property of the value.
Categorical	A categorical value can have one of a pre-specified set of values. Typically the set of categorical values is relatively small and may be as small as two, although the set can be quite large. Boolean values are generally treated as categorical values. Another example might be a vendor id.
Word-like	A word-like value is like a categorical value, but it has an open-ended set of possible values.
Text-like	A text-like value is a sequence of word-like values, all of the same kind. Text is the classic example of a text-like value, but a list of email addresses or URL’s is also text-like.

 

数据类型

Name	Type	Value
from-address	word-like	George <george@fumble-tech.com>
in-address-book?	categorical(TRUE, FALSE)	TRUE
non-spam-words	text-like	“Ted”, “Mahout”, “User”, “lunch”
spam-words	text-like	“available”
unknown-words	continuous	0
message-length	continuous	31

 

分类的应用步骤

Stage	Step
1. Training the model	Define target variable Collect historical data Define predictor variables Select a learning algorithm Use learning algorithm to train model
2. Evaluating the model	Run test data Adjust input (different predictor variables and/or algorithm)
3. Using model in production	Input the new examples to estimate unknown target values Retrain model as needed

 

Mahout的命令行工具

$ $MAHOUT_HOME/bin/mahout

An example program must be given as the first argument.

Valid program names are:

canopy: : Canopy clustering

cat : Print a file or resource as the logistic regression models would see it

...

runlogistic : Run a logistic regression model against CSV data

...

trainlogistic : Train a logistic regression using stochastic gradient descent

 

demo：

cat 查看一个文件

$ bin/mahout cat donut.csv

"x","y","shape","color","k","k0","xx","xy","yy","a","b","c","bias"

0.923307513352484,0.0135197141207755,21,2,4,8,0.852496764213146,...,1

0.711011884035543,0.909141522599384,22,2,3,9,0.505537899239772,...,1

...

0.67132937326096,0.571220482233912,23,1,5,2,0.450683127402953,...,1

0.548616112209857,0.405350996181369,24,1,5,3,0.300979638576258,...,1

0.677980388281867,0.993355110753328,25,2,3,9,0.459657406894831,...,1

$

 

Trainlogistic：根据数据训练生成model

$ $MAHOUT_HOME/bin/mahout trainlogistic --input donut.csv \

--output ./model \

--target color --categories 2 \

--predictors x y --types numeric \

--features 20 --passes 100 --rate 50

...

color ~ -0.157*Intercept Term + -0.678*x + -0.416*y

Intercept Term -0.15655

x -0.67841

y -0.41587

...

Option	Whatit Does
--quiet	Produce less status and progress output
--input <file-or-resource>	Use the specified file or resource as input
--output <file-for-model>	Put the model into the specified file
--target <variable>	Use the specified variable as the target
--categories <n>	How many categories does the target variable have?
--predictors <v1> ... <vn>	A list of the names of the predictor variables
--types <t1> ... <tm>	A list of the types of the predictor variables. Each type should be one of numeric, word or text. Types can be abbreviated to their first letter. If too few types are given, the last one is used again as necessary. Use word for categorical variables.
--passes	The number of times the input data should be re-examined during training. Small input files may need to be examined dozens of times. Very large input files probably don’t even need to be completely examined
--lambda	Controls how much the algorithm tries to eliminate variables from the final model. A value of 0 indicates no effort is made. Typical values are on the order of 0.00001 or less.
--rate	The initial learning rate. This can be large if you have lots of data or use lots of passes because it is decreased progressively as data is examined.
--noBias	Do not use the built-in constant in the model (this eliminates the Intercept Term from the model. Occasionally this is a good idea, but generally it is not since the SGD learning algorithm can usually eliminate the intercept term if warranted.
--features	The size of the internal feature vector to use in building the model. A larger value here can be helpful, especially with text-like input data.

 

Runlogistic model评价

$ bin/mahout runlogistic --input donut.csv --model ./model \

--auc --confusion

AUC = 0.57

confusion: [[27.0, 13.0], [0.0, 0.0]]

AUC 和confusion表示分类准确率 AUC（readingData的正确率越接近1越好） confusion（识别率和误识率）

 

参数说明

Option	What it Does
--quiet	Produce less status and progress output
--auc	Print out AUC score for model versus input data after reading data
--scores	Print target variable value and scores for each input example
--threshold <t>	Set the threshold for confusion matrix computation to t (default 0.5)
--confusion	Print out confusion matrix for a particular threshold (See --threshold)
--input <input>	Read data records from specified file or resource
--model <model>	Read model from specified file