基于Spark的电影推荐系统（推荐系统~4）
第四部分-推荐系统-模型训练
- 本模块基于第3节 数据加工得到的训练集和测试集数据 做模型训练，最后得到一系列的模型，进而做 预测。
- 训练多个模型，取其中最好，即取RMSE(均方根误差)值最小的模型
说明几点
1.ALS 算法不需要自己实现，Spark MLlib 已经实现好了，可以自己 跟源码学习
花时间钻研，动手写，写代码 翻译论文 写博客 多下功夫
2. 最新http://spark.apache.org/docs/latest/ml-guide.html
3. spark1.6.3
spark.mllib contains the original API built on top of RDDs.
spark.ml provides higher-level API built on top of DataFrames for constructing ML pipelines.
==》 我们采用spark.mllib ,也就是基于RDD之上来构建
学习：http://spark.apache.org/docs/1.6.3/mllib-collaborative-filtering.html#collaborative-filtering
import org.apache.spark.mllib.recommendation.ALS
import org.apache.spark.mllib.recommendation.MatrixFactorizationModel
import org.apache.spark.mllib.recommendation.Rating
// Load and parse the data
val data = sc.textFile("data/mllib/als/test.data")
val ratings = data.map(_.split(',') match { case Array(user, item, rate) =>
Rating(user.toInt, item.toInt, rate.toDouble)
})
// Build the recommendation model using ALS
val rank = 10
val numIterations = 10
val model = ALS.train(ratings, rank, numIterations, 0.01)
// Evaluate the model on rating data
val usersProducts = ratings.map { case Rating(user, product, rate) =>
(user, product)
}
val predictions =
model.predict(usersProducts).map { case Rating(user, product, rate) =>
((user, product), rate)
}
val ratesAndPreds = ratings.map { case Rating(user, product, rate) =>
((user, product), rate)
}.join(predictions)
val MSE = ratesAndPreds.map { case ((user, product), (r1, r2)) =>
val err = (r1 - r2)
err * err
}.mean()
println("Mean Squared Error = " + MSE)
// Save and load model
model.save(sc, "target/tmp/myCollaborativeFilter")
val sameModel = MatrixFactorizationModel.load(sc, "target/tmp/myCollaborativeFilter")
看官方是怎么写代码的，参照着写
开始项目Coding
步骤一： 继续在前面的项目中，新建ml包，再新建ModelTraining
package com.csylh.recommend.ml
import org.apache.spark.mllib.recommendation.{ALS, Rating}
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.SparkSession
/**
* Description:
* 训练多个模型，取其中最好，即取RMSE(均方根误差)值最小的模型
*
* @Author: 留歌36
* @Date: 2019-07-17 16:56
*/
object ModelTraining {
def main(args: Array[String]): Unit = {
// 面向SparkSession编程
val spark = SparkSession.builder()
.enableHiveSupport() //开启访问Hive数据, 要将hive-site.xml等文件放入Spark的conf路径
.getOrCreate()
val sc = spark.sparkContext
// 在生产环境中一定要注意设置spark.sql.shuffle.partitions，默认是200,及需要配置分区的数量
val shuffleMinPartitions = "8"
spark.sqlContext.setConf("spark.sql.shuffle.partitions",shuffleMinPartitions)
// 训练集，总数据集的70%
val trainingData = spark.sql("select * from trainingData")
// 测试集，总数据集的30%
val testData = spark.sql("select * from testData")
//--------------------------
// 训练集，转为Rating格式
val ratingRDD = trainingData.rdd.map(x => Rating(x.getInt(0), x.getInt(1), x.getDouble(2)))
// 用于计算模型的RMSE Rating(userid, movieid, rating) ==>转为tuple (userid, movieid)
val training2 :RDD[(Int,Int)] = ratingRDD.map{ case Rating(userid, movieid, rating) => (userid, movieid)}
// 测试集，转为Rating格式
val testRDD = testData.rdd.map(x => Rating(x.getInt(0), x.getInt(1), x.getDouble(2)))
val test2 :RDD[((Int,Int),Double)]= testRDD.map {case Rating(userid, movieid, rating) => ((userid, movieid), rating)}
//--------------------------
// 特征向量的个数
val rank = 1
// 正则因子
// val lambda = List(0.001, 0.005, 0.01, 0.015)
val lambda = List(0.001, 0.005, 0.01)
// 迭代次数
val iteration = List(10, 15, 18)
var bestRMSE = Double.MaxValue
var bestIteration = 0
var bestLambda = 0.0
// persist可以根据情况设置其缓存级别
ratingRDD.persist() // 持久化放入内存，迭代中使用到的RDD都可以持久化
training2.persist()
test2.persist()
for (l <- lambda; i <- iteration) {
// 循环收敛这个模型
//lambda 用于表示过拟合的这样一个参数，值越大，越不容易过拟合，但精确度就低
val model = ALS.train(ratingRDD, rank, i, l)
//---------这里是预测-----------------
val predict = model.predict(training2).map {
// 根据 (userid, movieid) 预测出相对应的rating
case Rating(userid, movieid, rating) => ((userid, movieid), rating)
}
//-------这里是实际的predictAndFact-------------------
// 根据(userid, movieid)为key，将提供的rating与预测的rating进行比较
val predictAndFact = predict.join(test2)
// 计算RMSE(均方根误差)
val MSE = predictAndFact.map {
case ((user, product), (r1, r2)) =>
val err = r1 - r2
err * err
}.mean() // 求平均
val RMSE = math.sqrt(MSE) // 求平方根
// RMSE越小，代表模型越精确
if (RMSE < bestRMSE) {
// 将模型存储下来
model.save(sc, s"/tmp/BestModel/$RMSE")
bestRMSE = RMSE
bestIteration = i
bestLambda = l
}
println(s"Best model is located in /tmp/BestModel/$RMSE")
println(s"Best RMSE is $bestRMSE")
println(s"Best Iteration is $bestIteration")
println(s"Best Lambda is $bestLambda")
}
}
}
步骤二：将创建的项目进行打包上传到服务器
mvn clean package -Dmaven.test.skip=true
步骤三：编写shell 执行脚本
[root@hadoop001 ml]# vim model.sh
export HADOOP_CONF_DIR=/root/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop
$SPARK_HOME/bin/spark-submit \
--class com.csylh.recommend.ml.ModelTraining \
--master spark://hadoop001:7077 \
--name ModelTraining \
--driver-memory 10g \
--executor-memory 5g \
/root/data/ml/movie-recommend-1.0.jar
步骤四：执行 sh model.sh 即可
sh model.sh之前：
[root@hadoop001 ~]# hadoop fs -ls /tmp
19/10/20 20:53:59 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 10 items
drwx------ - root supergroup 0 2019-04-01 16:27 /tmp/hadoop-yarn
drwx-wx-wx - root supergroup 0 2019-04-02 09:33 /tmp/hive
drwxr-xr-x - root supergroup 0 2019-10-20 19:42 /tmp/links
drwxr-xr-x - root supergroup 0 2019-10-20 19:42 /tmp/movies
drwxr-xr-x - root supergroup 0 2019-10-20 19:43 /tmp/ratings
drwxr-xr-x - root supergroup 0 2019-10-20 19:43 /tmp/tags
drwxr-xr-x - root supergroup 0 2019-10-20 20:19 /tmp/testData
drwxr-xr-x - root supergroup 0 2019-10-20 20:19 /tmp/trainingData
drwxr-xr-x - root supergroup 0 2019-10-20 20:18 /tmp/trainingDataAsc
drwxr-xr-x - root supergroup 0 2019-10-20 20:19 /tmp/trainingDataDesc
[root@hadoop001 ~]#
sh model.sh之后：
这里运行很长时间，而且很有可能出现OOM。耐心等待~~
这些点都是要关注的，再就是shuffle 很重要
等待中。。。
[root@hadoop001 ~]# hadoop fs -ls /tmp/BestModel
19/10/20 21:26:36 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 2 items
drwxr-xr-x - root supergroup 0 2019-10-20 21:00 /tmp/BestModel/0.8521581387523667
drwxr-xr-x - root supergroup 0 2019-10-20 20:56 /tmp/BestModel/0.853805599360297
[root@hadoop001 ~]#
这里得到model /tmp/BestModel/0.8521581387523667 ，感觉不是很好。资源要是多一点的话，可以把迭代次数调大一点，估计模型可以更好。这里为了演示整个流程，模型差点就差点吧。思路搞懂就好。
有任何问题，欢迎留言一起交流~~
更多文章：基于Spark的电影推荐系统：https://blog.csdn.net/liuge36/column/info/29285
