Mahout实战---编写自己的相似度计算方法
Mahout本身提供了很多的相似度计算方法,如PCC,COS等。但是当需要验证自己想出来的相似度计算公式是否是好的,这时候需要自己实现相似度类。研究了Mahout-core-0.9.jar的源码后,自己实现了一篇论文上面的相似度公式。:
论文题目:An effective collaborative filtering algorithm based on user preference clustering
具体公式如下:
实现过程:具体实现参考了LogLikeHoodSimilarity类的实现
1,实现org.apache.mahout.cf.taste.similarity.UserSimilarity接口
该接口有三个方法:
public interface UserSimilarity extends Refreshable { double userSimilarity(long userID1, long userID2) throws TasteException; void setPreferenceInferrer(PreferenceInferrer inferrer); void refresh(Collection<Refreshable> alreadyRefreshed);//是Refreshable的方法 }
2,void refresh(Collection<Refreshable> alreadyRefreshed);
该方法用于刷新组件(Mahout对于数据改变的时候做出的应对方法。《Mahout实战》中3.2.3节可刷新组件中提到);具体实现如下:
public void refresh(Collection<Refreshable> alreadyRefreshed) { // TODO Auto-generated method stub alreadyRefreshed = RefreshHelper.buildRefreshed(alreadyRefreshed); RefreshHelper.maybeRefresh(alreadyRefreshed, getDataModel()); }
3,void setPreferenceInferrer(PreferenceInferrer inferrer);
这个方法我没有实现,它的作用:可以通过PreferenceInferrer 得到对未打分项的预测评分。
4,double userSimilarity(long userID1, long userID2) throws TasteException;
该方法需要根据公式实现:计算user1和user2的相似度。
在这之前需要传递一个DataModel进来(定义成类的成员变量,由构造函数传递进来)。
具体实现如下:
/** * 实现该方法即实现了相似度计算方法 */ public double userSimilarity(long userID1, long userID2) throws TasteException { // TODO Auto-generated method stub DataModel dataModel = getDataModel(); //获取用户打分项的id集合 FastIDSet prefs1 = dataModel.getItemIDsFromUser(userID1); FastIDSet prefs2 = dataModel.getItemIDsFromUser(userID2); long prefs1Size = prefs1.size(); long prefs2Size = prefs2.size(); /* * long intersectionSize = prefs1Size < prefs2Size ? * prefs2.intersectionSize(prefs1) : prefs1.intersectionSize(prefs2); */ // 计算交集的大小和产生新的FastIDSet作为交集 FastIDSet pre_a, pre_b;// a为大的集合 FastIDSet pre_com = new FastIDSet(); if (prefs1Size < prefs2Size) { pre_a = prefs2; pre_b = prefs1; } else { pre_a = prefs1; pre_b = prefs2; } int intersectionSize = 0; Iterator<Long> iterator = pre_b.iterator(); while (iterator.hasNext()) { long type = (long) iterator.next(); if (pre_a.contains(type)) { pre_com.add(type); } } intersectionSize = pre_com.size(); // 如果交集为0,则相似度为0 if (intersectionSize == 0) { return 0; } // 计算并集的大小 long unionSize = unionSize(pre_a, pre_b); // 计算userID1的平均打分 float avg_1 = avgPreferences(userID1, prefs1); // 计算userID2的平均打分 float avg_2 = avgPreferences(userID2, prefs2); // 计算共同打分项的打分差的和 double sum = 0.0; iterator = pre_com.iterator(); while (iterator.hasNext()) { long itemID = iterator.next(); sum += Math .abs(dataModel.getPreferenceValue(userID1, itemID) - dataModel.getPreferenceValue(userID2, itemID)); } return Math.exp(-((sum * 1.0) / intersectionSize) * Math.abs(avg_1 - avg_2)) * ((intersectionSize * 1.0) / unionSize); } /** * FastIDSet只实现了intersectionSize(求交集), 现实现求并 */ private int unionSize(FastIDSet a, FastIDSet b) { int count = a.size(); Iterator<Long> iterator = b.iterator(); while (iterator.hasNext()) { long type = (long) iterator.next(); if (!a.contains(type)) { count++; } } return count; } /** * 计算用户的打分平均值 * * @throws TasteException */ private float avgPreferences(long userID, FastIDSet set) throws TasteException { float score = (float) 0.0; Iterator<Long> iterator = set.iterator(); while (iterator.hasNext()) { long type = (long) iterator.next(); score += dataModel.getPreferenceValue(userID, type); } return score / set.size(); }
5,测试实现的正确性
根据论文的测试数据对实现的正确性进行测试
生成ups.csv
![](https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif)
1,101,1.0 1,102,2.0 1,104,3.0 1,105,2.0 1,107,2.0 2,101,2.0 2,102,4.0 2,103,4.0 2,105,4.0 2,108,2.0 2,109,3.0 3,101,5.0 3,102,5.0 3,104,4.0 3,106,4.0 3,107,3.0 3,109,4.0 4,101,5.0 4,103,5.0 4,104,4.0 4,105,4.0 4,107,4.0 4,108,4.0 5,101,1.0 5,105,2.0 5,109,2.0
测试程序如下:
public class UPSTest { public static void main(String[] args) throws IOException, TasteException { String projectDir = System.getProperty("user.dir"); DataModel model = new FileDataModel(new File(projectDir + "/src/main/ups.csv")); UserSimilarity similarity = new UPSSimiliarity(model); DecimalFormat df = new DecimalFormat("#,##0.0000");// 保留4位小数 System.out.println(df.format(similarity.userSimilarity(1, 2))); System.out.println(df.format(similarity.userSimilarity(1, 3))); System.out.println(df.format(similarity.userSimilarity(1, 4))); System.out.println(df.format(similarity.userSimilarity(1, 5))); System.out.println(df.format(similarity.userSimilarity(2, 3))); System.out.println(df.format(similarity.userSimilarity(2, 4))); System.out.println(df.format(similarity.userSimilarity(2, 5))); System.out.println(df.format(similarity.userSimilarity(3, 4))); System.out.println(df.format(similarity.userSimilarity(3, 5))); System.out.println(df.format(similarity.userSimilarity(4, 5))); } }
运行结果如下:
与论文中的结果基本相同:
参考 论文:[1] Zhang, Jia, et al. "An effective collaborative filtering algorithm based on user preference clustering." Applied Intelligence (2016): 1-11.
[2] Mahout实战