Spark2 Dataset统计指标:mean均值,variance方差,stddev标准差,corr(Pearson相关系数),skewness偏度,kurtosis峰度
val df4=spark.sql("SELECT mean(age),variance(age),stddev(age),corr(age,yearsmarried),skewness(age),kurtosis(age) FROM Affairs") df4.show +--------+------------------+------------------+-----------------------+-----------------+--------------------+ |avg(age)| var_samp(age)| stddev_samp(age)|corr(age, yearsmarried)| skewness(age)| kurtosis(age)| +--------+------------------+------------------+-----------------------+-----------------+--------------------+ | 34.0|173.33333333333334|13.165611772087667| 0.7456766124552038|0.965388004190285|-0.43417159763313595| +--------+------------------+------------------+-----------------------+-----------------+--------------------+