通过createDataFrame方法创建DataFrame对象
从集合中借助createDataFrame函数创建DataFrame
- createDataFrame(Seq[T]) 列名会自动生成
- 案例:
val dataFrame: DataFrame = session.createDataFrame(Array(
("zs", 20, "男"), ("ls", 21, "女"), ("ww", 32, "男")
))
dataFrame.show()
- createDataFrame(RDD[Row],StructType) ----根据数据和schema创建DataFrame ---结构体
- 案例:
val list:util.List[Row] = new util.ArrayList[Row]()
list.add(Row("zs", 20, "男"))
list.add(Row("ls", 21, "女"))
list.add(Row("ww", 22, "男"))
val structType: StructType = StructType(Array(
StructField("name", DataTypes.StringType),
StructField("age", DataTypes.IntegerType),
StructField("sex", DataTypes.StringType)
))
val dataFrame1: DataFrame = session.createDataFrame(list, structType)
dataFrame1.show()
- createDataFrame(List[Row],StructType) 借助反射创建DF
- 案例:
/**
* 限制:理论上,样例类前面的RDD是什么类型都无所谓,不报错可以运行,但是结果不对
* 如果要使用样例类,RDD类型最好是样例类的类型,还得需要配置一些东西(第二个参数)
* 样例类的反射机制去创建DataFrame,第二个参数传递的是一个Class对象。但是一定要注意
* 传递的是一个Class对象的实例--样例类的Class实例对象。实例是这样获取的classOf[Student]
* 千万不能使用Student.getClass去获取
*
* 【注意】scala中getClass和classOf的区别
* getClass获取回来的是Class这个类----String
* classOf获取的是Class的实例对象-----"zs"
*/
val list1:util.List[Student] = new util.ArrayList[Student]()
list1.add(Student("zs", 20, "男"));
list1.add(Student("ls", 21, "男"));
list1.add(Student("ww", 22, "女"));
list1.add(Student("ml", 20, "女"));
val dataFrame2:DataFrame = session.createDataFrame(list1, classOf[Student])
dataFrame2.show()
case class Student(@BeanProperty var name:String,@BeanProperty var age:Int,@BeanProperty var sex:String)
从RDD中借助createDataFrame函数创建DataFrame
- createDataFrame(RDD[T]) ----创建的DF列名是随机生成的
- 案例:
val sc = session.sparkContext
val rdd: RDD[(String, Int, String)] = sc.makeRDD(Array(("zs", 20, "男")))
val frame = session.createDataFrame(rdd)
frame.show()
- createDataFrame(RDD[T],beanClass) ----借助反射创建DF
- 案例:
val rdd1: RDD[(String, Int, String)] = sc.makeRDD(Array(("zs", 20, "男")))
val value: RDD[Student] = rdd1.map((tuple3: (String, Int, String)) => {
Student(tuple3._1, tuple3._2, tuple3._3)
})
val dataFrame = session.createDataFrame(value, classOf[Student])
dataFrame.show()
case class Student(@BeanProperty var name:String,@BeanProperty var age:Int,@BeanProperty var sex:String)
- createDataFrame(RDD[Row],StructType) ----根据数据和schema创建DataFrame ---结构体
val rdd2: RDD[(String, Int, String)] = sc.makeRDD(Array(("zs", 20, "男")))
val value2: RDD[Row] = rdd2.map((tuple3: (String, Int, String)) => {
Row(tuple3._1, tuple3._2, tuple3._3)
})
val structType = StructType(Array(
StructField("name", DataTypes.StringType),
StructField("age", DataTypes.IntegerType),
StructField("sex", DataTypes.StringType)
))
val dataFrame1 = session.createDataFrame(value2, structType)
dataFrame1.show()
本文来自博客园,作者:jsqup,转载请注明原文链接:https://www.cnblogs.com/jsqup/p/16627169.html