pyspark的dataframe的一些问题

pandas 的dataframe转spark的dataframe时报错 Can not merge type ?

可以将字段类型全部转成string

from pyspark.sql.types import StructField, StringType, FloatType, StructType

#字段之间用空格分隔
schemaString = "label_word word_weight word_flag"
fields = [StructField(field_name, StringType(), True) for field_name in schemaString.split(' ')]
schema = StructType(fields)
schemaPeople = spark.createDataFrame(owords_result, schema)

spark的df写csv带表头?

df.write.option("header", True).format("csv").save("output/csv/")
posted @ 2021-02-02 17:41  吸血鬼尼克  阅读(274)  评论(0编辑  收藏  举报