pyspark的dataframe的一些问题
pandas 的dataframe转spark的dataframe时报错 Can not merge type ?
可以将字段类型全部转成string
from pyspark.sql.types import StructField, StringType, FloatType, StructType
#字段之间用空格分隔
schemaString = "label_word word_weight word_flag"
fields = [StructField(field_name, StringType(), True) for field_name in schemaString.split(' ')]
schema = StructType(fields)
schemaPeople = spark.createDataFrame(owords_result, schema)
spark的df写csv带表头?
df.write.option("header", True).format("csv").save("output/csv/")