spark event log

val df = spark.read.json("/spark2x/xxx")
df.printSchema
df.select("Event").groupBy("Event").count.show(20,false)
df.createOrReplaceTempView("t1")
sql("select Event,count(*) from t1 group by Event").show(30,false)

val df2 = df.filter("Event='SparkListenerStageCompleted'").select("Stage Info.*")

df2.createOrReplaceTempView("t2")

val df4 = sql("select *  from t2")

df4.show(20,false)

df4.createOrReplaceTempView("t4")

参考：https://github.com/LucaCanali/Miscellaneous/blob/master/Spark_Notes/Spark_EventLog.md

 

Spark 3.0 终于支持 event logs 滚动 https://blog.csdn.net/wypblog/article/details/104765691/