Spark 调优

 

partitionBy 调优

  1. https://mungingdata.com/apache-spark/partitionby/
  2. http://tantusdata.com/spark-shuffle-case-1-partition-by-and-repartition/

 

Join 调优

  1. https://www.waitingforcode.com/apache-spark-sql/shuffle-join-spark-sql/read#shuffle_join_explained
  2. https://www.waitingforcode.com/apache-spark-sql/broadcast-join-spark-sql/read#:~:text=Broadcast%20join%20explained,variable%20(so%20only%20once).&text=The%20broadcast%20join%20is%20controlled%20through%20spark.
  3. https://www.waitingforcode.com/apache-spark-sql/sort-merge-join-spark-sql/read#:~:text=In%20Spark%20SQL%20the%20sort,is%20implemented%20in%20similar%20manner.&text=Thus%20it's%20important%20to%20ensure,can%20be%20activated%20through%20spark.
  4. https://mungingdata.com/apache-spark/broadcast-joins/

数据倾斜调优

  1. https://www.cnblogs.com/qingyunzong/p/8946679.html
posted @ 2020-08-16 21:21  mashuai_191  阅读(110)  评论(0)  编辑  收藏  举报