sparkcontext 和 sparkconf

Posted on 2017-11-07 17:47  打杂滴  阅读(142)  评论(0编辑  收藏  举报

http://spark.apache.org/docs/latest/rdd-programming-guide.html

The first thing a Spark program must do is to create a SparkContext object, which tells Spark how to access a cluster. To create a SparkContext you first need to build a SparkConf object that contains information about your application.

Only one SparkContext may be active per JVM. You must stop() the active SparkContext before creating a new one.

SparkConf:

Configuration for a Spark application. Used to set various Spark parameters as key-value pairs.

All setter methods in this class support chaining. For example, you can write 

new SparkConf().setMaster("local").setAppName("My app").

 The appName parameter is a name for your application to show on the cluster UI. master is a Spark, Mesos or YARN cluster URL, or a special “local” string to run in local mode. In practice, when running on a cluster, you will not want to hardcode master in the program, but rather launch the application with spark-submit and receive it there. However, for local testing and unit tests, you can pass “local” to run Spark in-process.

Copyright © 2024 打杂滴
Powered by .NET 8.0 on Kubernetes