SparkConf
Spark的配置类,通过ConcurrentHashMap来存储配置,泛型为String类型
private val settings = new ConcurrentHashMap[String, String]()
Spark配置可以通过3种方式获取:
1.系统默认加载
如果loadDefaults为true,会加载系统配置,通过Utils工具类来取以spark.开头的key,再通过调用set方法配置参数
if (loadDefaults) {
loadFromSystemProperties(false)
}
private[spark] def loadFromSystemProperties(silent: Boolean): SparkConf = {
// Load any spark.* system properties
for ((key, value) <- Utils.getSystemProperties if key.startsWith("spark.")) {
set(key, value, silent)
}
this
}
set方法如下:
private[spark] def set(key: String, value: String, silent: Boolean): SparkConf = {
if (key == null) {
throw new NullPointerException("null key")
}
if (value == null) {
throw new NullPointerException("null value for " + key)
}
if (!silent) {
logDeprecationWarning(key)
}
settings.put(key, value)
this
}
2.通过set方法设置
SparkConf中有一系列set方法,比如setMaster、setAppName等,实际上也是调用了set方法对settings进行设置
3.对SparkConf进行克隆
SparkConf类继承了Clonable特质并实现了clone方法,可以对Spark的配置进行复制,以便于在任何地方使用
override def clone: SparkConf = {
val cloned = new SparkConf(false)
settings.entrySet().asScala.foreach { e =>
cloned.set(e.getKey(), e.getValue(), true)
}
cloned
}